search

Malshare / MalShare

http://www.malshare.com
22 stars 4 forks source link

Unplanned outage - 2019 September 17 #35

Closed silascutler closed 5 years ago

silascutler commented 5 years ago

Host was not responding to remote connections. Webserver was online, however, displayed error saying system was offline.

silascutler commented 5 years ago

Error message points to samples drive was disconnected. Working on analysis of root cause

silascutler commented 5 years ago 
ID                              : 0:3
Status                          : Critical
Name                            : Physical Disk 0:3
State                           : Failed
Power Status                    : Not Applicable
Bus Protocol                    : SATA
Media                           : SSD
Part of Cache Pool              : Not Applicable
Remaining Rated Write Endurance : Not Applicable
Failure Predicted               : No
Revision                        : UH4400RL
Driver Version                  : Not Applicable
Model Number                    : Not Applicable
T10 PI Capable                  : No
Certified                       : Not Applicable
Encryption Capable              : No
Encrypted                       : Not Applicable
Progress                        : Not Applicable
Mirror Set ID                   : Not Applicable
Capacity                        : 931.00 GB (999653638144 bytes)
Used RAID Disk Space            : 931.00 GB (999653638144 bytes)
Available RAID Disk Space       : 0.00 GB (0 bytes)
Hot Spare                       : No
Vendor ID                       : 
Product ID                      : SanDisk SSD PLUS 1000GB
Part Number                     : Not Available
Negotiated Speed                : Not Available
silascutler commented 5 years ago

System crashed again. 

Sep 17 20:08:50 hydron rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
Sep 17 20:08:50 hydron rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="1900" x-info="http://www.rsyslog.com"] start
Sep 17 17:51:28 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:44:28 hydron Server_Administrator: 3116 2350 - Storage Service  There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:44:28 hydron Server_Administrator: message repeated 54 times: [ 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1]
Sep 17 17:44:27 hydron kernel: [ 4506.077278] megaraid_sas 0000:03:00.0: 11642 (622057467s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 04(e0xff/s4) at 4cc1ee68
Sep 17 17:42:29 hydron kernel: [ 4387.787592] print_req_error: I/O error, dev sdb, sector 2877792096
Sep 17 17:42:29 hydron kernel: [ 4387.787588] sd 2:2:1:0: [sdb] tag#24 CDB: Read(16) 88 00 00 00 00 00 ab 87 9f 60 00 00 00 08 00 00
Sep 17 17:42:29 hydron kernel: [ 4387.787568] sd 2:2:1:0: [sdb] tag#24 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Sep 17 17:42:10 hydron kernel: [ 4368.285621] print_req_error: I/O error, dev sdb, sector 2877792096
Sep 17 17:42:10 hydron kernel: [ 4368.285619] sd 2:2:1:0: [sdb] tag#7 CDB: Read(16) 88 00 00 00 00 00 ab 87 9f 60 00 00 00 08 00 00
Sep 17 17:42:10 hydron kernel: [ 4368.285604] sd 2:2:1:0: [sdb] tag#7 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Sep 17 17:41:50 hydron kernel: [ 4348.912125] print_req_error: I/O error, dev sdb, sector 2877791752
Sep 17 17:41:50 hydron kernel: [ 4348.912122] sd 2:2:1:0: [sdb] tag#2 CDB: Read(16) 88 00 00 00 00 00 ab 87 9e 08 00 00 02 00 00 00
Sep 17 17:41:50 hydron kernel: [ 4348.912106] sd 2:2:1:0: [sdb] tag#2 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
Sep 17 17:41:33 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:39:01 hydron CRON[13883]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
Sep 17 17:29:50 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_05] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 66 to 64
Sep 17 17:29:50 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_04] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 65 to 64
Sep 17 17:29:49 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_02] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 59 to 58
Sep 17 17:29:45 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_01] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 33 to 37
Sep 17 17:29:45 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_01] [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 63
Sep 17 17:29:45 hydron smartd[2136]: Device: /dev/bus/2 [megaraid_disk_01] [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 78 to 83
Sep 17 17:19:16 hydron Server_Administrator: 3116 2350 - Storage Service  There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:19:16 hydron Server_Administrator: message repeated 3 times: [ 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1]
Sep 17 17:19:15 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:19:13 hydron kernel: [ 2991.784032] megaraid_sas 0000:03:00.0: 11586 (622055953s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 04(e0xff/s4) at 2c830060
Sep 17 17:19:09 hydron Server_Administrator: 3116 2350 - Storage Service  There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:19:09 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:19:08 hydron kernel: [ 2986.899008] megaraid_sas 0000:03:00.0: 11581 (622055948s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 04(e0xff/s4) at 2c830060
Sep 17 17:17:01 hydron CRON[10592]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 17 17:16:57 hydron Server_Administrator: message repeated 14 times: [ 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1]
Sep 17 17:16:43 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:16:35 hydron Server_Administrator: 3116 2350 - Storage Service  There was an unrecoverable disk media error during the rebuild or recovery operation:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:16:35 hydron Server_Administrator: 3116 2095 - Storage Service  Unexpected sense. SCSI sense data: Sense key:  3 Sense code: 11 Sense qualifier:  0:  Physical Disk 1:4 Controller 0, Connector 1
Sep 17 17:16:35 hydron kernel: [ 2833.385982] megaraid_sas 0000:03:00.0: 11564 (622055795s/0x0002/FATAL) - Unrecoverable medium error during recovery on PD 04(e0xff/s4) at 2e482868
Sep 17 17:09:01 hydron CRON[9300]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)
Sep 17 17:07:17 hydron Server_Administrator: 3116 2358 - Storage Service  The battery charge cycle is complete.:  Battery 0 Controller 0
silascutler commented 5 years ago

Recreating the vdisk. Running full initialization.

larsborn commented 5 years ago

Restoring most recent snapshot of sample collection.

silascutler commented 5 years ago

It looks like we are stable. Closing out