Open deric opened 2 months ago
here's full log of another drive in similar condition:
smartctl -a /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-16-amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Crucial/Micron MX100/MX200/M5x0/M600 Client SSDs
Device Model: Crucial_CT250MX200SSD1
Serial Number: 16271327ED78
LU WWN Device Id: 5 00a075 11327ed78
Firmware Version: MU03
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Sep 11 18:38:08 2024 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 795) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 5) minutes.
Conveyance self-test routine
recommended polling time: ( 3) minutes.
SCT capabilities: (0x0035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 1126
5 Reallocate_NAND_Blk_Cnt 0x0032 100 100 010 Old_age Always - 1
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 69602
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 14
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Ave_Block-Erase_Count 0x0032 001 001 000 Old_age Always - 3657
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 7
180 Unused_Reserve_NAND_Blk 0x0033 000 000 000 Pre-fail Always - 2600
183 SATA_Interfac_Downshift 0x0032 100 100 000 Old_age Always - 0
184 Error_Correction_Count 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 244390
194 Temperature_Celsius 0x0022 068 050 000 Old_age Always - 32 (Min/Max 27/50)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 1
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
202 Percent_Lifetime_Used 0x0030 000 000 001 Old_age Offline FAILING_NOW 100
206 Write_Error_Rate 0x000e 100 100 000 Old_age Always - 0
210 Success_RAIN_Recov_Cnt 0x0032 100 100 000 Old_age Always - 0
246 Total_Host_Sector_Write 0x0032 100 100 000 Old_age Always - 340365865430
247 Host_Program_Page_Count 0x0032 100 100 000 Old_age Always - 10917042908
248 Bckgnd_Program_Page_Cnt 0x0032 100 100 000 Old_age Always - 30729664257
SMART Error Log Version: 1
Warning: ATA error count 47782 inconsistent with error log pointer 5
ATA Error Count: 47782 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 47782 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 c8 5b 40 40 Error: UNC at LBA = 0x00405bc8 = 4217800
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 c8 5b 40 40 08 16d+21:32:16.832 READ FPDMA QUEUED
47 00 01 30 08 00 e0 08 16d+21:32:16.832 READ LOG DMA EXT
47 00 01 13 00 00 e0 08 16d+21:32:16.832 READ LOG DMA EXT
47 00 01 00 00 00 e0 08 16d+21:32:16.832 READ LOG DMA EXT
ef 10 02 00 00 00 a0 08 16d+21:32:16.832 SET FEATURES [Enable SATA feature]
Error 47781 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 c8 5b 40 40 Error: UNC at LBA = 0x00405bc8 = 4217800
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 c8 5b 40 40 08 16d+21:32:16.832 READ FPDMA QUEUED
ea 00 00 00 00 00 e0 08 16d+21:32:16.832 FLUSH CACHE EXT
61 00 08 d8 1a 77 40 08 16d+21:32:16.832 WRITE FPDMA QUEUED
ea 00 00 00 00 00 e0 08 16d+21:32:16.832 FLUSH CACHE EXT
61 00 68 70 1a 77 40 08 16d+21:32:16.832 WRITE FPDMA QUEUED
Error 47780 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 c8 5b 40 40 Error: UNC at LBA = 0x00405bc8 = 4217800
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 c8 5b 40 40 08 16d+21:30:16.832 READ FPDMA QUEUED
47 00 01 30 08 00 e0 08 16d+21:30:16.832 READ LOG DMA EXT
47 00 01 13 00 00 e0 08 16d+21:30:16.832 READ LOG DMA EXT
47 00 01 00 00 00 e0 08 16d+21:30:16.832 READ LOG DMA EXT
ef 10 02 00 00 00 a0 08 16d+21:30:16.832 SET FEATURES [Enable SATA feature]
Error 47779 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 c8 5b 40 40 Error: UNC at LBA = 0x00405bc8 = 4217800
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 03 08 c8 5b 40 40 08 16d+21:30:16.832 READ FPDMA QUEUED
ea 00 00 00 00 00 e0 08 16d+21:30:16.832 FLUSH CACHE EXT
61 00 08 b8 08 77 40 08 16d+21:30:16.832 WRITE FPDMA QUEUED
ea 00 00 00 00 00 e0 08 16d+21:30:16.832 FLUSH CACHE EXT
61 10 a8 10 08 77 40 08 16d+21:30:16.832 WRITE FPDMA QUEUED
Error 47778 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 c8 5b 40 40 Error: UNC at LBA = 0x00405bc8 = 4217800
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 c8 5b 40 40 08 16d+21:28:16.832 READ FPDMA QUEUED
47 00 01 30 08 00 e0 08 16d+21:28:16.832 READ LOG DMA EXT
47 00 01 13 00 00 e0 08 16d+21:28:16.832 READ LOG DMA EXT
47 00 01 00 00 00 e0 08 16d+21:28:16.832 READ LOG DMA EXT
ef 10 02 00 00 00 a0 08 16d+21:28:16.832 SET FEATURES [Enable SATA feature]
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Vendor (0xff) Completed without error 00% 7927 -
# 2 Vendor (0xff) Completed without error 00% 7909 -
# 3 Vendor (0xff) Completed without error 00% 7863 -
# 4 Vendor (0xff) Completed without error 00% 7817 -
# 5 Vendor (0xff) Completed without error 00% 7772 -
# 6 Vendor (0xff) Completed without error 00% 7726 -
# 7 Vendor (0xff) Completed without error 00% 7681 -
# 8 Extended offline Completed without error 00% 7627 -
# 9 Vendor (0xff) Completed without error 00% 7626 -
#10 Extended offline Completed without error 00% 12 -
#11 Extended offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
What does ./check_smart.pl -i ata -d /dev/sdb -s
say?
default check is failing due to Percent_Lifetime_Used
./check_smart.pl -i ata -d /dev/sdb
CRITICAL: Drive Crucial_CT250MX200SSD1 S/N 16271327ED78: Health status: FAILED!, Reported_Uncorrect is non-zero (245410), Reallocated_Event_Count is non-zero (1), Attribute Percent_Lifetime_Used failed at FAILING_NOW, ata_errors is non-zero (48802), |Raw_Read_Error_Rate=1126;;;; Reallocate_NAND_Blk_Cnt=1;;;; Power_On_Hours=69619;;;; Power_Cycle_Count=14;;;; Program_Fail_Count=0;;;; Erase_Fail_Count=0;;;; Ave_Block-Erase_Count=3658;;;; Unexpect_Power_Loss_Ct=7;;;; Unused_Reserve_NAND_Blk=2600;;;; SATA_Interfac_Downshift=0;;;; Error_Correction_Count=0;;;; Reported_Uncorrect=245410;;;; Temperature_Celsius=33;;;; Reallocated_Event_Count=1;;;; Current_Pending_Sector=0;;;; Offline_Uncorrectable=0;;;; UDMA_CRC_Error_Count=0;;;; Percent_Lifetime_Used=100;;;; Write_Error_Rate=0;;;; Success_RAIN_Recov_Cnt=0;;;; Total_Host_Sector_Write=340518286562;;;; Host_Program_Page_Count=10921806085;;;; Bckgnd_Program_Page_Cnt=30739590916;;;; ata_errors=48802;;;;
-s
: gives WARNING
$ ./check_smart.pl -i ata -d /dev/sdb -E Percent_Lifetime_Used --skip-self-assessment -s
WARNING: Drive Crucial_CT250MX200SSD1 S/N 16271327ED78: Reported_Uncorrect is non-zero (245412), Reallocated_Event_Count is non-zero (1), ata_errors is non-zero (48804), |Raw_Read_Error_Rate=1126;;;; Reallocate_NAND_Blk_Cnt=1;;;; Power_On_Hours=69619;;;; Power_Cycle_Count=14;;;; Program_Fail_Count=0;;;; Erase_Fail_Count=0;;;; Ave_Block-Erase_Count=3658;;;; Unexpect_Power_Loss_Ct=7;;;; Unused_Reserve_NAND_Blk=2600;;;; SATA_Interfac_Downshift=0;;;; Error_Correction_Count=0;;;; Reported_Uncorrect=245412;;;; Temperature_Celsius=33;;;; Reallocated_Event_Count=1;;;; Current_Pending_Sector=0;;;; Offline_Uncorrectable=0;;;; UDMA_CRC_Error_Count=0;;;; Write_Error_Rate=0;;;; Success_RAIN_Recov_Cnt=0;;;; Total_Host_Sector_Write=340518649314;;;; Host_Program_Page_Count=10921817422;;;; Bckgnd_Program_Page_Cnt=30739592766;;;; ata_errors=48804;;;;
Yep, -s
checks the selftest logs and detected ATA alerts in the output: ata_errors is non-zero (48804)
Would that be enough or does your commit enhance this capability? (besides adding an option to skip the log check)
ata_errors
are added by this PR.
-s
self-test doesn't detect anything:
sudo /usr/sbin/smartctl -d ata -q silent -A /dev/sdb
(debug) exit code:
0
(debug) zero exit code, status OK
(debug) selftest log check activated
(debug) exit code:
0
Ah. Mixed something up, sorry. Too many projects at the same time...
Correct, the selftest logs would only alert if someone ran a selftest (smartctl -t ...
) and this logged an error.
So yeah, your contribution is very helpful for detecting errors not showing up in any attribute. I've had a few of such cases as well in the past and would have been happy if check_smart would have detected these :+1:
No problem.
Currently the script executes following commands:
smartctl -d ata -Hi /dev/sdb
smartctl -d ata -q silent -A /dev/sdb
smartctl -d ata -A /dev/sdb
smartctl -d ata -l error /dev/sdb
we could merge the last two if we execute -a
instead of -A
, but the code would be probably more complicated.
Yeah that improvement would be #14, which would be nice, but not urgent.
Ok, then we should add more command executions.
@Napsty Now it's using smarctl -a
and then checking the whole output. Is the attribute name ata_errors
ok?
Is the attribute name ata_errors ok?
Yeah, sounds good to me. It doesn't confuse with any existing SMART attribute, so this should be fine.
The idea is to inspect ATA device logs, which are also part of
smartctl -a /dev/sda
output. If the device log contains some errors it means that some operations are already failing, non-zero value would be escalated to aWARNING
kernel log might contain errors like this:
from man page:
=== START OF READ SMART DATA SECTION === SMART Error Log Version: 1 ATA Error Count: 16 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 16 occurred at disk power-on lifetime: 9377 hours (390 days + 17 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER ST SC SN CL CH DH
40 51 c0 b0 92 94 40 Error: UNC at LBA = 0x009492b0 = 9736880
Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
60 08 c0 b0 92 94 40 18 15:11:02.789 READ FPDMA QUEUED 60 08 b8 c8 92 94 40 17 15:11:02.789 READ FPDMA QUEUED 60 08 b0 90 92 94 40 16 15:11:02.789 READ FPDMA QUEUED 60 08 a8 60 13 94 40 15 15:11:02.789 READ FPDMA QUEUED 60 08 a0 f0 13 94 40 14 15:11:02.789 READ FPDMA QUEUED
Error 15 occurred at disk power-on lifetime: 9377 hours (390 days + 17 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER ST SC SN CL CH DH
40 51 c0 50 90 94 40 Error: UNC at LBA = 0x00949050 = 9736272
Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
60 08 c0 50 90 94 40 18 15:11:02.778 READ FPDMA QUEUED 61 00 b8 00 10 94 40 17 15:11:02.778 WRITE FPDMA QUEUED 61 e8 b0 40 15 74 40 16 15:11:02.778 WRITE FPDMA QUEUED 60 08 a8 20 94 94 40 15 15:11:02.778 READ FPDMA QUEUED 61 40 a0 00 10 74 40 14 15:11:02.778 WRITE FPDMA QUEUED
Error 14 occurred at disk power-on lifetime: 9377 hours (390 days + 17 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER ST SC SN CL CH DH
40 51 d0 40 9b 94 40 Error: UNC at LBA = 0x00949b40 = 9739072
Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
60 08 d0 40 9b 94 40 1a 15:11:02.777 READ FPDMA QUEUED 60 08 c8 30 9b 94 40 19 15:11:02.777 READ FPDMA QUEUED 60 08 c0 18 9b 94 40 18 15:11:02.777 READ FPDMA QUEUED 60 08 b8 08 9b 94 40 17 15:11:02.777 READ FPDMA QUEUED 60 08 b0 88 9b 94 40 16 15:11:02.777 READ FPDMA QUEUED
Error 13 occurred at disk power-on lifetime: 9377 hours (390 days + 17 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER ST SC SN CL CH DH
40 51 38 a0 97 94 40 Error: UNC at LBA = 0x009497a0 = 9738144
Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
60 08 38 a0 97 94 40 07 15:11:02.773 READ FPDMA QUEUED 60 08 30 b0 97 94 40 06 15:11:02.773 READ FPDMA QUEUED 60 08 28 d8 97 94 40 05 15:11:02.773 READ FPDMA QUEUED 61 08 18 e0 30 d0 40 03 15:11:02.773 WRITE FPDMA QUEUED 47 00 01 13 00 00 40 1e 15:11:02.773 READ LOG DMA EXT
Error 12 occurred at disk power-on lifetime: 9377 hours (390 days + 17 hours) When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were: ER ST SC SN CL CH DH
40 51 f0 70 97 94 40 Error: UNC at LBA = 0x00949770 = 9738096
Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
60 08 f0 70 97 94 40 1e 15:11:02.771 READ FPDMA QUEUED 60 08 e8 d8 95 94 40 1d 15:11:02.771 READ FPDMA QUEUED 60 08 e0 a0 96 94 40 1c 15:11:02.771 READ FPDMA QUEUED 60 08 d8 c0 96 94 40 1b 15:11:02.771 READ FPDMA QUEUED 60 08 d0 c0 95 94 40 1a 15:11:02.771 READ FPDMA QUEUED
./check_smart.pl -i ata -d /dev/sdb CRITICAL: Drive SAMSUNG MZ7LM480HCHP-00003 S/N S1YJNXAG900574: ATA Error Count: 16, Reallocated_Sector_Ct is non-zero (331), Runtime_Bad_Block is non-zero (331), Uncorrectable_Error_Cnt is non-zero (16), |Reallocated_Sector_Ct=331;;;; Power_On_Hours=75546;;;; Power_Cycle_Count=48;;;; Wear_Leveling_Count=6210;;;; Used_Rsvd_Blk_Cnt_Tot=331;;;; Unused_Rsvd_Blk_Cnt_Tot=2129;;;; Program_Fail_Cnt_Total=0;;;; Erase_Fail_Count_Total=0;;;; Runtime_Bad_Block=331;;;; End-to-End_Error=0;;;; Uncorrectable_Error_Cnt=16;;;; Airflow_Temperature_Cel=37;;;; ECC_Error_Rate=16;;;; Current_Pending_Sector=0;;;; CRC_Error_Count=0;;;; Exception_Mode_Status=0;;;; POR_Recovery_Count=37;;;; Total_LBAs_Written=4595275641341;;;; Total_LBAs_Read=50608004051;;;; SATA_Downshift_Ct=1;;;; Thermal_Throttle_St=0;;;; Timed_Workld_Media_Wear=65535;;;; Timed_Workld_RdWr_Ratio=65535;;;; Timed_Workld_Timer=65535;;;; NAND_Writes=6628479574168;;;; ata_errors=16;;;;