Closed xsmile closed 11 months ago
Another issue with with nvme effects-log
that might be related:
Command output
Backtrace from another run
We've covered most of the places in libblockdev by https://github.com/storaged-project/libblockdev/pull/969
Looking at libblockdev, the crash takes place while calling nvme_scan(). libnvme calls nvme_ns_identify_descs().
Ha, that reminds me that libnvme is actually doing some calls when scanning the tree. Let me have a look at that...
Could you please test https://github.com/linux-nvme/libnvme/pull/727 ? Make sure you have https://github.com/storaged-project/libblockdev/pull/969
Another issue with with
nvme effects-log
that might be related:
Right, same issue, the tree scan within libnvme. Majority of other calls should've been fixed by https://github.com/linux-nvme/nvme-cli/pull/2051, released as nvme-cli-2.6
.
Could you please test linux-nvme/libnvme#727 ? Make sure you have storaged-project/libblockdev#969
Both udisksd
and nvme effects-log
do not produce the stack smashing error anymore. Thanks for the quick fix.
Nice, thanks for testing!
Suffering from this bug on Ubuntu 23 Mantic. Unclear how to workaround it without upgrading to Ubuntu Noble.
Would anybody have a workaround? Thanks!
Suffering from this bug on Ubuntu 23 Mantic. Unclear how to workaround it without upgrading to Ubuntu Noble.
What versions of libnvme
, libblockdev
and udisks
are you running? I believe we've fixed all the cases. Of course you need to be running the fixed versions.
On Ubuntu 23 Mantic:
libblockdev3 3.0.3-1
libnvme1 1.5-3
udisks2 2.10.1-1ubuntu1.1
Hello,
after updating udisks from 2.9.4 to 2.10.1 most of the time udisksd fails to run and crashes with a stack smashing error.
The last working commit is ff5fddb16a97d8f1b8475a219566c2ee95f3c349, right before NVMe support was added with b7ccbf2fedb160c0cf6ecc0fc35b2d575c4dc6de, which suggests that this is a libnvme issue.
A similarity to #1152 is the drive model, which is also used by @bulletmark. However, this does not seem to be caused by the sanitize function.
Details
nvme id-cntrl
Click to expand
``` # nvme id-ctrl -H /dev/nvme0n1 NVME Identify Controller: vid : 0x1c5c ssvid : 0x1c5c sn : NS04N638710704H1Q mn : SKHynix_HFS001TD9TNI-L2B0B fr : 11720C10 rab : 4 ieee : ace42e cmic : 0 [3:3] : 0 ANA not supported [2:2] : 0 PCI [1:1] : 0 Single Controller [0:0] : 0 Single Port mdts : 6 cntlid : 0x1 ver : 0x10300 rtd3r : 0x7a120 rtd3e : 0x1e8480 oaes : 0x200 [31:31] : 0 Discovery Log Change Notice Not Supported [27:27] : 0 Zone Descriptor Changed Notices Not Supported [15:15] : 0 Normal NSS Shutdown Event Not Supported [14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported [13:13] : 0 LBA Status Information Notices Not Supported [12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported [11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported [9:9] : 0x1 Firmware Activation Notices Supported [8:8] : 0 Namespace Attribute Changed Event Not Supported ctratt : 0 [19:19] : 0 Flexible Data Placement Not Supported [15:15] : 0 Extended LBA Formats Not Supported [14:14] : 0 Delete NVM Set Not Supported [13:13] : 0 Delete Endurance Group Not Supported [12:12] : 0 Variable Capacity Management Not Supported [11:11] : 0 Fixed Capacity Management Not Supported [10:10] : 0 Multi Domain Subsystem Not Supported [9:9] : 0 UUID List Not Supported [8:8] : 0 SQ Associations Not Supported [7:7] : 0 Namespace Granularity Not Supported [6:6] : 0 Traffic Based Keep Alive Not Supported [5:5] : 0 Predictable Latency Mode Not Supported [4:4] : 0 Endurance Groups Not Supported [3:3] : 0 Read Recovery Levels Not Supported [2:2] : 0 NVM Sets Not Supported [1:1] : 0 Non-Operational Power State Permissive Not Supported [0:0] : 0 128-bit Host Identifier Not Supported rrls : 0 cntrltype : 0 [7:2] : 0 Reserved [1:0] : 0 Controller type not reported fguid : 00000000-0000-0000-0000-000000000000 crdt1 : 0 crdt2 : 0 crdt3 : 0 nvmsr : 0 [1:1] : 0 NVM subsystem Not part of an Enclosure [0:0] : 0 NVM subsystem Not part of a Storage Device vwci : 0 [7:7] : 0 VPD Write Cycles Remaining field is Not valid. [6:0] : 0 VPD Write Cycles Remaining mec : 0 [1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port [0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port oacs : 0x17 [10:10] : 0 Lockdown Command and Feature Not Supported [9:9] : 0 Get LBA Status Capability Not Supported [8:8] : 0 Doorbell Buffer Config Not Supported [7:7] : 0 Virtualization Management Not Supported [6:6] : 0 NVMe-MI Send and Receive Not Supported [5:5] : 0 Directives Not Supported [4:4] : 0x1 Device Self-test Supported [3:3] : 0 NS Management and Attachment Not Supported [2:2] : 0x1 FW Commit and Download Supported [1:1] : 0x1 Format NVM Supported [0:0] : 0x1 Security Send and Receive Supported acl : 3 aerl : 7 frmw : 0x16 [5:5] : 0 Multiple FW or Boot Update Detection Not Supported [4:4] : 0x1 Firmware Activate Without Reset Supported [3:1] : 0x3 Number of Firmware Slots [0:0] : 0 Firmware Slot 1 Read/Write lpa : 0xa [6:6] : 0 Telemetry Log Data Area 4 Not Supported [5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported [4:4] : 0 Persistent Event log Not Supported [3:3] : 0x1 Telemetry host/controller initiated log page Supported [2:2] : 0 Extended data for Get Log Page Not Supported [1:1] : 0x1 Command Effects Log Page Supported [0:0] : 0 SMART/Health Log Page per NS Not Supported elpe : 255 [7:0] : 255 (0's based) Error Log Page Entries (ELPE) npss : 4 [7:0] : 4 (0's based) Number of Power States Support (NPSS) avscc : 0x1 [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format apsta : 0x1 [0:0] : 0x1 Autonomous Power State Transitions Supported wctemp : 358 [15:0] : 85 °C (358 K) Warning Composite Temperature Threshold (WCTEMP) cctemp : 359 [15:0] : 86 °C (359 K) Critical Composite Temperature Threshold (CCTEMP) mtfa : 0 hmpre : 0 hmmin : 0 tnvmcap : 0 [127:0] : 0 Total NVM Capacity (TNVMCAP) unvmcap : 0 [127:0] : 0 Unallocated NVM Capacity (UNVMCAP) rpmbs : 0 [31:24]: 0 Access Size [23:16]: 0 Total Size [5:3] : 0 Authentication Method [2:0] : 0 Number of RPMB Units edstt : 25 dsto : 1 fwug : 0 kas : 0 hctma : 0x1 [0:0] : 0x1 Host Controlled Thermal Management Supported mntmt : 273 [15:0] : 0 °C (273 K) Minimum Thermal Management Temperature (MNTMT) mxtmt : 357 [15:0] : 84 °C (357 K) Maximum Thermal Management Temperature (MXTMT) sanicap : 0x3 [31:30] : 0 Additional media modification after sanitize operation completes successfully is not defined [29:29] : 0 No-Deallocate After Sanitize bit in Sanitize command Supported [2:2] : 0 Overwrite Sanitize Operation Not Supported [1:1] : 0x1 Block Erase Sanitize Operation Supported [0:0] : 0x1 Crypto Erase Sanitize Operation Supported hmminds : 0 hmmaxd : 0 nsetidmax : 0 endgidmax : 0 anatt : 0 anacap : 0 [7:7] : 0 Non-zero group ID Not Supported [6:6] : 0 Group ID does change [4:4] : 0 ANA Change state Not Supported [3:3] : 0 ANA Persistent Loss state Not Supported [2:2] : 0 ANA Inaccessible state Not Supported [1:1] : 0 ANA Non-optimized state Not Supported [0:0] : 0 ANA Optimized state Not Supported anagrpmax : 0 nanagrpid : 0 pels : 0 domainid : 0 megcap : 0 sqes : 0x66 [7:4] : 0x6 Max SQ Entry Size (64) [3:0] : 0x6 Min SQ Entry Size (64) cqes : 0x44 [7:4] : 0x4 Max CQ Entry Size (16) [3:0] : 0x4 Min CQ Entry Size (16) maxcmd : 0 nn : 1 oncs : 0x5e [8:8] : 0 Copy Not Supported [7:7] : 0 Verify Not Supported [6:6] : 0x1 Timestamp Supported [5:5] : 0 Reservations Not Supported [4:4] : 0x1 Save and Select Supported [3:3] : 0x1 Write Zeroes Supported [2:2] : 0x1 Data Set Management Supported [1:1] : 0x1 Write Uncorrectable Supported [0:0] : 0 Compare Not Supported fuses : 0 [0:0] : 0 Fused Compare and Write Not Supported fna : 0x4 [3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported [2:2] : 0x1 Crypto Erase Supported as part of Secure Erase [1:1] : 0 Crypto Erase Applies to Single Namespace(s) [0:0] : 0 Format Applies to Single Namespace(s) vwc : 0x1 [2:1] : 0 Support for the NSID field set to FFFFFFFFh is not indicated [0:0] : 0x1 Volatile Write Cache Present awun : 0 awupf : 0 icsvscc : 1 [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format nwpc : 0 [2:2] : 0 Permanent Write Protect Not Supported [1:1] : 0 Write Protect Until Power Supply Not Supported [0:0] : 0 No Write Protect and Write Protect Namespace Not Supported acwu : 0 ocfs : 0 [1:1] : 0 Controller Copy Format 1h Not Supported [0:0] : 0 Controller Copy Format 0h Not Supported sgls : 0 [15:8] : 0 SGL Descriptor Threshold [1:0] : 0 Scatter-Gather Lists Not Supported mnan : 0 maxdna : 0 maxcna : 0 oaqd : 0 subnqn : ioccsz : 0 iorcsz : 0 icdoff : 0 fcatt : 0 [0:0] : 0 Dynamic Controller Model msdbd : 0 ofcs : 0 [0:0] : 0 Disconnect command Not Supported ps 0 : mp:6.3000W operational enlat:5 exlat:5 rrt:0 rrl:0 rwt:0 rwl:0 idle_power:- active_power:- active_power_workload:- ps 1 : mp:2.4000W operational enlat:30 exlat:30 rrt:1 rrl:1 rwt:1 rwl:1 idle_power:- active_power:- active_power_workload:- ps 2 : mp:1.9000W operational enlat:100 exlat:100 rrt:2 rrl:2 rwt:2 rwl:2 idle_power:- active_power:- active_power_workload:- ps 3 : mp:0.0500W non-operational enlat:1000 exlat:1000 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- active_power_workload:- ps 4 : mp:0.0040W non-operational enlat:1000 exlat:9000 rrt:3 rrl:3 rwt:3 rwl:3 idle_power:- active_power:- active_power_workload:- ```Backtrace without breakpoints
Click to expand
``` (gdb) r Starting program: /usr/lib/udisks2/udisksd [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". udisks-Message: 22:19:32.583: udisks daemon version 2.10.1 starting [New Thread 0x7ffff6dff6c0 (LWP 2360)] [New Thread 0x7ffff65fe6c0 (LWP 2361)] [New Thread 0x7ffff5dfd6c0 (LWP 2362)] [New Thread 0x7ffff55fc6c0 (LWP 2363)] [New Thread 0x7fffe7fff6c0 (LWP 2364)] *** stack smashing detected ***: terminated Thread 1 "udisksd" received signal SIGABRT, Aborted. __pthread_kill_implementation (threadid=EDIT:
The relevant function call is bd_nvme_find_ctrls_for_ns(). Disabling it greatly reduces the amount of crashes. The remaining ones might point to another issue.
Looking at libblockdev, the crash takes place while calling nvme_scan(). libnvme calls nvme_ns_identify_descs().
Backtrace with breakpoint at nvme_ns_identify_descs()
Click to expand
``` Starting program: /usr/lib/udisks2/udisksd [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". udisks-Message: 14:45:41.965: udisks daemon version 2.10.1 starting [New Thread 0x7ffff6dff6c0 (LWP 6211)] [New Thread 0x7ffff65fe6c0 (LWP 6212)] [New Thread 0x7ffff5dfd6c0 (LWP 6213)] [New Thread 0x7ffff55fc6c0 (LWP 6214)] [New Thread 0x7fffe7fff6c0 (LWP 6215)] Thread 1 "udisksd" hit Breakpoint 1, nvme_ns_identify_descs (n=n@entry=0x555555655420, descs=descs@entry=0x7fffffffaad0) at ../src/nvme/tree.c:2205 2205 { (gdb) n 2207 return nvme_identify_ns_descs(nvme_ns_get_fd(n), nvme_ns_get_nsid(n), descs); (gdb) n *** stack smashing detected ***: terminated Thread 1 "udisksd" received signal SIGABRT, Aborted. __pthread_kill_implementation (threadid=Backtrace with breakpoint at nvme_submit_passthru()
Click to expand
``` Starting program: /usr/lib/udisks2/udisksd [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". udisks-Message: 21:32:58.928: udisks daemon version 2.11.0 starting [New Thread 0x7ffff6dff6c0 (LWP 105927)] [New Thread 0x7ffff65fe6c0 (LWP 105928)] [New Thread 0x7ffff5dfd6c0 (LWP 105929)] [New Thread 0x7ffff55fc6c0 (LWP 105930)] [New Thread 0x7fffe7fff6c0 (LWP 105931)] ... (gdb) c Continuing. Thread 1 "udisksd" hit Breakpoint 1, nvme_submit_passthru (fd=14, ioctl_cmd=ioctl_cmd@entry=3225964097, cmd=cmd@entry=0x7fffffff9a00, result=0x0) at ../src/nvme/ioctl.c:139 139 err = ioctl(fd, ioctl_cmd, cmd); (gdb) info stack #0 nvme_submit_passthru (fd=14, ioctl_cmd=ioctl_cmd@entry=3225964097, cmd=cmd@entry=0x7fffffff9a00, result=0x0) at ../src/nvme/ioctl.c:139 #1 0x00007ffff475fcec in nvme_submit_admin_passthru (fd=