robn
commented
5 months ago The kernel code has a nice comment on it:
/* In case the discard granularity isn't set by buggy device driver */
if (WARN_ON_ONCE(!bdev_discard_granularity(bdev))) {
pr_err_ratelimited("%pg: Error: discard_granularity is 0.\n",
bdev);
return -EOPNOTSUPP;
}Given the hit-and-miss nature of USB adapters, my money is on firmware or driver.
That said, it looks like OpenZFS could be handling this better, by not enabling/issuing TRIM if it can detect this case, and/or by handling TRIM error responses better. If I get some spare time this week I'll have a look at it.
multi
System information
Describe the problem you're observing
Kernel warning from
__blkdev_issue_discardIt was triggered while syncing snapshots with
syncoidFrom dmesg
[30860.067492] sdg1: Error: discard_granularity is 0.sdgis a 2TB nvme (Seagate ZP2000CV30002) plugged into nvme2usb adapter (RTL9210B chipset, flashed with firmware version1.32.87.082923), used for theusbnvmepool :) With discard support enabled, by using the following udev rule (why udev, see here):Attaching more details below.
Not sure, about the root cause of all - is it a buggy RTL9210B firmware or a kernel/zfs issue?
Describe how to reproduce the problem
Not sure :/ I rebooted after sawing the load of the machine and the error in dmesg. Pulled
zfs-2.2.99-412_g99741bde59and rebooted again. Issued a scrub onusbnvme, which finished without errors. After that a trim, which also showed no sign of errors100% trimmed, nor kernel errors. Then, I've decided to try running againsyncoidwhich started with the following:and finished without errors...
Include any warning/errors/backtraces from the system logs
stack trace dmesg
``` [30860.066588] ------------[ cut here ]------------ [30860.066600] WARNING: CPU: 2 PID: 558845 at block/blk-lib.c:50 __blkdev_issue_discard+0x14a/0x190 [30860.066608] Modules linked in: uas ts_bm xt_LOG nf_log_syslog xt_string xt_tcpudp nft_chain_nat xt_MASQUERADE nf_nat nct6775 nft_compat nct6775_core hwmon_vid nf_tables drivetemp intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul i915 polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel snd_hda_intel snd_intel_dspcfg vfat snd_intel_sdw_acpi sha512_ssse3 fat sha256_ssse3 sha1_ssse3 at24 iTCO_wdt drm_buddy i2c_algo_bit intel_pmc_bxt ttm iTCO_vendor_support mei_hdcp snd_hda_codec mei_pxp dm_crypt rapl intel_cstate snd_hda_core drm_display_helper snd_hwdep snd_pcm option intel_uncore cec i2c_i801 snd_timer mxm_wmi pcspkr i2c_smbus rc_core mei_me lpc_ich usb_wwan aesni_intel snd intel_gtt soundcore crypto_simd mei cryptd cbc joydev encrypted_keys trusted mousedev asn1_encoder input_leds tee evdev mac_hid nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crc32c_generic crc32c_intel fuse loop dm_mod nfnetlink [30860.066666] dmi_sysfs ip_tables x_tables wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 libchacha poly1305_x86_64 ip6_udp_tunnel udp_tunnel usb_storage hid_logitech_hidpp raid1 hid_logitech_dj md_mod usbhid nvme nvme_core xhci_pci nvme_auth xhci_pci_renesas video wmi zfs(POE) spl(OE) r8168(OE) tg3 libphy [30860.066686] CPU: 2 PID: 558845 Comm: z_trim_iss Tainted: P OE T 6.7.11-hardened1-1-hardened #1 [30860.066688] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Extreme4, BIOS P2.90P 12/17/2015 [30860.066689] RIP: 0010:__blkdev_issue_discard+0x14a/0x190 [30860.066692] Code: 48 39 d8 75 90 f7 da 89 d3 c1 eb 09 eb 8a 48 8b 04 24 4c 89 20 48 83 c4 08 31 c0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc <0f> 0b 48 c7 c6 90 90 0c 94 48 c7 c7 e0 77 93 94 e8 21 91 7b 00 85 [30860.066694] RSP: 0018:ffffa50fa5b53cd0 EFLAGS: 00010246 [30860.066696] RAX: ffff99d5dddb9e40 RBX: ffff99d681e264a0 RCX: ffff99d400485100 [30860.066697] RDX: 0000000000000000 RSI: 000000004c042110 RDI: ffff99d70b84e540 [30860.066698] RBP: 0000000000000c40 R08: ffffa50fa5b53d10 R09: 0000000000000000 [30860.066699] R10: ffff99d4bafbc028 R11: ffff99d681e264a0 R12: 0000000000000000 [30860.066700] R13: 0000000000000040 R14: 0000000000000000 R15: ffff99d70b84e540 [30860.066702] FS: 0000000000000000(0000) GS:ffff99d70f300000(0000) knlGS:0000000000000000 [30860.066703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30860.066704] CR2: 000000f19a9c4018 CR3: 00000001b0108004 CR4: 00000000000606f0 [30860.066706] Call Trace: [30860.066708]usb device dmesg
``` [ 795.878231] usb 4-1: new SuperSpeed USB device number 2 using xhci_hcd [ 795.911243] usb 4-1: New USB device found, idVendor=0bda, idProduct=9210, bcdDevice=f0.01 [ 795.911253] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 795.911256] usb 4-1: Product: RTL9210B-CG [ 795.911259] usb 4-1: Manufacturer: Realtek [ 795.911261] usb 4-1: SerialNumber: 012345678932 [ 795.939524] usb 4-1: Enable of device-initiated U1 failed. [ 795.940026] usb 4-1: Enable of device-initiated U2 failed. [ 795.970070] usb 4-1: Enable of device-initiated U1 failed. [ 795.970710] usb 4-1: Enable of device-initiated U2 failed. [ 795.972684] scsi host13: uas [ 796.419421] scsi 13:0:0:0: Direct-Access Realtek RTL9210B-CG 1.00 PQ: 0 ANSI: 6 [ 796.459184] sd 13:0:0:0: [sdg] 488378646 4096-byte logical blocks: (2.00 TB/1.82 TiB) [ 796.460603] sd 13:0:0:0: [sdg] Write Protect is off [ 796.460608] sd 13:0:0:0: [sdg] Mode Sense: 37 00 00 08 [ 796.463754] sd 13:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 796.464938] sd 13:0:0:0: [sdg] Preferred minimum I/O size 4096 bytes [ 796.464944] sd 13:0:0:0: [sdg] Optimal transfer size 268431360 bytes [ 796.480698] sdg: sdg1 sdg9 [ 796.481050] sd 13:0:0:0: [sdg] Attached SCSI disk ```lsusb.py -ciu
``` usb4 1d6b:0003 09 1IF [USB 3.00, 5000 Mbps, 0mA] (xhci-hcd 0000:00:14.0) hub 4-1 0bda:9210 00 1IF [USB 3.20, 5000 Mbps, 896mA] (Realtek RTL9210B-CG 012345678932) 4-1:1.0 (IF) 08:06:62 4EPs (Mass Storage) uas host13 (sdg) ```lsblk -Df
``` NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO FSTYPE FSVER sdg 0 4K 32G 0 ├─sdg1 0 4K 32G 0 zfs_member 5000 └─sdg9 0 4K 32G 0 ```zpool get all usbnvme | grep local
``` usbnvme ashift 12 local usbnvme autotrim on local usbnvme feature@async_destroy enabled local usbnvme feature@empty_bpobj active local usbnvme feature@lz4_compress active local usbnvme feature@multi_vdev_crash_dump enabled local usbnvme feature@spacemap_histogram active local usbnvme feature@enabled_txg active local usbnvme feature@hole_birth active local usbnvme feature@extensible_dataset active local usbnvme feature@embedded_data active local usbnvme feature@bookmarks enabled local usbnvme feature@filesystem_limits enabled local usbnvme feature@large_blocks enabled local usbnvme feature@large_dnode active local usbnvme feature@sha512 enabled local usbnvme feature@skein enabled local usbnvme feature@edonr enabled local usbnvme feature@userobj_accounting active local usbnvme feature@encryption active local usbnvme feature@project_quota active local usbnvme feature@device_removal enabled local usbnvme feature@obsolete_counts enabled local usbnvme feature@zpool_checkpoint enabled local usbnvme feature@spacemap_v2 active local usbnvme feature@allocation_classes enabled local usbnvme feature@resilver_defer enabled local usbnvme feature@bookmark_v2 enabled local usbnvme feature@redaction_bookmarks enabled local usbnvme feature@redacted_datasets enabled local usbnvme feature@bookmark_written enabled local usbnvme feature@log_spacemap active local usbnvme feature@livelist enabled local usbnvme feature@device_rebuild enabled local usbnvme feature@zstd_compress enabled local usbnvme feature@draid enabled local usbnvme feature@zilsaxattr enabled local usbnvme feature@head_errlog active local usbnvme feature@blake3 enabled local usbnvme feature@block_cloning enabled local usbnvme feature@vdev_zaps_v2 active local usbnvme feature@redaction_list_spill enabled local usbnvme feature@raidz_expansion enabled local ```