search

maharmstone / btrfs

WinBtrfs - an open-source btrfs driver for Windows
GNU Lesser General Public License v3.0
5.8k stars 222 forks source link

Metadata checksum error #524

Open 13xforever opened 2 years ago

13xforever commented 2 years ago

During weekly check up on winbtrfs-owned filesystem in linux found some issues today.

Using WinBtrfs 1.8.1 on Win11 22H2, linux btrfs-progs 5.18 on Fedora with 5.19.8 kernel. Really not sure when it could've happened (though I did have one full-system lock up the other day). Googling around didn't produce anything useful, so I'm not sure if I can ignore this. Running scrub on Windows results in BSoD.

Noticed some corruption events in dmesg first

[  162.052758] BTRFS info (device dm-1): disk space caching is enabled
[  162.052761] BTRFS info (device dm-1): has skinny extents
[  162.544690] BTRFS info (device dm-1): bdev /dev/mapper/ArchiMedia errs: wr 0, rd 0, flush 0, corrupt 134, gen 0
[  162.544728] BTRFS info (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 21, gen 0
[  185.592207] BTRFS info (device dm-1): checking UUID tree

running btrfs check hangs on fs roots:

Opening filesystem to check...
Checking filesystem on /dev/dm-1
UUID: 6e222b94-0fa2-237d-284c-176e55472e30
[1/7] checking root items
[2/7] checking extents
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[3/7] checking free space cache
[4/7] checking fs roots
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6

running scrub

[ 6663.817055] BTRFS info (device dm-1): scrub: started on devid 1
[ 6663.818097] BTRFS info (device dm-1): scrub: started on devid 2
[ 6683.992056] BTRFS warning (device dm-1): checksum error at logical 5962932944896 on dev /dev/mapper/AM-2, physical 1498415104: metadata leaf (level 0) in tree 7
[ 6683.992079] BTRFS warning (device dm-1): checksum error at logical 5962932944896 on dev /dev/mapper/AM-2, physical 1498415104: metadata leaf (level 0) in tree 7
[ 6683.992092] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 22, gen 0
[ 6683.992254] BTRFS warning (device dm-1): checksum error at logical 5962932961280 on dev /dev/mapper/AM-2, physical 1498431488: metadata leaf (level 0) in tree 7
[ 6683.992263] BTRFS warning (device dm-1): checksum error at logical 5962932961280 on dev /dev/mapper/AM-2, physical 1498431488: metadata leaf (level 0) in tree 7
[ 6683.992270] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 23, gen 0
[ 6683.992477] BTRFS warning (device dm-1): checksum error at logical 5962932977664 on dev /dev/mapper/AM-2, physical 1498447872: metadata leaf (level 0) in tree 7
[ 6683.992485] BTRFS warning (device dm-1): checksum error at logical 5962932977664 on dev /dev/mapper/AM-2, physical 1498447872: metadata leaf (level 0) in tree 7
[ 6683.992492] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 24, gen 0
[ 6683.992647] BTRFS warning (device dm-1): checksum error at logical 5962932994048 on dev /dev/mapper/AM-2, physical 1498464256: metadata leaf (level 0) in tree 7
[ 6683.992653] BTRFS warning (device dm-1): checksum error at logical 5962932994048 on dev /dev/mapper/AM-2, physical 1498464256: metadata leaf (level 0) in tree 7
[ 6683.992663] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 25, gen 0
[ 6683.997006] BTRFS warning (device dm-1): checksum error at logical 5962933010432 on dev /dev/mapper/AM-2, physical 1498480640: metadata leaf (level 0) in tree 7
[ 6683.997028] BTRFS warning (device dm-1): checksum error at logical 5962933010432 on dev /dev/mapper/AM-2, physical 1498480640: metadata leaf (level 0) in tree 7
[ 6683.997041] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 26, gen 0
[ 6702.992865] BTRFS warning (device dm-1): checksum error at logical 5964006686720 on dev /dev/mapper/AM-2, physical 2572156928: metadata leaf (level 0) in tree 7
[ 6702.992880] BTRFS warning (device dm-1): checksum error at logical 5964006686720 on dev /dev/mapper/AM-2, physical 2572156928: metadata leaf (level 0) in tree 7
[ 6702.992888] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 27, gen 0
[ 6702.992985] BTRFS warning (device dm-1): checksum error at logical 5964006703104 on dev /dev/mapper/AM-2, physical 2572173312: metadata leaf (level 0) in tree 7
[ 6702.992990] BTRFS warning (device dm-1): checksum error at logical 5964006703104 on dev /dev/mapper/AM-2, physical 2572173312: metadata leaf (level 0) in tree 7
[ 6702.992994] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 28, gen 0
[ 6702.993083] BTRFS warning (device dm-1): checksum error at logical 5964006719488 on dev /dev/mapper/AM-2, physical 2572189696: metadata leaf (level 0) in tree 7
[ 6702.993086] BTRFS warning (device dm-1): checksum error at logical 5964006719488 on dev /dev/mapper/AM-2, physical 2572189696: metadata leaf (level 0) in tree 7
[ 6702.993090] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 29, gen 0
[ 6702.993177] BTRFS warning (device dm-1): checksum error at logical 5964006735872 on dev /dev/mapper/AM-2, physical 2572206080: metadata leaf (level 0) in tree 7
[ 6702.993181] BTRFS warning (device dm-1): checksum error at logical 5964006735872 on dev /dev/mapper/AM-2, physical 2572206080: metadata leaf (level 0) in tree 7
[ 6702.993185] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 30, gen 0
[ 6721.512615] BTRFS warning (device dm-1): checksum error at logical 369098752 on dev /dev/mapper/AM-2, physical 3645898752: metadata leaf (level 0) in tree 7
[ 6721.512627] BTRFS warning (device dm-1): checksum error at logical 369098752 on dev /dev/mapper/AM-2, physical 3645898752: metadata leaf (level 0) in tree 7
[ 6721.512633] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 31, gen 0
[ 6721.512703] BTRFS warning (device dm-1): checksum error at logical 369115136 on dev /dev/mapper/AM-2, physical 3645915136: metadata leaf (level 0) in tree 7
[ 6721.512707] BTRFS warning (device dm-1): checksum error at logical 369115136 on dev /dev/mapper/AM-2, physical 3645915136: metadata leaf (level 0) in tree 7
[ 6721.512709] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 32, gen 0
[ 6721.512775] BTRFS warning (device dm-1): checksum error at logical 369131520 on dev /dev/mapper/AM-2, physical 3645931520: metadata leaf (level 0) in tree 7
[ 6721.512777] BTRFS warning (device dm-1): checksum error at logical 369131520 on dev /dev/mapper/AM-2, physical 3645931520: metadata leaf (level 0) in tree 7
[ 6721.512780] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 33, gen 0
[ 6721.516630] BTRFS warning (device dm-1): checksum error at logical 369147904 on dev /dev/mapper/AM-2, physical 3645947904: metadata leaf (level 0) in tree 7
[ 6721.516655] BTRFS warning (device dm-1): checksum error at logical 369147904 on dev /dev/mapper/AM-2, physical 3645947904: metadata leaf (level 0) in tree 7
[ 6721.516668] BTRFS error (device dm-1): bdev /dev/mapper/AM-2 errs: wr 29, rd 0, flush 0, corrupt 34, gen 0
[ 6771.128355] BTRFS error (device dm-1): bad tree block start, want 5962933010432 have 0
[ 6771.157972] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933010432 (dev /dev/mapper/AM-2 sector 2926720)
[ 6771.157979] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933014528 (dev /dev/mapper/AM-2 sector 2926728)
[ 6771.159093] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933018624 (dev /dev/mapper/AM-2 sector 2926736)
[ 6771.159422] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933022720 (dev /dev/mapper/AM-2 sector 2926744)
[ 6913.335267] BTRFS warning (device dm-1): checksum error at logical 43113840640 on dev /dev/mapper/ArchiMedia, physical 44184436736, root 5, inode 4981, offset 131072, length 4096, links 1 (path: Archive/Develop/SDKs/Mobile/Android/android-sdk-windows/docs/reference/android/widget/MediaController.html)
[ 6913.335277] BTRFS error (device dm-1): bdev /dev/mapper/ArchiMedia errs: wr 0, rd 0, flush 0, corrupt 135, gen 0
[ 7229.380941] BTRFS error (device dm-1): bad tree block start, want 5962932994048 have 0
[ 7229.425140] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932994048 (dev /dev/mapper/AM-2 sector 2926688)
[ 7229.425160] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932998144 (dev /dev/mapper/AM-2 sector 2926696)
[ 7229.425170] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933002240 (dev /dev/mapper/AM-2 sector 2926704)
[ 7229.425176] BTRFS info (device dm-1): read error corrected: ino 0 off 5962933006336 (dev /dev/mapper/AM-2 sector 2926712)
[ 7241.946885] BTRFS error (device dm-1): bad tree block start, want 5962932944896 have 0
[ 7241.982749] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932944896 (dev /dev/mapper/AM-2 sector 2926592)
[ 7241.982775] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932948992 (dev /dev/mapper/AM-2 sector 2926600)
[ 7241.982788] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932953088 (dev /dev/mapper/AM-2 sector 2926608)
[ 7241.982797] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932957184 (dev /dev/mapper/AM-2 sector 2926616)
[ 7314.107076] BTRFS error (device dm-1): bad tree block start, want 5962932961280 have 0
[ 7314.131362] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932961280 (dev /dev/mapper/AM-2 sector 2926624)
[ 7314.131386] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932965376 (dev /dev/mapper/AM-2 sector 2926632)
[ 7314.131396] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932969472 (dev /dev/mapper/AM-2 sector 2926640)
[ 7314.131403] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932973568 (dev /dev/mapper/AM-2 sector 2926648)
[ 7353.926345] BTRFS error (device dm-1): bad tree block start, want 5962932977664 have 0
[ 7353.964151] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932977664 (dev /dev/mapper/AM-2 sector 2926656)
[ 7353.964169] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932981760 (dev /dev/mapper/AM-2 sector 2926664)
[ 7353.964178] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932985856 (dev /dev/mapper/AM-2 sector 2926672)
[ 7353.964185] BTRFS info (device dm-1): read error corrected: ino 0 off 5962932989952 (dev /dev/mapper/AM-2 sector 2926680)

basic info

[root@ls-fedora ArchiMedia]# btrfs filesystem usage .
Overall:
    Device size:          27.28TiB
    Device allocated:         12.02TiB
    Device unallocated:       15.27TiB
    Device missing:          0.00B
    Used:             11.84TiB
    Free (estimated):         15.44TiB  (min: 7.80TiB)
    Free (statfs, df):        15.44TiB
    Data ratio:               1.00
    Metadata ratio:           2.00
    Global reserve:      512.00MiB  (used: 0.00B)
    Multiple profiles:              no

Data,single: Size:11.82TiB, Used:11.65TiB (98.57%)
   /dev/mapper/ArchiMedia     11.82TiB

Metadata,RAID1: Size:102.00GiB, Used:100.80GiB (98.82%)
   /dev/mapper/ArchiMedia    102.00GiB
   /dev/mapper/AM-2  102.00GiB

System,RAID1: Size:32.00MiB, Used:1.30MiB (4.05%)
   /dev/mapper/ArchiMedia     32.00MiB
   /dev/mapper/AM-2   32.00MiB

Unallocated:
   /dev/mapper/ArchiMedia      4.45TiB
   /dev/mapper/AM-2   10.81TiB
maharmstone commented 2 years ago

Thanks. You're using device-mapper rather than raw block devices - do you know what /dev/mapper/ArchiMedia and /dev/mapper/AM-2 map to?

13xforever commented 2 years ago

It's still my two disks with bitlocker volumes

[root@ls-fedora ArchiMedia]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root       7 Sep 18 21:11 AM-2 -> ../dm-2
lrwxrwxrwx. 1 root root       7 Sep 18 21:12 ArchiMedia -> ../dm-1
crw-------. 1 root root 10, 236 Sep 18 21:09 control
lrwxrwxrwx. 1 root root       7 Sep 18 21:09 luks-d98d14c3-4c92-43fa-8517-4a06d5f027cc -> ../dm-0

[root@ls-fedora ArchiMedia]# cryptsetup status /dev/dm-1
/dev/dm-1 is active and is in use.
  type:    BITLK
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sda2
  sector size:  512
  offset:  16 sectors
  size:    35156619264 sectors
  skipped: 16 sectors
  mode:    read/write
[root@ls-fedora ArchiMedia]# cryptsetup status /dev/dm-2
/dev/dm-2 is active and is in use.
  type:    BITLK
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sdb2
  sector size:  512
  offset:  16 sectors
  size:    23437733888 sectors
  skipped: 16 sectors
  mode:    read/write
[root@ls-fedora ArchiMedia]#
13xforever commented 2 years ago

results of scrub

UUID:             6e222b94-0fa2-237d-284c-176e55472e30
Scrub started:    Sun Sep 18 16:13:42 2022
Status:           finished
Duration:         20:24:58
Total to scrub:   11.91TiB
Rate:             182.72MiB/s
Error summary:    csum=1063
  Corrected:      13
  Uncorrectable:  1050
  Unverified:     0

about 20 uncorrectable errors were in 6 files (that I recovered from backups), but other errors had no additional information what they were

photo_2022-09-18_20-11-05

Interesting that btrfs check gives the same 13 checksum errors for roots.

13xforever commented 2 years ago

full btrfs check result

[root@ls-fedora ~]# btrfs check -p /dev/mapper/ArchiMedia 
Opening filesystem to check...
Checking filesystem on /dev/mapper/ArchiMedia
UUID: 6e222b94-0fa2-237d-284c-176e55472e30
[1/7] checking root items                      (0:06:50 elapsed, 17461366 items checked)
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[2/7] checking extents                         (0:17:42 elapsed, 6613880 items checked)
[3/7] checking free space cache                (0:06:57 elapsed, 12204 items checked)
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[4/7] checking fs roots                        (1:18:02 elapsed, 129168 items checked)
checksum verify failed on 5962933010432 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x929dbce6274a6515cb28dcc6e1a5ee75bfdd7247c15ca701b2d272c358d1c544
checksum verify failed on 5962932994048 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932944896 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932961280 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5962932977664 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006686720 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006703104 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006719488 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 5964006735872 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369098752 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369115136 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369147904 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
checksum verify failed on 369131520 wanted 0x0000000000000000000000000000000000000000000000000000000000000000 found 0x6290e6592047ec80140a1252d31a8ba8a33a34cf578b5c8c4a081706b9416fa6
[5/7] checking csums (without verifying data)  (1:10:29 elapsed, 25512445 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 4 items checked)
[7/7] checking quota groups skipped (not enabled on this FS)
found 12933797396480 bytes used, no error found
total csum bytes: 100175282368
total tree bytes: 108361662464
total fs tree bytes: 2123726848
total extent tree bytes: 1481572352
btree space waste bytes: 3398233945
file data blocks allocated: 15714799702016
 referenced 13476542222336
13xforever commented 2 years ago

Some things I've learnt so far:

  1. running scrub reveals that those 13 errors are in tree 7, which is a checksum tree in default filesystem layout (you can check with btrfs inspect-internal dump-tree -t 7 -r)
  2. running btrfs inspect-internal dump-tree -t 7 -b <logical_address> /dev/mapper/AM-2 (where logical_address is the one from btrfs check/dmesg output) gives the list of data extents associated with this checksum extent
  3. running btrfs inspect-internal logical-resolve <data_offset> /mnt/ArchiMedia/ (where data_offset is from the 2. output) gives you associated filenames

you can mount filesystem with the flag to ignore checksums and copy the data (it's probably fine), or delete and restore the files from your backup to fix this without rebuilding the checksum tree completely

aziztcf commented 1 year ago

Had a similar problem with loads of unrecoverable errors while scrubbing and errors during extent check done in Linux, for some reason WIndows didn't give me the memory.dmp file after the BSOD and I already gave up and formatted the drive. https://github.com/maharmstone/btrfs/issues/523#issuecomment-1275442587

Might be related to https://github.com/maharmstone/btrfs/issues/515 too, since I do remember having trouble scrubbing after that.

I do still have two btrfs partitions so if the issue happens again I'll take notes.

itsTyrion commented 1 year ago

Yeah same. I tried storing Fortnite on a BTRFS formatted SSD and had to repair game files at least 6 times over the last 2-3 months. I think the driver is not ready for daily driving it on a data or game drive yet.

Today, the game Valorant didn't open, I tried renaming it's folder back and forth so the launcher checks the files - which resulted in a "folder does not exist" error and it vanished in front of my eyes.

Disk passthrough to an Ubuntu VM for btrfs check and scrub revealed countless errors (will add a log later), scrub is currently sitting at 2753 csum errors. (Not finished yet)

S.M.A.R.T. values LGTM.

maharmstone commented 1 year ago

Either your disk is dying or you've got it mounted by both a VM and the host at the same time.

itsTyrion commented 1 year ago

SMART values look fine, disk self test passed. I set the disk to offline in diskmgmt and added it as hard drive in Hyper-V (if you don't, Hyper-V refuses to start the VM). Done that in the past with both NTFS and BTRFS drives, no issues with mount, btrfs-check or btrfs-scrub. As I said, a complete game folder just vanished when I tried to rename it O_o This part was already corrupted before, the game didn't launch.

Also I just saw that my log looks pretty different so this is probably the wrong issue

maharmstone commented 1 year ago

That's what's causing your corruption, then - marking a disk as offline won't unmount the btrfs volume, I don't think. If you're looking to share files with a VM, you'd be better off with virtiofs or a network share.

itsTyrion commented 1 year ago

AFAIK it does unmount the disk - it won't let me set it offline or start the VM when something is still accessing it. Vice versa, Windows won't let you set the disk to online while the VM is still accessing it (goes for Hyper-V and VMware Workstation) I checked with my Linux partition that I don't actively write to under Windows and there are zero csum errors coming up when passing it into hyper-v

I wasn't looking to share files, I was doing full disk passthrough to get access to btrfs-progs without rebooting

itsTyrion commented 1 year ago

I have now been daily-driving BTRFS on the same disk and same usage (games, general tasks, even Hyper-V VMs) WITHOUT compression enabled in the registry. Not only does read-write feel faster and more reliable, there have also been zero issues with data corruption. I'm taking a shot in the semi-dark and say that at least my issues could be compression-related (I was using zstd before).

maharmstone commented 1 year ago

Thanks @itsTyrion. Compression does seem to be a common factor among people reporting problems, but I've not been able to reproduce any particular issues yet.