btrfs-convert: corrupt leaf, invalid previous key objectid, failed to copy ext2 inode

[0e4ef622]

Attempting to convert a 4.5 TB ext4 partition resulted in the following:

$ sudo btrfs-convert -p /dev/sdb3
btrfs-convert from btrfs-progs v6.6.3

Source filesystem:
  Type:           ext2
  Label:
  Blocksize:      4096
  UUID:           cd33e309-7ab5-46b3-b9b8-6264d1927d40
Target filesystem:
  Label:
  Blocksize:      4096
  Nodesize:       16384
  UUID:           66dd991f-6297-4dbd-905b-df3ee97b91e7
  Checksum:       crc32c
  Features:       extref, skinny-metadata, no-holes, free-space-tree (default)
    Data csum:    yes
    Inline data:  yes
    Copy xattr:   yes
Reported stats:
  Total space:    5000488026112
  Free space:     2699396538368 (53.98%)
  Inode count:       152604672
  Free inodes:       145776053
  Block count:      1220822272
Create initial btrfs filesystem
Create ext2 image file
Create btrfs metadata
corrupt leaf: root=5 block=5001928998912 slot=1 ino=89911763, invalid previous key objectid, have 89911762 expect 89911763
ERROR: failed to copy ext2 inode 89911320: -5
ERROR: error during copy_inodes -5
WARNING: error during conversion, the original filesystem is not modified

btrfs-progs 6.6.3

Linux 6.6.10-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 05 Jan 2024 16:20:41 +0000 x86_64 GNU/Linux

[ErrorNoInternet]

Does fsck -f on the ext4 filesystem return any errors?

[0e4ef622]

No

[0e4ef622]

Full fsck output:

$ sudo fsck -fvy /dev/sdb3
fsck from util-linux 2.39.3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

     6828622 inodes used (4.47%, out of 152604672)
     1063143 non-contiguous files (3.0%)
       12737 non-contiguous directories (0.2%)
             # of inodes with ind/dind/tind blocks: 1060017/18379/2
   207813163 blocks used (17.02%, out of 1220822272)
           0 bad blocks
          11 large files

     6374868 regular files
      441693 directories
           0 character device files
           0 block device files
          23 fifos
           0 links
       12020 symbolic links (9358 fast symbolic links)
           9 sockets
------------
     6828613 files

[adam900710]

Mind to run this branch and re-try the convert? https://github.com/adam900710/btrfs-progs/tree/convert_debug

EDIT: After more digging, I believe the root cause is the sequence when we insert inode item and its dir items/file extents/xattrs. Thus please see if this branch fixes your problem.

[0e4ef622]

I get the same error, although the inode is different, not sure if that means anything

$ sudo btrfs-convert -p /dev/sdb3
btrfs-convert from btrfs-progs v6.6.3

Source filesystem:
  Type:           ext2
  Label:
  Blocksize:      4096
  UUID:           cd33e309-7ab5-46b3-b9b8-6264d1927d40
Target filesystem:
  Label:
  Blocksize:      4096
  Nodesize:       16384
  UUID:           d45ad84c-7713-41cd-a67a-730689286e77
  Checksum:       crc32c
  Features:       extref, skinny-metadata, no-holes, free-space-tree (default)
    Data csum:    yes
    Inline data:  yes
    Copy xattr:   yes
Reported stats:
  Total space:    5000488026112
  Free space:     2699396538368 (53.98%)
  Inode count:       152604672
  Free inodes:       145776050
  Block count:      1220822272
Create initial btrfs filesystem
Create ext2 image file
Create btrfs metadata
corrupt leaf: root=5 block=5001931145216 slot=1 ino=89911763, invalid previous key objectid, have 89911762 expect 89911763
ERROR: failed to copy ext2 inode 89911509: -5
ERROR: error during copy_inodes -5
WARNING: error during conversion, the original filesystem is not modified

[adam900710]

My bad, missing the debug patch.

Mind to try the branch again? This time it should dump the offending tree block.

[0e4ef622]

btrfs-convert-output.txt

[adam900710]

Got it. There is a missing location which can lead to the problem. When copying a directory, we copy all the child entries, along with a INODE_REF. The INODE_REF would not have a INODE_ITEM, thus leading to the problem.

Fix the hole by inserting a dummy inode item, and let later iteration on the child inode to properly update the inode item.

The same branch is updated, mind to give it another (hopefully final) try?

[0e4ef622]

No luck, but different a error this time.

corrupt leaf: root=5 block=5001935601664 slot=0 ino=89911630, invalid mode: has 00 expect valid S_IF* bit(s)
leaf 5001935601664 items 120 free space 23 generation 99 owner FS_TREE
leaf 5001935601664 flags 0x1(WRITTEN) backref revision 1

btrfs-convert-output.txt

[adam900710]

OK, it's the problem of the dummy inode item.

I'll add a way to enable tree-checker for cached tree blocks, so that I can emulate the problem locally, then finally fix it forever.

[adam900710]

Now I can reproduce it locally just using the selftests of btrfs-progs.

And I hope this is the last time you need to try the update convert_debug branch.

[0e4ef622]

The conversion succeeded! Thanks for the help!

[adam900710]

Just don't run scrub on the converted fs, unless you have fully balanced all data chunks.

There is a known bug in the recently reworked scrub code, that if the length of a chunk is not 64K aligned, it can lead to false alerts or even crash during scrub.

We have pinned down the cause and the fix would be backported soon.