Open ernsteiswuerfel opened 4 years ago
Thanks for the report, so the guesswork to create the filesystem with multiple mixed profiles does not work on your version 5.7 or it maybe depends on the underlying devices too. I'm testing that on 5.5 and netowrk backed disk and the test passes.
Same error on my PowerMac G4 too (kernel 5.7-rc7 and 5.6.14). Device on the G4 would be a SATA-SSD connected to a SiL3112. On the Pentium4 it is a HDD connected to internal IDE. G4_cli-tests-results.txt
Re-test with btrfs-progs v5.7 and kernel 5.8-rc5 on x86 (Pentium4, 32bit) too. Turns out exactly the same tests fail on ppc and x86, 32bit. So I will close the newer ppc-specific issue #262 in favour of this one as it's the older bug reported.
@ernsteiswuerfel I found a git commit that introduced this behavior change for fsck-test/037, and I'm now preparing a patch to submit to my peers. I believe this is a regression, but let's see how it goes when they review the patch.
Thanks again for submitting these reports, and for waiting for us to fix them :)
@adam900710 what do you think about https://github.com/marcosps/linux/commit/12084186851d519d0401060e683d1fb469e64880 ? it fixes fsck/037 for me. let me know if you agree with the change. If yes, I'll send to the ML :)
@marcosps You're right! It's my fault to cause the regression.
Great fix!
@marcosps You're right! It's my fault to cause the regression.
Great fix!
Thanks :) Patch sent to ML
@ernsteiswuerfel can you please retry convert/017 again with the current devel branch? There were some changes regarding convert, specially 38f220d6eea51344603e5a213c21500ec671b8b8, which can help us to detect if the problem happened in the kernel side or in the convert tool itself.
At least it didn't fail for me using a more recent devel branch.
@ernsteiswuerfel same goes for cli-test/014, there are some patches in devel that should address the multiple profiles detection. Can you please test again? Thanks in advance for your work!
For me these tests still fail unfortunately. Did run the tests specifically, kernel 5.8.3 on the G4. Built btrfs-progs from devel ce6e0f572f44c46e57e3c19bdd9105e2ab2bf296
.
fsck-tests-results.txt
convert-tests-results.txt
cli-tests-results.txt
For me these tests still fail unfortunately. Did run the tests specifically, kernel 5.8.3 on the G4. Built btrfs-progs from devel
ce6e0f572f44c46e57e3c19bdd9105e2ab2bf296
. fsck-tests-results.txt
This one is indeed prone to fail since the kernel patch is missing. Would you like to test the latest version of the patch and then reexecute btrfsprogs tests on it? Also, if you do, you could add a tested-by tag in the kernel patch: https://lore.kernel.org/linux-btrfs/20200821145444.25791-1-marcos@mpdesouza.com/T/#u
convert-tests-results.txt The first thing that I can spot right by checking your report is that the same test on my machine reports different free space: free: 49676288 (4.63%) Which is 10Mb bigger than yours: free: 39124992 (3.64%)
@adam900710 do you have any idea if different archs should report different usage? The result for me is the same using v.5.8.3 or running misc-next, and it works.
cli-tests-results.txt In this case, the balance didn't manage to convert some of the single block groups to raid1. For me it relocated two chunks:
Done, had to relocate 2 out of 4 chunks
That explains why the test fails and will require more investigation on kernel side, as for me it works testing on misc-next and in v5.8.3 and it works =/
Applied 101ea93 on top of kernel 5.9-rc2 and fsck-tests/037-freespacetree-repair passes now on the G4. Thanks!
The first thing that I can spot right by checking your report is that the same test on my machine reports different free space: free: 49676288 (4.63%) Which is 10Mb bigger than yours: free: 39124992 (3.64%)
@adam900710 do you have any idea if different archs should report different usage? The result for me is the same using v.5.8.3 or running misc-next, and it works.
convert-tests/017-fs-near-full passes too now, but don't know whether it's the patch or 5.9-rc2 by itself. I could test this out when it is of some interest? Anyway now this test is working it also reports
====== RUN CHECK /root/build/btrfs-progs/btrfs-convert /root/build/btrfs-progs/tests/test.img
create btrfs filesystem:
blocksize: 4096
nodesize: 16384
features: extref, skinny-metadata (default)
checksum: crc32c
free space report:
total: 1073741824
free: 49676288 (4.63%)
on the G4. cli-tests/014-multiple-profiles-warning still fails.
@ernsteiswuerfel great. For the fsck/037, this is expected. Thanks for testing!
About convert/017, well, the issue was odd really. BNut, good that it passes. Report again if this fails.
i'll try to take some time to investigate this more carefully. Thanks for report!
btw, would you like to edit the bug with the only remaining test failing?
Just did a re-run of the full testsuite on top of 5.9-rc2 + patch. Strangely convert/017 failed again, run via ./convert-tests.sh but it passes when run as single test.
However I noticed 018-fs-size-overflow is failing now with convert/main.c:1161: do_convert: Assertion cctx.total_bytes failed, value 0
on the G4.
All the other tests pass or were correctly skipped. convert-tests-results.txt
Just did a re-run of the full testsuite on top of 5.9-rc2 + patch. Strangely convert/017 failed again, run via ./convert-tests.sh but it passes when run as single test.
However I noticed 018-fs-size-overflow is failing now with
convert/main.c:1161: do_convert: Assertion cctx.total_bytes failed, value 0
on the G4.All the other tests pass or were correctly skipped. convert-tests-results.txt
Thanks for the report! I fixed the issue in my local branch: https://github.com/marcosps/btrfs-progs/tree/mpdesouza_convert_018
This fix was sent to ML, so I expect a review soon. I was able to create a x86 32bit VM, and so I managed to reproduce this issue and also convert/017 test.
I hope to have a fix for convert/017 soon too.
@ernsteiswuerfel for convert/017 test I suspect that there is an issue in e2fsprogs w.r.t 32bit machines. I'm still investigating and already for help of some ext4 experts to see if this is the case. I should have an update soon.
After analyzing better and having a response from an ext4 specialist, it does seem to be an issue in the e2fsprogs. The ext[234] code can change the allocation start/end in a non-deterministic way, so it does not seems to be related with the real issue. I'll keep checking.
By "non-deterministic", we still need to ensure that, e2fsprogs would never report a used space as free, right?
If e2fsprogs just reports more space than it really used, that won't cause any real problem except the failed tests, and we only need to update that convert/017 test case.
By "non-deterministic", we still need to ensure that, e2fsprogs would never report a used space as free, right?
It seems not being the case as the used space is the same on a ext4 created on a i586 and x64 machine.
To update my current findings, as I didn't have time to check this again:
The test convert/017 creates a different allocation map when created on a x32 machine. In this particular case, there is some minor blocks being grouped, one block group using 10Mb, and another one with 50Kb IIRC.
Our convert code makes sure that a data chunk (the term used as a block group in btrfs) has at least 32Mb of size, so when it finds these two block groups it extents them to 32Mb. I still need to check whether this is strictly necessary or not.
This behavior makes btrfs reports more space being used (after all the calculations it shows 10Mb less of free space if we compare with the same ext4 image from an x64 machine), the btrfs-convert fails to find free space in the to be converted disk (test.img file in this case).
I hope to have more free time to work on it in the next few days.
@ernsteiswuerfel can you confirm that issue 018-fs-size-overflow is fixed?
@marcosps Yes, 018-fs-size-overflow passes now, thanks!
Tested on both ppc32/x86, Kernel 5.8.11, btrfs-devel + commit 8847cfe. All the other convert tests pass too, except 017-fs-near-full.
Both cli-tests/014-multiple-profiles-warning and convert-tests/017-fs-near-full pass now on 32bit x86 (testet with btrfs-progs v5.10 on kernel 5.11-rc6).
@ernsteiswuerfel good to know! I wonder what fixed them :) Anything, thanks a lot for testing!
Hmm, seems I have to reopen this. Tested now on my Pentium4 box, btrfs-progs 5.16.1, kernel 5.17-rc3...
====== RUN CHECK mount -o loop -t ext4 /root/build/btrfs-progs/tests/test.img /root/build/btrfs-progs/tests/mnt
====== RUN CHECK dd if=/dev/zero bs=1M count=64 of=/root/build/btrfs-progs/tests/mnt/convert_space_holder status=noxfer
64+0 Datens?tze ein
64+0 Datens?tze aus
====== RUN CHECK fallocate -l 200M /root/build/btrfs-progs/tests/mnt/file1
====== RUN CHECK fallocate -l 200M /root/build/btrfs-progs/tests/mnt/file2
====== RUN CHECK fallocate -l 200M /root/build/btrfs-progs/tests/mnt/file3
====== RUN CHECK fallocate -l 200M /root/build/btrfs-progs/tests/mnt/file4
====== RUN CHECK fallocate -l 205M /root/build/btrfs-progs/tests/mnt/file1
====== RUN CHECK fallocate -l 205M /root/build/btrfs-progs/tests/mnt/file2
====== RUN CHECK fallocate -l 205M /root/build/btrfs-progs/tests/mnt/file3
====== RUN CHECK fallocate -l 205M /root/build/btrfs-progs/tests/mnt/file4
====== RUN CHECK umount /root/build/btrfs-progs/tests/test.img
====== RUN CHECK /root/build/btrfs-progs/btrfs-convert /root/build/btrfs-progs/tests/test.img
ERROR: failed to reserve 65536 bytes for temporary superblock, largest available: 0 bytes
ERROR: unable to create initial ctree: No space left on device
WARNING: error during conversion, the original filesystem is not modified
btrfs-convert from btrfs-progs v5.16.1
Source filesystem:
Type: ext2
Label:
Blocksize: 4096
UUID: 281f49b6-7592-41ed-ab80-a95dc4d461e9
Target filesystem:
Label: p�跴朿�朿p���tַ
locksize: 4096
Nodesize: 16384
UUID: 153f54f8-7a9f-4997-9801-c1e31a0210ea
Checksum: crc32c
Features: extref, skinny-metadata, no-holes (default)
Data csum: yes
Inline data: yes
Copy xattr: yes
Reported stats:
Total space: 1073741824
Free space: 0 (0.00%)
Inode count: 65536
Free inodes: 65520
Block count: 262144
Create initial btrfs filesystem
failed: /root/build/btrfs-progs/btrfs-convert /root/build/btrfs-progs/tests/test.img
test failed for case 017-fs-near-full
No change with btrfs-progs 5.18, kernel 5.18.0.
====== RUN CHECK /root/build/btrfs-progs/btrfs-convert /root/build/btrfs-progs/tests/test.img
ERROR: failed to reserve 33554432 bytes for metadata chunk, largest available: 29294592 bytes
ERROR: unable to create initial ctree: No space left on device
WARNING: error during conversion, the original filesystem is not modified
btrfs-convert from btrfs-progs v5.18
Source filesystem:
Type: ext2
Label:
Blocksize: 4096
UUID: 2e9b1051-f0d5-44dd-932b-37e8c0b50c9a
Target filesystem:
Label:
Blocksize: 4096
Nodesize: 16384
UUID: 17012638-1c8c-4591-b27a-c3ac781b091b
Checksum: crc32c
Features: extref, skinny-metadata, no-holes (default)
Data csum: yes
Inline data: yes
Copy xattr: yes
Reported stats:
Total space: 1073741824
Free space: 37027840 (3.45%)
Inode count: 65536
Free inodes: 65520
Block count: 262144
Create initial btrfs filesystem
failed: /root/build/btrfs-progs/btrfs-convert /root/build/btrfs-progs/tests/test.img
test failed for case 017-fs-near-full
btrfs-progs was built from current git-master.
Some data about the machine: