search

kdave / btrfs-progs

Development of userspace BTRFS tools
GNU General Public License v2.0
561 stars 244 forks source link

(Not sure if this is the right bug tracker) Btrfs with 6.9.10 built for PinePhone, kernel variant "postmarketos-allwinner", has constant filesystem corruption after copying a larger amount of files with syncthing with different SD Card vendors #860

Open ell1e opened 4 months ago

ell1e commented 4 months ago

Btrfs with 6.9.10 built for PinePhone, kernel variant "postmarketos-allwinner", has constant filesystem corruption after copying a larger amount of files with syncthing with different SD Card vendors. I assume this might as well be a chipset bug of some sort, I don't know, but I figured that you guys might maybe have an idea how to find out what's going on. My apologies if this is the wrong bug tracker, I guess it doesn't really affect the btrfs userspace tools or I don't know.

I reported the original downstream distribution bug here: https://gitlab.com/postmarketOS/pmaports/-/issues/3058

I tested this on this device: PinePhone Allwinner 3GB RAM that is pretty close to an ARM64 single board PC like a Raspberry but connected via USB with a modem and other phone peripherals.

I tested it with these two SD Card vendors: 512GB SanDisk card, 1TB Lexar card

I tested with this operating system: postmarketOS 24.06, which is based on Alpine Linux

The corruption happens immediately after moving to another card, so it doesn't seem to be wear related in any obvious way.

Here are the steps to reproduce from the downstream issue:

  1. Set up your PinePhone with btrfs. (Note for people new to postmarketOS: to make such a postmarketOS install, this would be usually done using https://wiki.postmarketos.org/wiki/Pmbootstrap to make an install image with BTRFS on your desktop pc. The pmbootstrap tool will ask you which filesystem to use in the initial setup. I also installed LUKS encryption, which I'm not sure whether it's relevant or not. then to dd that image to some microsd card using an sd card reader, and then put that microsd card into the PinePhone. Due to how the Allwinner bootloader works, it should boot from the microsd install with a higher priority than whatever is on the internal eMMC at the time, which should then be ignored.)

  2. Get syncthing for ARM64, without any UI or whatever, just download the binary from https://github.com/syncthing/syncthing/releases for ARM64 and install gcompat to run the glibc binary directly without any containers.

  3. Install syncthing on some other device of yours, let's assume for this example that would be a laptop.

  4. Set up the PinePhone as wifi hotspot. I assume USB networking or having the PinePhone as a wifi client might also work, but this is the setup I used.

  5. Connect the laptop to the wifi hotspot.

  6. Set up syncthing on PinePhone and laptop such that they see each other as a device.

  7. Add an empty folder on each side to be synced in both directions.

  8. Now on the laptop, copy 100-200GB worth of tons of small files into the folder as fast as possible. Ideally, your laptop has an SSD. It should now start pushing those with a couple of megabytes per second, as fast as the PinePhone wifi chip can handle, and syncthing on the PinePhone should really heat up that CPU while streaming all the huge amounts of incoming scattered data onto the filesystem.

  9. Let it sync for half an hour. Then disconnect the laptop. It doesn't matter whether the sync finished by then, most likely it won't have.

  10. Start btrfs scrub start / on the PinePhone and gradually check dmesg for the inevitable corruption errors listed. The path listed for each error should usually point back into the new syncthing folder where it placed the new files.

ell1e commented 3 months ago

There's now a corresponding conversation going on here: https://lore.kernel.org/linux-btrfs/9b4f0e79-6e77-48a4-b87d-b27454ffb399@horse64.org/