uSD removal while writing results in empty files

[lucaceresoli]

Removing an exFAT-formatted uSD card while writing a large file results in the file appearing with zero length after remount.

My test procedure:

1. cat /dev/urandom >/media/sd/bigfile
2. wait for the file to be >3 GB large (RAM size is 2 GB)
3. remove the card (without stopping the 'cat' command)
4. wait for the system to report errors and settle
5. re-insert the card
6. run exfatfsck

    # exfatfsck /dev/mmcblk1p3 
    exfatfsck 1.2.8
    WARN: volume was not unmounted cleanly.
    Checking file system on /dev/mmcblk1p3.
    File system version           1.0
    Sector size                 512 bytes
    Cluster size                 32 KB
    Volume size                  13 GB
    Used space                 2016 KB
    Available space              13 GB
    Totally 3 directories and 4 files.
    File system checking finished. No errors found.
    #

7. mount the partition again

Result: now /media/sd/bigfile appears, but has zero length.

This happens on an ARM64 Xilinx ZynqMP SoC, Linux 4.14.0, 64 bit, fuse 2.9.8.

Variations I tried:

• fuse-exfat 1.2.3, 1.2.8 and 1.3.0, same result
• two different uSD cards: a 16 GB and a 64 GB, both formatted as exFAT, same result
• with default kernel settings or with sysctl values vm.dirty_writeback_centisecs=50, vm.dirty_expire_centisecs=100, vm.dirty_ratio=50 and vm.dirty_background_ratio=5, same result

This is different on my PC (Ubuntu 18.04, Linux 4.18.0, 64 bit, fuse 2.9.7, exfat-fuse 1.2.8). The same cat /dev/urandom test on the same cards on the PC produces files that are smaller than expected, but definitely not empty. E.g. after writing 5 GB and removing the card I got a 700 MB file after removal and reinsertion.

The same test on the same ARM64 board and the same uSD cards formatted as vfat produces files whose size is truncated, but most of the content is present. Is it possible to obtain the same result with exfat-fuse?

Thanks

[relan]

Is it possible to obtain the same result with exfat-fuse?

It's indeed possible to sync changes periodically (not only on file close) but this will degrade performance (BTW you can run sync(1) in a loop in another process to achieve this). Sounds like you have a specific use-case (some embedded application?). Could you explain why you need this behavior?

[lucaceresoli]

Is it possible to obtain the same result with exfat-fuse?

It's indeed possible to sync changes periodically (not only on file close) but this will degrade performance (BTW you can run sync(1) in a loop in another process to achieve this).

While cat /dev/urandom >/media/sd/bigfile was running I launched sync in another shell, then waited for a couple minutes and removed the card. The result is the same, including the exfatfsck results.

Sounds like you have a specific use-case (some embedded application?).

Indeed I'm developing an embedded Linux application where multimedia data is saved continuously to the uSD card.

Could you explain why you need this behavior?

The card could be disconnected without notice while writing. Of course we encourage our users not to do it, but it still may happen due to mechanical vibrations. In this case we need to have the file content until a few seconds before the removal, just like we do with vfat.

Thanks.

[relan]

While cat /dev/urandom >/media/sd/bigfile was running I launched sync in another shell, then waited for a couple minutes and removed the card. The result is the same, including the exfatfsck results.

Did you call sync with no arguments? You should specify the mountpoint or FUSE handler won't be called:

sync /media/sd

I'm developing an embedded Linux application where multimedia data is saved continuously to the uSD card.

I see. How important is performance in your case?

[lucaceresoli]

While cat /dev/urandom >/media/sd/bigfile was running I launched sync in another shell, then waited for a couple minutes and removed the card. The result is the same, including the exfatfsck results.

Did you call sync with no arguments? You should specify the mountpoint or FUSE handler won't be called:

sync /media/sd

Indeed, sync /media/sd (or sync -f ...) saves the whole file up to the current length!

Incidentally I also had to rebuild Busybox with CONFIG_FEATURE_SYNC_FANCY enabled. Before that it was ignoring all flags and parameters.

I'm developing an embedded Linux application where multimedia data is saved continuously to the uSD card.

I see. How important is performance in your case?

I'm saving video at about 5 MB/s, so I need enough performance to save that, and I think it's way below the performance of current cards. Other than that performance is not much relevant. What i strongly care about is to not lose data upon card removal. The sync /media/sd trick seems a good way to solve my problem.

It is a bit annoying that exFAT needs an explicit sync while FAT does not. Do you think exfat-fuse could be improved to sync data once in a while without need to call fsync()/syncfs() periodically?

Thanks a lot!

[lucaceresoli]

Here's a report after detailed testing of the various sync APIs.

TL;DR: fsync() works even if passed the wrong path; syncfs() does not work.

Details follow.

Test procedure:

1. uSD exFAT partition mounted on /media/sd
2. start writing a file to /media/sd/DCIM/1000TEST/FILExxxx.MP4 (xxxx = progressive number)
3. when the file is ~40 MB, issue one of the sync commands below
4. when the file is ~60 MB, remove card
5. insert card in PC, look at file

The sync command is Busybox sync with CONFIG_FEATURE_SYNC_FANCY enabled. As far as I can tell it bevaves like coreutils sync. I checked with strace that it calls the expected syscalls.

The following commands succeed in saving a ~40 MB file:

• sync /media/sd/DCIM/ or sync /media/sd
• sync -d /media/sd/DCIM/ or sync -d /media/sd

These call openat() + fsync(). This result was not expected (but still OK): fsync() and fdatasync() are supposed to synchronize the file, not the whole filesystem, and I'm not sure they are supposed to work recursively.

The following commands fail, resulting in a zero byte file:

• sync -f /media/sd/DCIM/1000TEST/FILExxxx.MP4
  • Note: works if xxxx is the number of the file currently being written
• sync -f /media/sd
• sync -f /media/sd/
• sync -f /media/sd/DCIM/1000TEST/

These call openat() + syncfs(), so it looks like syncfs() does not work as the manpage describes.

I would like to use syncfs() since while saving the video file I also write other precious files, and synchronizing the whole filesystem is simpler and more efficient than synchronizing each file individually.

Do you think syncfs() could be implemented?

Thanks.