Rare filesystem corruption with extra files added

[MoleMan1024]

Describe the issue Two users still have reported filesystem corruption where extra files are being added to the filesystem after indexing via AudioWagon.

To Reproduce Unknown, seems to happen with certain sets of files only (I suspect a certain distribution in the FAT 32 chains). If you have a small set of files that reliably reproduces the issue, please let me know, I want those files.

[gahabana]

not sure if it is the same issue. now for 2nd time in few weeks here is what happened to me:

1. am using USB-C enclosure with NVME drive. Capacity 2TB, formatted as FAT32 and filled 1.1TB with 40k FLAC files
2. Delete all databases in AudioWagon, connect the drive and manually start the scan
3. After 20 minutes of driving it has scanned only about 3000 songs and i stop the car.
4. 30 mins later i return, start the car, drive gets recognised but sees only half of the directories (seems it stops at those starting with K) ...
5. disconnecting the drive, taking it to home-pc and checking it there are some errors, 2 LOST.DIRs on the drive and about half of the content has disappeared.
6. It is very time consuming and if i did multi hours drives every day i may be able to log the issue and upload (as app does not crash) but logging makes it even slower so it is hard

Above has happened 2 times ... even the file that was playing from the playlist (Roxy Music so it was starting with letter R has disappeared) .

will try one more time from scratch ... i really like AW a lot as app when it works but this is now really difficult to be usable w/o metadata for larger library (other then browsing files and playing playlists which works great).

[MoleMan1024]

I sounds like the same issue. However I would need a small reproduction setup to analyze this. Testing with 1.1 TB of files is not efficient, this can happen also with a couple of files but so far no one has been able to reliably reproduce it with a small set of files. If anyone finds such a set of files, please provide to me.

[gahabana]

initially (at step 2) i did have logging turned on. i turned it off few minutes after 1st scan started (w/o interrupting the scan). not sure if that helps at all but maybe it does ... my feeling is since only things written to the USB drive are either the DB or the log files. maybe there are some ways to check ... on the other hand if libaums has issues .... ahhhh there it is. can you take a look i would bet this is the root cause: https://github.com/magnusja/libaums/issues/298 and this one is referring to that one above: https://github.com/magnusja/libaums/issues/353

My guess is then for you to test is just to have 200 files/directories in the root ?

[MoleMan1024]

@gahabana : For the issues you mentioned we already have workarounds, see https://github.com/MoleMan1024/audiowagon/issues/37 and https://github.com/MoleMan1024/audiowagon/issues/13 The current issue is different and even more rare but has the same effect of corrupting the filesystem.

[gahabana]

I can reproduce the issue. Sadly due to filesystem corruption I find no log files on SSD afterwards.

I took my library of 40k files in .flac format. Applied simple python script that went into each subdirectory and where it found more then 2 .flac files it deleted randomly all but 2. So each album was 'shrunk' to 2 songs instead of 5, 10 or 20 which it might ve been before.

As a result instead of it 40k+ tracks , i was left with slightly over 5000 music files (aka incomplete albums) on the drive.

Steps that reproduce the same problem that i have had with 40k files I experienced with 5000.

1. delete all databases from Audio Wagon
2. plug in USB drive. Dismiss Volvo message. Accept AudioWagon message to access external drive
3. Logging turned on. Metadata was at manual to begin with
4. Told it(clicked) 'Read Metadata now'
5. in about 6-7 minutes it reached 3000 files indexed (out of 5000)
6. i clicked on 'Eject USB drive'
7. then i reconnected the drive, accepted messages as usual (Dismiss and allow acccess)
8. clicked on 'Read Metadata now' again
9. it finished suspiciously quickly
10. when looking at 'Files' i realised only directories from A to K are visible ....
11. Ejected the drive again and brought it home
12. Examining what i can see on a PC it is only top level directories (Artist Names) which are doubled and they have empty content.
13. the only files that I could still find on it are '.m3u' playlist files which now point to nothing since there are no visible other files

Not sure how to take it from here in absence of log files but now within 10 minutes I can reproduce the issue ... my guess it still has to do with >200 entries in a directory...

Way you could test it with 2000 files - I am attaching my directory structure. If you put 2 small mp3 files into each directory you will have same directory and file structure as i had (my guess it would be good to have files be different so they dont produce same tags but that may not be important).

If i wanted to share FLAC library with you, the one that is shrank to 5000 files is 135GB. If you really need me to make it very replicable i can convert it to mp3 or opus with 64kbit encoding and my guess is it would be 10GB only.

Let me know how we can help each other in a best way.

output of directory tree below: music.directories.tree-output.txt

[MoleMan1024]

@gahabana : I will try to reproduce it again with generated data of 2 files in lots of "artist" directories. I need a small enough test set for debugging because I will kill the filesystem often while investigating the bug and uploading 2000 files to USB will just take too long.

[gahabana]

i am sure it should take less time to transfer 2000 files to USB then for me to walk to the car with reduced set of files, see if it fails and if it does go back to appt, reformat, copy all files and walk again until it does not fail 🥇 🗡️ :)

anyhow, am converting library now to 32kb opus files. that will be 38GB. will reduce 41000 files down to 5000 and then down to 2500 (one per directory). each file is about 1MB large. that is 2GB. they dont make USB sticks so small anymore anyhow.

even at USB 2.0 speeds that should take less then 2 minutes to transfer 2GB.

Of course I am assuming it will still fail. Will let you know !

update: made 3 different copies of said 32kbit Opus files. one 2500 music files, one 999 and one 250 and have put it on 3 different USB drives. Will test probably tomorrow and report back.

[gahabana]

ok. have tested 2500 files as well as 999 . following the scenario earlier in both cases directories after certain letter disappear (from the root directory) and existing ones (A-K or A-L) show up 3 times. there are no files to be found.

So there are no extra files but rather simmilar to the problem i hinted earlier with libaums ... any writing (log files) i believe triggers it.

In both cases I started reading metadata and when it hit close to only 200 files (mabe a minute from the start) the start i clicked on 'eject USB drive' ...

So, will try tomorrow smallest dataset (250 music files) but am pretty certain it will finish the same.

I don't know if audiowagon overrides parts of libaums library but for this specific problem to be replicated all i think is sufficient is https://github.com/magnusja/libaums/issues/298:

1. have root directory with >600 directories (in my case they are artist names)
2. have one of the subdirectories have >400 directories (that is my 'Various Artists' directory)

[MoleMan1024]

@gahabana : I already tried yesterday using the directory structure from your textfile. I recreated that using a tool, put it on my USB drive and also put a single .mp3 file in each of those directories. This did not trigger the issue, so it is not only the number of directory entries. But thank you for continuing, so far I have never been able to reproduce this on my side, you seem to be on a good track.

[gahabana]

@gahabana : I already tried yesterday using the directory structure from your textfile. I recreated that using a tool, put it on my USB drive and also put a single .mp3 file in each of those directories. This did not trigger the issue, so it is not only the number of directory entries. But thank you for continuing, so far I have never been able to reproduce this on my side, you seem to be on a good track.

thank you for your time and efforts. so if you had a file in each directory it must've been 2500+ (mp3) files ? i doubt filetype plays a role .... have you interrupted scan with 'eject' before it finished ... as well as i did (i did that only to simulate what happens if i switch off the car during scan) ... if so i am not sure how my efforts help you ...am confused :)

[gahabana]

ok. have tested 2500 files as well as 999 . following the scenario earlier in both cases directories after certain letter disappear (from the root directory) and existing ones (A-K or A-L) show up 3 times. there are no files to be found.

So there are no extra files but rather simmilar to the problem i hinted earlier with libaums ... any writing (log files) i believe triggers it.

In both cases I started reading metadata and when it hit close to only 200 files (mabe a minute from the start) the start i clicked on 'eject USB drive' ...

So, will try tomorrow smallest dataset (250 music files) but am pretty certain it will finish the same.

I don't know if audiowagon overrides parts of libaums library but for this specific problem to be replicated all i think is sufficient is https://github.com/magnusja/libaums/issues/298:

1. have root directory with >600 directories (in my case they are artist names)
2. have one of the subdirectories have >400 directories (that is my 'Various Artists' directory)

tried now with only 250 files on USB drive (.opus format). starting reading Metadata and after it reached 60 i clicked on 'Eject drive' ... 10 seconds later i re-connected it and filesystem was corrupted. Directories during letter L disappeared on the screen. ... that is 270 MB of files not 250 GB (or 1.1TB as it was initialy) - nothing else on the drive - I've also removed cover.jpg and index.jpg so it was just directories and music files.

I really dont know what else to do. Tried 3 different USB-Nvme enclosures with 3 different chipsets (just to eliminate the possibility) ... please advise.

The only other thing i can try to do but if that is success it means issue is different - to try using stick/USB key. I have one with 32GB capacity ... so it will not serve key purpose (having music library in lossless format in the car) nor it would fit entire music library in mp3 format ... but if it will help to maybe show that works (or not) i will do it ?

[MoleMan1024]

@gahabana : "270 MB of files" sounds ok to work with. Do you know how to create a disk image? (e.g. using dd command in linux or win32diskimager in Windows?). Maybe you could find a way to provide these files to me (or better a disk image) then I can try to reproduce the issue on my side. You will need some kind of large file transfer service, you can find those online and they should offer that for free. You can use my e-mail if you don't want to share the link here.

[gahabana]

@gahabana : "270 MB of files" sounds ok to work with. Do you know how to create a disk image? (e.g. using dd command in linux or win32diskimager in Windows?). Maybe you could find a way to provide these files to me (or better a disk image) then I can try to reproduce the issue on my side. You will need some kind of large file transfer service, you can find those online and they should offer that for free. You can use my e-mail if you don't want to share the link here.

thank you - email sent and files shared via google drive hope that's ok !

[MoleMan1024]

@gahabana : Issue is reproducible! Thank you so much for all your work, I think the hardest part is done, now I can analyze.

[MoleMan1024]

This will be fixed in version 2.3.7