Scanning library with no changes takes even longer then building library the 1st time

[gahabana]

With 41k .flac files library build lasts about 50-few minutes with fast NVME2USB enclosure. With recent changes in AudioWagon v2.3.8 I expected speed-up when drive was not removed from the car (i.e. no changes at all to files on the drive) - in expectation that smaller updates would then be done in minutes rather then in an hour.

To Reproduce Scan initally library with 40k tracks - will take 50minutes or so Power down the car, start it again, accept USB msg and access to UBS drive, and in AudioWagon click on 'Read Metadata Now' It takes about 55minutes with said library to complete scan.

Expected behavior I expected scan to finish in few minutes since drive was not at all removed from the car.

AudioWagon version 2.3.8 Car version 2.7

Logfiles here is the log-file: audiowagon_2023-04-28_09-51-33.log

[MoleMan1024]

Indexing always means to visit all files on the USB drive ( see FAQ ). The expectation that this will be faster by some orders of magnitude will only be true if the bottleneck is the metadata extraction, not the file visiting itself. However the latter seems to be the case here: there were no metadata changes, still it took around 80ms to visit and compare timestamps of each file against database which leads to 55 minutes total at 41k files.

Essentially this means going through all the files is the procedure that is slowing AudioWagon down. There could be multiple causes:

External

• Polestar/Volvo might be accessing the USB drive in parallel to AudioWagon, leading to slowdowns. I don't know if that is the case, and even if I knew, I could not do anything about it
• problems in lower layers of Android USB implementation by Google or Volvo/Polestar. I do not have root access to the car, so I don't know
• slow USB drive (old / overheating / too little current / ...)

Internal

• the libaums library does not support multithreading. Rewriting that library to support multithreading could improve this, however this would be a large effort

Possible workarounds

• use less data, i.e. convert your .flac files to .mp3 for example. At a decent bitrate you will not be able to hear any difference
• put less files on USB drive
• index metadata once, then leave your 41k files alone, so you don't have to re-index
• do not index metadata at all
• do not do anything else with USB drive while indexing to lower the workload on the USB drive, i.e. turn off log file writing, do not start playback, do not scroll album lists (this will read album cover data).

I will leave this ticket open but I don't expect I will rewrite libaums library anytime soon, so don't expect any progress here, sorry.

[gahabana]

Thank you @MoleMan1024 , think your analysis is very correct. One question though - if it takes 80ms to visit each file name and timestamp (if i make an assumption that actual file access is really not needed (and it shouldn't) all that requires is reading content of a FAT32 directory) - either libaums is to blame or MAYBE some 'strange' database access is happening. Would it be possible for you to create 'test' or fake test that would only scan filenames and filedates and store them in (later) discarded list or object list .... if that takes 40ms per file then there is really not much you can do to make 80ms lot shorter. but (just maybe) issue is not at reading fienames and filestamps but maybe in reading/querying DB (and who knows if 'room' library is doing something else) ? ... Or maybe without changing too much code just add logging to see DB operations vs getting filename and file-date ?

thank you again !

[MoleMan1024]

@gahabana : Every file needs to be accessed, modifcation time is stored inside the file.

Unless there are a lot more people who feel strongly about this, I am okay with the current performance. You can investigate more if you want, but I will not spend a lot more time on this.

[gahabana]

@gahabana : Every file needs to be accessed, modifcation time is stored inside the file.

Do not beleive that is the case. On FAT32 filesystems file modification time is stored in few (2 i believe) bytes in the dirctory structure/clusters where all files belonging to the same directory are stored. also 1st cluster address is there. hence reading directory or 40000 files in 1500 directories really bogs down into reading about 1500 clusters at most.

Unless there are a lot more people who feel strongly about this, I am okay with the current performance. You can investigate more if you want, but I will not spend a lot more time on this.

I understand :(

[nramsbottom]

Does the entire disk need to be indexed every time the vehicle is started? Could this be on an on-demand feature?

The memory stick I have in my vehicle has around 1500 files (mostly MP4A files) on it and it takes around a minute to index after every startup.

I don't add new files to the stick all that often. Could it be an option to cache previously scanned files unless instructed to perform a reindex?

[MoleMan1024]

@nramsbottom : It is already an on-demand feature, please check the FAQ. You need to set option "Read metadata" to "Manually".

[nramsbottom]

I went to the issues list before the FAQ. Excellent there’s a solution already!

[MoleMan1024]

improved in version 2.4.5

[gahabana]

hi, i did not see measureable difference. To index new library with 40k songs it still takes about 50 minutes and then w/o removing the drive running manual scan to see if there are any differences takes as long. maybe just released 2.4.6 will do better i hope. i do like AudioWagon a lot, it works well, i just cant realistically update any content / add new music as i rarely drive more then an hour. . i do update playlists so that's how i work around it when adding some new songs.

tx, still appreciate your efforts and maintenance of the app !

[gahabana]

hi, i upgraded to 2.4.6.

net - @MoleMan1024 you have done some GREAT work with latest updates for us with large libraries. With no changes to 41k songs (in .FLAC format) - now it takes less then 20 minutes to scan (and find no changes) ... which is for me sufficient for many drives during the week.

I havent added any new music and tested how much longer that takes but simple logic implies that if new scan takes less then an hour for 41k songs and finding no changes less then 20 minutes ... that means adding e.g. 1k songs will make it slightly more then 20minutes which is AWESOME.

I think with this is at is - it is very very usable ... maybe someone could have 100k songs and then it might take still less then an hour, but my take is - i am just lazy to select what NOT to put on SSD in a car rather then actually listening to even half of it :)

THANKS AGAIN MOLEMAN1024 - really great progress and appreciate what you do !!!!!