Performance issue - request for enhancement

[gahabana]

As per https://github.com/MoleMan1024/audiowagon/issues/103 metadata extraction for larger library has improved speed dramatically. With 40k tracks library it takes about an hour to scan it and ready for playback with metadata.

However, if i add new album to the drive it will still take an hour or more for scanning to complete.

Would it be possible for scanning engine (i have it set to manually 'read metadata now') to be updated to look only for changed or new files rather then extracting metadata from each and every file (which is already in the database).

My guess is easiest would be to add filemodification time and if file with the path and same date is already in the DB then there is no need to extract metadata from it.

I think that would allow for dramatic speed up when library is slightly changed.

This would also allow for files that have been removed to be deleted from the DB and only new/changed ones would need to be fully indexed with metadata extraction.

Thank you for your outstanding work and changes so far :) !!!!

[MoleMan1024]

This is already implemented as you request, see FAQ. To find out which files have changed you still have to look at every file.

[gahabana]

This is not implemented.It is hard to wait (again) for an hour for one new song to be found and metadata extracted.

I think you misunderstood my request - instead of re-reading metadata from each and every file on the USB drive, would it be hard to implement reading only filenames (and maybe dates) and ONLY (and ONLY IF) that file had not been indexed before read metadata from it and update the DB.

Hope this makes it little clearer - apologies if i was not clear earlier.

Or i am wrong and this is how it works, yet for some reason it is still super slow in my case ?

thank you again !!!

[MoleMan1024]

@gahabana : checking the file modification date and not reading metadata if it has not changed since last time is already implemented, see

https://github.com/MoleMan1024/audiowagon/blob/master/automotive/src/main/java/de/moleman1024/audiowagon/repository/AudioItemRepository.kt#L98 and https://github.com/MoleMan1024/audiowagon/blob/master/automotive/src/main/java/de/moleman1024/audiowagon/medialibrary/AudioItemLibrary.kt#L177

Maybe you could upload another (partial) log when changing only a few tracks, I can have a look at the timestamps to see the speed you get during re-indexing and try to reproduce it on my side.

[gahabana]

hi @MoleMan1024 , thank you for the latest updates with 2.3.7- i can see that scan continues after i power up the car and have no longer issue of corrupted filesystem. For what it's worth here is some performance data.

1. when reading new files it updates screen (every 100 files) every 8.7 seconds on average. 87ms for a single file. should take less then an hour for 41000 files. FLAC files on fairly fast USB to NVME enclosure
2. after power up when it scans files it had already stored in DB speed is about 100 files for 2.6 seconds aka 26ms for single file. meaning after the scan is complete it will take 15-20 mins to check for any changes.

For me this is reasonable ... my guess is that #2 could be optimised but am happy with it as it is 3x+ faster speed of scanning existing files compared to those that are not yet in DB.

THANK YOU !

[gahabana]

hi again, after 2.3.7 i no longer have issues with corruption of the filesystem and am able to capture .log files.

Am attaching 6 log files and short description of actions preceding each. Key point is that drive was not removed from the car so files have not changed and after initial scan of library (2nd log file) which did take an hour and 20 mins roughly i tried in log file no5 to scan only for changes (though nothing has changed) manually. It seemed way to slow based on what you had described (checked for file date/time and presence in metadata database). speed was almost the same and would've taken at least 30-40-50 minutes to scan for any changes (and there were none). Last log file ( no6 ) is just to see if there is any difference if i select 'update DB on USB insert' as oposed to 'Read Metadata Manually' but speed seem to be about the same

Hopefully this may be helpful in identifying if all is working as it should or there may be some inefficienses when scanning for new files and there aren't any changes in the filesystem.

Thank you !!!

Log file no.1 (not important, just playing some files before any indexing) audiowagon_2023-03-11_11-23-18.log.tar.gz

Log file no.2 (create manually completely new index/scan of metadata) audiowagon_2023-03-11_11-34-44.log.tar.gz

Log file no.3 (some playback) audiowagon_2023-03-11_14-18-46.log.tar.gz

Log file no.4 (playback, next day) audiowagon_2023-03-12_10-00-47.log.tar.gz

Log file no.5 (start manually reading metadata (there are no changes to filesystem)) audiowagon_2023-03-12_10-19-13.log.tar.gz

Log file no.6 (after aborting 5th log file, drive was ejected and options changed to 'automatic scan of metadata') audiowagon_2023-03-12_10-23-36.log.tar.gz