optimize caching

[y-brehm]

Our caching can be optimized in various ways:

• switch from a dict to a dataclass when reading from a yaml file.
• we have to evaluate if we always want to write a new cache file or if we want to overwrite the existing one.
• we can write more data into the cache file, like all original levels (peak, lufs). This way we can further speed up processing.

[y-brehm]

Point 2 was already solved by @SimonZimmer in the scope of #35.

Regarding point 3: Thinking about this, I can see an edge case where this could lead to an error: If a user for example waveAligns their USB stick and afterwards adds some files to the stick as well as changes the loudness target and waveAligns again, we would actually process with wrong data. Recordbox does not replace the original files that are already existent on the stick, so we would process files that were already waveAligned, but considering the original levels. It is not impossible to solve this: We would need to save more data or some kind of processing history. But since this can easily become a big pain in the butt, I would suggest to skip this for now. Any opinions @maosi100 @sabjorn @SimonZimmer?

[sabjorn]

@y-brehm agreed about 3. Since it’s a “destructive” process, limiting users’ ability to completely crush their tracks.

Although — you could run a process beforehand to check the current settings against the cache and warn the user—requiring user input to proceed. The warning could include:

warning — the audio tracks you are trying to process have already been processed and the new settings could cause issues. If you are processing copies of your audio (for example—a Rekordbox USB stick) — consider replacing the copies with the originals and re-run this process.

A -f flag could then be added to override.

It is unfortunate because some audio formats do have support for a gain level in the metadata but it seems as though no audio players support it (CDJs and Rekordbox included).

[SimonZimmer]

@y-brehm @sabjorn Hmmm, so this would be the use-case:

• There is a usb stick with processed (wave-aligned) tracks at -12dBLUFS and a written .wavealign_cache.yml in /Volumes/myusbstick/. For each track, the yaml file contains the filepath, original peak and LUFS level as well as the target LUFS that was given when the track-cache-entry was written.
• New files are added to /Volumes/myusbstick/Contents/...
• wavealign is run with -t -10dBLUFS

In this case, wavealign would read the cache file in and recognize that the target level has changed from -12 to -10dbLUFS for all entries. Therefore it will use the original peak and LUFS values for each entry, adjust the gain to the difference between that LUFS and the -10, encode the files and replace the target LUFS value with the new one, leaving the original peak/LUFS values. It also recognizes there are new files added and therefore processes those from scratch, making entries for each.

Because each destination given by the -o flag gets it's own .wavealign_cache.yml, this can be repeated for as usb sticks as possible. So as long as we stick with the paradigm "align USB sticks not the library" I think this feature would be just fine. Am I missing something?

[maosi100]

I'd also follow @SimonZimmer's suggestions on this topic.

However I want to use this place to raise another issue. Caching does not seem to work properly on my machine and I assume this is due to the last modified date of the file not being the same as written into the cache.

Starting with a new file:

last modified cache: None
last modified file: 1594487240.0

A new file does not have a cache entry and only it's native last modified date.

Processing the same file again:

last modified cache: 1594487240.0
last modified file: 1718106214.1836145

The cache date was written with the native last modified date. Not the date of the modification via wavealign. The data within the file represents the time of modification.

Processing the same file again:

last modified cache: 1718106214.1836145
last modified file: 1718106257.9980752

The cache data represents the data of the penultimate processing, not the last one whiche would be the same as the last modified date within the file.

Does it work on your machines? If not I suggest to implement a fix by who's currently working on the caching topic.. @y-brehm ?

[SimonZimmer]

I spoke a bit with @y-brehm today and we reached the conclusion that the caching should only need to hold two fields of data:

str: filename float: lufs_level

This would solve the initial issue of using any wrong original data since the only thing that is being read from the cache is the resulting lufs after the last processing along with the filename that it belongs to. It might also solve the problem @maosi100 pointed out, since we won't have any date information present in the cache file anymore.