processmedia3

[shish]

I spent a few days trying to add threading into processmedia2 and gave up as I kept running into caching and stateful-ness issues; so now I'm trying a version that has minimal caching and no state. No unit tests yet, but everything else seems to work:

• the actual encoding is the same as before (subtitle processing, tag processing, and ffmpeg command lines directly copy-pasted) - just the scheduling parts of "look at the source directory, decide what needs to be encoded, call ffmpeg a bunch of times" is redone
• much less caching
  • the only caching is on the individual-file, individual-attribute level:
  • for files in the source directory, we cache hash, duration, lyrics, and tags
  • intermediate results aren't cached at all
  • mapping of Source files to Tracks is done at runtime
  • mapping of Tracks to Target files in the processed directory is done at runtime
• made a lot of things stateless
  • no more metadata manager - whenever we want some metadata, we get it direct from the source file
• every stage (scan, encode, export, cleanup) is multi-threaded

Practical benefits:

• bulk-encodes can now take advantage of a 64-core server without ugly hacks, for a ~20x speedup compared to just ffmpeg -threads N
• no possibility of problems where source/metadata or metadata/processed directories get out of sync, since we always fetch data direct from the source
• errors in later stages are caught now - no more "source directory hasn't changed, so I'll skip encoding that missing file in the processed directory"
• around 1/3rd the total lines of code, and it's easier to understand what's going on since there are way fewer moving parts
• only one dependency outside the python standard library (tqdm, for multi-threaded map() with a progress bar)
• computing everything at runtime ends up being faster than loading thousands of .json files from the meta directory

Notes for code review:

• main.py covers the same phases as processmedia2 (scan, encode, export, cleanup)
• lib/kktypes.py has classes to represent Source (a file in the source dir), Target (a file in the processed dir), and Track (an entry in tracks.json, which connects sources to targets)
• if you read the two files above, that's all of the processing logic, the rest is implementation details
• lib/tag_processsor.py is copy-pasted from processmedia2
• lib/subtitle_processor.py is copy-pasted from processmedia2
• lib/encoders.py is a list of encoding methods for source and target pairs, eg "given 'an image and an audio track', how can we generate 'a .webm file with av1 video and aac audio'?"

[shish]

While trying to make sure this generates exactly-compatible outputs with pm2, I found a bug - PM2 generates the output filename based on the hash of the input media, not the input file -- ie, the filename for the generated subtitles is based on the input video. So when the input video changes, the subtitles are re-encoded; but when the input subtitles are changed, output subtitles are left alone... That explains why I was having trouble getting it to recognise my subtitle typo fixes >_<