Closed jelmervdl closed 2 years ago
Also that loadTranslationModel
function is called once for each worker, so effectively with 4 workers I'm now extracting the tarball 4 times 😅
Performance recording: https://share.firefox.dev/3HWms77 (tested on Wikipedia)
Proof-of-concept download the .tar.gz files, extract those in memory, parse the YAML file to find the filenames, and then just continue as normal.
The whole finding the right file, and not actually reading the config file in detail, is pretty much a quick hack to make it just work for now.