Better cache for cardshop workers

kelson42 commented 2 years ago

Goal is to secure that in 99% of cases ZIM files are already on the disk of the worker before card creation process starts. This will allow to save us the time of download and should speed-up even more the card creation. It will also make the whole system less sensitive to network/download problems.

Technical approach:

Implement an instrumentable proxy at worker level
Imlement ability for the cardshop to: make an union list beetween official ZIM library and ZIM in card recipes, tell the workers prox to delete deprecated ZIM files and (pre-)download new ZIM files (which can potentially be required if a recipe is used to build a card).

kelson42 commented 5 months ago

@rgaudin I have the feeling this issue is a duplicate of https://github.com/offspot/image-creator/issues/1 (and therefore solved meanwhile).

rgaudin commented 5 months ago

It's not exactly a duplicate because it proposes a solution instead of describing an issue. We solved this in a different manner in that other ticket indeed.

offspot / imager-service

Better cache for cardshop workers #288