Replay from cached artefact

[kelson42]

It would make a few things easier if we would be able to place a recipe from stored artefacts. This is in particular true for Zimit, where we have many scenarios were we want to relaunch the recipe without the burden (for both us and the upstream Web site) of the spider part.

This could work like this:

• Boolean settings at the recipe: "Download artifact if available"
• The Zimfarm worker would just download the artifacts and put them at the same place where they have been downloaded
• Scraper could resume (WARC) files are already available (via dedicated option?)

[benoit74]

I like the idea.

Some suggestions:

• we should use the last available artifacts, not the one from last task (which might have produced no artifacts since already running from previous artifacts)
• we should find a way to place downloaded artifacts in a location which will not be considered for archiving at the end of the task, so that we do not archive again and again the same artifacts
• I would rather rename the boolean "Download latest artifacts before task execution" ("if available" is implicit but quite obvious)

A corollary of this feature is https://github.com/openzim/zimfarm/issues/940

[rgaudin]

This is in particular true for Zimit, where we have many scenarios were we want to relaunch the recipe without the burden (for both us and the upstream Web site) of the spider part.

That's an interesting question because zimit is not a regular scraper. It plays the role of a scraper but it's two tools run one after the other: a scraper (albeit for the WARC format) and a converter from WARC to ZIM. It's because WARC is a standard format and it provides value to convert from it that we didn't plug into browsertrix to write ZIM content directly.

All scrapers purposely (performance) run in one pass and directly store transformed data in ZIM. Specific data are sent to our S3 Optimization cache to save on both spiding and processing time when there are good chances this data is static.

I don't see how this can be useful for non-zimit scrapers.

For zimit, I see how this feature can be useful but only for development: when would we want to re-create a ZIM from the same WARCs? When our converter code failed. Are there other use cases?

We've refused to use Zimfarm as a dev tool for now but adding the artifacts was a clear step towards it…