search

ArchiveTeam / seesaw-kit

Making a reusable toolkit for writing seesaw scripts
Other
69 stars 30 forks source link

Strict wget-lua version and compatibility checking #43

Open chfoo opened 10 years ago

chfoo commented 10 years ago

The find_executable() API call isn't sufficient to determine if wget-lua is fully compatible. Pipelines assume that wget-lua has been compiled with zlib support to produce warc.gz. If a version of wget-lua is used without zlib, 0 sized files are uploaded by the pipelines.

chfoo commented 10 years ago

For dogster-grab, we are now checking if there is a plain warc file. If it exists, then the pipeline raises an Exception.