webrecorder / warcit

Convert Directories, Files and ZIP Files to Web Archives (WARC)
https://pypi.python.org/pypi/warcit
Apache License 2.0
81 stars 13 forks source link

Atomic conversion outputs #28

Open anjackson opened 1 year ago

anjackson commented 1 year ago

This adds features intended to cope in situations where a lot of conversions are needed and the process may get interrupter. The changes are that the conversion command will be skipped if the output file already exists, and the actual conversion uses a temporary file name so we know whether the conversion command ran to completion. Hence, if a large set of conversions is interrupted, this will continue the process, restarting the last interrupted conversion.