search

nodiscc / hecat

Generic automation tool around data stored as plaintext YAML files
GNU General Public License v3.0
29 stars 5 forks source link

processors/archive_webpages: skip rewriting the data file if `archive_error: True` and a new attempt also fails #131

Open nodiscc opened 11 months ago

nodiscc commented 11 months ago

When re-attempting to archive an URL for which archive_error is True, and the new download attempt fails, the data file will be re-written to write the archive_error: True field.

However this is not needed (since the key is already set), and wastes time writing the temporary file and copying it back to the data file (especially with large data files).