archiver: handle upload failures better, handle .tmp file removal

mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.

https://mediacloud.org

Apache License 2.0

2 stars 5 forks source link

Closed philbudne closed 2 months ago

philbudne commented 2 months ago

keep local WARC file if any upload fails (as it did in the past)
add/use blobstore upload_fileobj; will upload archive even if .tmp file accidentally removed
deploy.sh: read docker/USERNAME.sh, if it exists, in addition to dev.sh for dev stacks
log and report stats for upload times
report noupload stat if any upload failed