deal with partial arc files

internetarchive / liveweb

Liveweb proxy of the Wayback Machine project

https://web.archive.org/

44 stars 13 forks source link

Open rajbot opened 12 years ago

rajbot commented 12 years ago

When uwsgi is restarted, partial arc files are left behind that never get moved to the records/complete dir, and never get uploaded.

anandology commented 12 years ago

How about delegating that job to s3-uploader?

s3-uploader can look at "partial/" directory and moves the files which are more than an old and have no references to "completed/".

ersi commented 11 years ago

When is a ARC file complete? Is it complete when reaching a certain size?