Open suchthis opened 7 years ago
See https://github.com/edgi-govdata-archiving/archivers.space/issues/38 for UI recommendation on how to keep this from happening in the future.
I'm using the "data integrity" label for cleanup tasks like this. I'd love to hear from devs with more MongoDB experience how they would handle a task like this on the db side.
Other options:
It looks like at least some URLs that may have already been harvested are still in Harvest phase in the app, since the user did not click the checkbox next to Harvest. Since this step was not included in previous event workflow documentation, it may be a widespread issue, so it may make sense to programmatically compare UUIDs of datasets already uploaded with UUIDs still in Harvest phase in app (and change status in app for any UUIDs with uploaded datasets).
Example: http://www.archivers.space/urls/F68DCA69-4377-40DA-B576-7D3C88CC6C2A
Harvest notes: "Over 6,000 files totaling 82 GB. Largest file is 12 GB, which is a massive orthographic mosaic tif. Zip file of 62 GB was uploaded via AWS token, appears to have completed successfully at 5:26 PM, though this site does not seem to acknowledge it."
This may explain why there are relatively few URLs in post-harvest phases in the app, despite the many recent events.