transfers to GCS should happen after each resource is harvested

VertNet / gulo

Shredding Darwin Core Archives with ferocity, strength, and Cascalog.

7 stars 5 forks source link

transfers to GCS should happen after each resource is harvested #100

Closed robinkraft closed 11 years ago

robinkraft commented 11 years ago

Currently, the transfer happens all at once after all harvesting has occurred. Better to do "incremental" uploads in case we get booted off a spot instance.

eightysteele commented 11 years ago

beast