uploader improvements - Githubissues

AllenInstitute / segmentation-labeling-app

Data pipeline and UI for human labeling of putative ROIs from 2p cell segmentations

Other

0 stars 0 forks source link

uploader improvements #112

Closed djkapner closed 4 years ago

djkapner commented 4 years ago

The uploader was pretty slow, and there was no mechanism for recovering from a single failure within an entire manifest upload attempt.

[x] implement some parallelization to increase speed
[x] increase speed by removing round-trip check and relying on botocore has checking.
[x] expose control of underlying botocore.config.Config
[x] modify the uploader to attempt to clean up failed uploads, and, at the very least, warn and log them without failing.

I plan to close #111 and #114 in favor of #115

kschelonka commented 4 years ago

yay, leveraging the checksum from s3