AllenInstitute / segmentation-labeling-app

Data pipeline and UI for human labeling of putative ROIs from 2p cell segmentations
Other
0 stars 0 forks source link

Feature/faster upload validation #114

Closed djkapner closed 4 years ago

djkapner commented 4 years ago

Easier to review if #111 is approved and merged first.

111 improved upload speed by about 2x through parallelization. We considered that further improvements in speed could come from eliminating the file i/o incorporated in the roundtrip validation. This PR makes that roundtrip optional and off by default. When off, it is replaced by just checking whether an object exists.

This improved speed by another 2x. This indicates that disk i/o is not a significant contributor, but, that there is some network bottleneck we are hitting. Without round-tripping the objects, we have 2x less traffic. I will run another 1000 ROI upload to check this. When writing, I encountered some latency in objects appearing on S3. I added retries and delays, and hope this longer test will confirm that these default settings are sufficiently robust.

codecov-commenter commented 4 years ago

Codecov Report

Merging #114 into master will decrease coverage by 0.07%. The diff coverage is 93.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #114      +/-   ##
==========================================
- Coverage   91.50%   91.42%   -0.08%     
==========================================
  Files          14       14              
  Lines         683      700      +17     
==========================================
+ Hits          625      640      +15     
- Misses         58       60       +2     
Impacted Files Coverage Δ
slapp/transfers/utils.py 97.01% <88.23%> (-2.99%) :arrow_down:
slapp/transfers/upload.py 93.15% <100.00%> (+0.84%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 82640a8...d77476b. Read the comment docs.