Closed djkapner closed 4 years ago
Merging #114 into master will decrease coverage by
0.07%
. The diff coverage is93.93%
.
@@ Coverage Diff @@
## master #114 +/- ##
==========================================
- Coverage 91.50% 91.42% -0.08%
==========================================
Files 14 14
Lines 683 700 +17
==========================================
+ Hits 625 640 +15
- Misses 58 60 +2
Impacted Files | Coverage Δ | |
---|---|---|
slapp/transfers/utils.py | 97.01% <88.23%> (-2.99%) |
:arrow_down: |
slapp/transfers/upload.py | 93.15% <100.00%> (+0.84%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update 82640a8...d77476b. Read the comment docs.
Easier to review if #111 is approved and merged first.
111 improved upload speed by about 2x through parallelization. We considered that further improvements in speed could come from eliminating the file i/o incorporated in the roundtrip validation. This PR makes that roundtrip optional and off by default. When off, it is replaced by just checking whether an object exists.
This improved speed by another 2x. This indicates that disk i/o is not a significant contributor, but, that there is some network bottleneck we are hitting. Without round-tripping the objects, we have 2x less traffic. I will run another 1000 ROI upload to check this. When writing, I encountered some latency in objects appearing on S3. I added retries and delays, and hope this longer test will confirm that these default settings are sufficiently robust.