search

novoda / download-manager

A library that handles long-running downloads, handling the network interactions and retrying downloads automatically after failures
Apache License 2.0
483 stars 63 forks source link

DLM-491/File Size Requesting #492

Closed Mecharyry closed 5 years ago

Mecharyry commented 5 years ago

Problem

As detailed in #491, sometimes clients need to process small requests quickly to avoid performance issues where a batch has a lot of small files to process.

Solution

As a workaround, until we investigate having parallel head requests for the file sizes we have opted to add an Optional FileSize to the initial batch builder. If specified then the head requests will be ignored. This is potentially dangerous, if the size is incorrect the batch will fail during the download. We are leaving this to the clients.

Screen Capture

Before Screenshot 2019-04-17 at 08 54 20
After Screenshot 2019-04-17 at 08 54 44

On incorrect size specified

when_size_incorrect

Mecharyry commented 5 years ago

❌ DO NOT MERGE AT THE MOMENT ❌ I need to do some CI work before we merge this.

ouchadam commented 5 years ago

@Mecharyry is CI work still needed for this? Would be great to have this merged in & released!! 🙏

Mecharyry commented 5 years ago

@ouchadam thanks for reminding me about this! All done now.