bug: Allow resumable download

riderx commented 8 months ago

Currently, if a network issue happen or the user kill the app, the download is lost and restarted from 0, so implementing resumable download will improve a lot the bandwidth usage and for user with bad connexion

riderx commented 8 months ago

/bounty $200

algora-pbc[bot] commented 8 months ago

💎 $200 bounty • Capgo

Steps to solve:

Start working: Comment /attempt #255 with your implementation plan
Submit work: Create a pull request including /claim #255 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional notes:

ℹ️ If something is not clear ask before working on it, otherwise your chance to rework it is high 🎥 To claim you need to provide in your PR a demo video of the change 👨‍👩‍👧‍👦 Join the Discord to get help 📏 Check all Bounty rules

Additional opportunities:

🔴 Livestream on Algora TV while solving this bounty & earn $200 upon merge! Comment /livestream once live

Thank you for contributing to Cap-go/capacitor-updater!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @neo773	Aug 29, 2023, 3:58:42 PM	WIP
🔴 @ayewo	Sep 20, 2023, 9:32:06 AM	WIP
🔴 @yashk7oo	Oct 19, 2023, 5:04:34 AM	WIP
🔴 @Sambit003	Dec 20, 2023, 1:36:13 PM	WIP
🔴 @Samankhalid01	Feb 6, 2024, 7:23:59 PM	WIP

neo773 commented 8 months ago

/attempt #255 continuing this from our DM in discord

Options

Cancel my attempt

WcaleNieWolny commented 8 months ago

Isn't the partial download (individual file download) already a partial fix for this?

riderx commented 8 months ago

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

algora-pbc[bot] commented 8 months ago

@neo773: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc[bot] commented 8 months ago

The bounty is up for grabs! Everyone is welcome to /attempt #255 🙌

ayewo commented 8 months ago

/attempt #255

Options

Cancel my attempt

WcaleNieWolny commented 8 months ago

@ayewo as far as I know @neo773 has already wrote some code for this and he is nearly completed this. Although he said that it is 90% complete 5 days ago

ayewo commented 8 months ago

I'm sure @neo773 has no trouble speaking for him/herself so there's absolutely no need for you @WcaleNieWolny to speak on their behalf.

WcaleNieWolny commented 8 months ago

And I am sure he will. I am not saying you should not attempt this. Go ahead, just letting you know what I know about the status of this issue

neo773 commented 8 months ago

@ayewo Feel free to work on it, my attempt wasn't successful @WcaleNieWolny that "90% completion" was actually partial downloads

ayewo commented 8 months ago

@neo773 gracias 👍!

@riderx I did some preliminary research and I have a few quick questions:

the response headers for requests to the https://api.capgo.app/updates endpoint includes this header: CF-Cache-Status: DYNAMIC which suggests you have Cloudflare caching enabled, correct?

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
CF-Ray: 7fac66fd7dd30b85-GPS
CF-Cache-Status: DYNAMIC
...
...
Server: cloudflare
Content-Encoding: br

if you have caching enabled, I want to assume you also have caching enabled for updates that are downloaded from the capgo bucket hosted on R2, correct? In other words, downloads from a presigned URL like https://9ee3d7479a3c359681e3fab2c8cb22c0.r2.cloudflarestorage.com/capgo/apps/d94748c9-e609-449e-84d1-30cbfbd4dde5/ee.forgr.demoapp/versions/1.0.0.zip are cached, correct?
if yes to 1.0 and 2.0, then Cloudflare's caching is going to interfere with the ability to resume downloads from the iOS/Android plugin unless caching is turned off for the capgo R2 bucket through cache rules.

For instance, with caching disabled, if a 100MB download is interrupted after 70% has been downloaded, when the download is resumed, the server will send the remaining 30MB. But with caching enabled, the server will always send the full 100MB any time an end user attempts to continue an interrupted download.

Do you still need this feature now that the partial update feature is ready for review?

@WcaleNieWolny it seems resumable is easier to do than partial.

So I had conversation with @neo773 that easier to do, so I want to add this now and partial when ready

I can assure it is not easier 😀. It is closer in complexity to the partial update feature as it will require opening PRs in this repo as well as the Cap-go/capgo repo.

For this repo, the background download logic will have to be refactored so that background downloads can survive network interruptions, end user interruption or termination by the operating system.

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

The payment on this issue completely underestimates the effort: it is currently too low.

neo773 commented 7 months ago

@ayewo

For the Cap-go/capgo repo, an additional endpoint will be added to add support for resuming a download from your R2 bucket since you currently use presigned URLs to authorise access.

Pre-signed URLs have content-length header so not sure why you need to an additional API endpoint I did some initial testing with MZDownloadManager and it did work, but couldn't manage to integrate the library since I'm new to Swift.

WcaleNieWolny commented 7 months ago

Can't you just add "Range: bytes=0-NN" header from iOS or android and then just get the r2 file?

I think you can download part of a S3 file https://stackoverflow.com/questions/36436057/s3-how-to-do-a-partial-read-seek-without-downloading-the-complete-file

ayewo commented 7 months ago

To support resumable downloads you need 2 headers. The Content-Length: header is one piece. The other piece is the Range: header.

There are 2 reasons why a new endpoint is required:

Capgo currently creates a presigned URL with a validity of 120s (2 min). If a download interruption lasts for more than 2mins, the presigned URL would have expired, so the plugin will need the Capgo server to generate another presigned URL. There is currently no endpoint that handles the renewal of presigned URLs and this is needed for downloads to be resumable.
In order to support arbitrary range seeking using the Range: header, the presigned URL must first encode the actual Range: for a valid URL to be generated. So, if a client wants to fetch Range: bytes=0-19 the Capgo server must encode that extra bit of information before returning a presigned URL. Here's an example using the AWS S3 library in JS: https://github.com/aws/aws-sdk-js-v3/issues/4823#issuecomment-1587749816.

As for why caching will need to be disabled for downloads from R2, it because we don't want Cloudflare to ignore the Range: header when it is included by the plugin. This Cloudflare forum discusion makes the case for why caching will need to be turned off.

WcaleNieWolny commented 7 months ago

Capgo currently creates a presigned URL with a validity of 120s

That is not true, capgo has a way longer validity due to redis caching the url

This used to be true, but is no longer the case

ayewo commented 7 months ago

That is not true, capgo has a way longer validity due to redis caching the url

lol 😂🤣!

This was true just 2 weeks ago so not sure what point you are trying to make. In any case your rebuttal doesn't invalidate what I wrote as to why another endpoint is needed.

You could have corrected me by saying Martin has since changed it from 120 to 604800 ... instead you chose to come out guns blazing like someone trying to win an argument (that I didn't start).

Anyway just take it easy as I can't possibly be on top of all the changes you and Martin are making✌️.

WcaleNieWolny commented 7 months ago

I was just trying to inform you about the change to the R2 validity time and I was trying to argue against creating a new endpoint

I am sorry if I did not made it clear enough

algora-pbc[bot] commented 7 months ago

@ayewo: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

ayewo commented 7 months ago

@algora-pbc yes, I know but without @riderx's input, I can't really forge ahead. Martin is working on fixing issues with billing (which is of higher priority).

neo773 commented 7 months ago

@ayewo Bruh, you’re replying to a bot 💀

ayewo commented 7 months ago

@neo773 I know, right 😁!

(I also know that the founders of Algora keep an eye on the bot's email notifications too 😉).