colenso commented 1 year ago

Would help if the OTA updates could be downloaded as individual files rather than one big .zip file. Especially helpful with app having a bigger OTA update size and users downloading the OTA on Mobile Data.

anbraten commented 1 year ago

As the client apps are sending there current version to the server there could also be some system generating diff zips. So in case the user had for example the previous version and now updates to the latest version he could just get a to file large diff containing the two updated files. If another user is just installing the app he would get the full zip. The diffed zips could be cached based on from and to version identifiers.

Or something like a diff -ruaBN previous new > test1.diff

riderx commented 1 year ago

@anbraten since user can be in any current previous version, that would mean produce a diff zip for all previous version, plus Capgo will need to understand what is a diff in your build who seems very complex since project can be react/svelte/angular/vue/js/etc a believe for now a easier upgrade would be using better the background job feature from iOS and android. to have more than 30 sec in background to finish the work.

anbraten commented 1 year ago

I guess creating the diff would be somehow possible without knowing what the details are, similar to how git, diff work, but sure the server side "calculation" whats the difference between two version would require some computation power even if the result of an upgrade / downgrade between two versions would be cached.

Doing that in a background job seems to be a nice idea in general.

One benefit of sending something like a diff, could be that you save a lot of network bandwidth which would be nice for the client, but also for your server if you really have a lot of clients.

Just wanted to do some brain dump in general as I read this issue 😃. I really like your lib and system.

riderx commented 1 year ago

I totally agree if i found a easy way to do diff i would love to have that, sending duplicated file is not nice

csotiMatyas commented 1 year ago

A comment on this, i was also thinking about sending just some files or folders in the bundle folder. My problem is having an asssets folder being around 1mb, a total waste to send on every download. So i was looking around in the code trying to figure out how the bundle swap happens, so maybe instead of a bundle "replace" i could just make a "copy and replace all" so i could just send what i want, and just override the needed folders/ files. But i can't really find the place where it happens on android.

riderx commented 1 year ago

the files are saved in a new folder and then the capgo define the capacitor web view new default location to it, so for your idea to work, it needs to copy previous app in new folder and then override certain files, the problem is: It's difficult to be sure to have one solution to fit all, since some people in web use unique file name to flush cache

riderx commented 1 year ago

/bounty $500

0x0elliot commented 1 year ago

hey, i would like to help out with this issue. would you mind giving me some context and assigning me to it? :)

riderx commented 1 year ago

Hey @0x0elliot i did assign you ! Currently, the updates are sent from the CLI as a single zip and downloaded by Android and iOS devices as single file. This is causing issue for big update (more than 10 MB). So, we have a high level of fail update from 10% to 50% for app used in location where internet is less stable.

What needs to be done:

Update the CLI to send multiple zip/files instead of one to store.
Update the backend to understand multi files by update.
Update the plugin code in android to download multi files updates.
Update the plugin code in IOS to download multi files updates.

To note:

There is a current end-to-end encryption system who makes this job a bit harder

The job is pretty big, and I believe it has to be split to be done. Are you interested in doing any of this task?

0x0elliot commented 1 year ago

i am interested. if you would be comfortable, we can split the bounty into two tasks and 4 PRs:

Part 1

Update the backend to understand multi files by update.
Update the CLI to send multiple zip/files instead of one to store.

Part 2

Update the plugin code in android to download multi files updates.
Update the plugin code in IOS to download multi files updates.

In chronological order.

I am of course very new to this. Would snoop around in the discord to figure it out properly.

Immediate bottlenecks

To begin with, some questions, which project exactly is the backend?

to help save some time, would you mind helping me figure out which files should i look into? i am thinking of creating another API endpoint (or the equivalent in this case) for this feature as that's the cleaner way to proceed further.

riderx commented 1 year ago

yes, of course, I can adjust the bounty accordingly. you are right, the backend is in the repo https://github.com/Cap-go/capgo in the Supabase folder. I have to create a reproducible env i believe for this task

0x0elliot commented 1 year ago

okay. let me know of the .env file then. why would you require it?

riderx commented 1 year ago

if you want to be able to try end to end, otherwise you can run in local

0x0elliot commented 1 year ago

been trying to set it up locally. it has surely been a bottleneck to set it up. would be less of one if you can manage to create one with some instructions so that i can just focus on the issue.

0x0elliot commented 1 year ago

having quite a difficult time with migrations here :') @riderx this should be a one time effort help though!

Tallyb commented 1 year ago

Some notes: Diff should be made on the md5 of the file, as this is a reliable way to know it has changes. Lost of modern tools use hash when building web output, e.g. webpack, for caching reasons, so file name is likely to be different.

riderx commented 1 year ago

Hey @0x0elliot sorry for the delay it's now way more easy to have local env, i made the seeb db and migration file. https://github.com/Cap-go/capgo#start-local-supabase-db

ayewo commented 1 year ago

Hey @riderx

Still open? I'm thinking I might want to take a stab at this ...

riderx commented 1 year ago

It's still open yes !

ayewo commented 1 year ago

@riderx Great!

I would like to tenatively adopt the same approach proposed above:

i am interested. if you would be comfortable, we can split the bounty into two tasks and 4 PRs:

Part 1

Update the backend to understand multi files by update.

Update the CLI to send multiple zip/files instead of one to store.

Part 2

Update the plugin code in android to download multi files updates.

Update the plugin code in IOS to download multi files updates.

Before I start work on Part 1, I'd like to assess if the scope is commensurate with the bounty on offer. Once I'm done with my initial assessment, I'll ping you here or in Discord so you can officially assign me to the issue.

ayewo commented 1 year ago

... or negotiate the terms before being assigned.

riderx commented 1 year ago

ok no problem mate, just to let you know as the same time someone else contacted me in private and is also doing a proposal on how to do the work

ayewo commented 1 year ago

@riderx Haha, no problem. Thanks for the heads up. (Sounds like that person was subscribed to be notified of any activity on this issue 😀).

Quick question

When I tried to stand up the DB using supabase start as per your README instructions, I get the following error related to migrations:

Applying migration 20230321023005_create_base_db.sql...
ERROR: schema "supabase_migrations" already exists (SQLSTATE 42P06)
At statement 242: --
-- Name: supabase_migrations; Type: SCHEMA; Schema: -; Owner: postgres
--

CREATE SCHEMA supabase_migrations

Seesm like the latest Supabase docker image now ships with the supabase_migrations schema already in place which is why there's a conflict?

ayewo commented 1 year ago

@riderx Nevermind, I figured it out 😎

You can assign me to the issue 🙂.

AitorG commented 1 year ago

@ayewo could be nice, as a first version, to update only those files included in zip and do not delete others in order to save memory and transfer data (it could be setted by config value), just saying :)

ayewo commented 1 year ago

@AitorG yea it would nice, but I haven't been assigned to this issue, yet 😀.

Someone else proposed something similar above to which @riderx replied with the following comment:

the files are saved in a new folder and then the capgo define the capacitor web view new default location to it, so for your idea to work, it needs to copy previous app in new folder and then override certain files, the problem is: It's difficult to be sure to have one solution to fit all, since some people in web use unique file name to flush cache

ayewo commented 1 year ago

@riderx

I've taken a good look and I now understand what needs to be done here. Essentially, will need to update 3 repos:

the cli to support multiple zip files;
the capgo backend and the supabase DB;
the capacitor-updater native code for each of iOS and Android.

Then there is the encryption which also needs to be handled for this task to be complete.

For comparison, this issue #119 requires significantly more effort than #72 where only the capacitor-updater repo has to be updated but you priced both issues the same 🤷.

riderx commented 1 year ago

Yes you are right, I didn’t price it well. I will update it

Martin

Maker of Capgo https://capgo.app/ Capacitor Live updates Podcast host at SOLOS https://solos.ventures/ Maker of Captime https://captime.app Crossfit timer app Mobile app Builder at Forgr https://forgr.ee/

On August 9, 2023, Martin DONADIEU @.***> wrote:

@riderx https://github.com/riderx

I've taken a good look and I now understand what needs to be done here. Essentially, will need to update 3 repos:

the cli https://github.com/Cap-go/CLI to support multiple zip files;

the capgo https://github.com/Cap-go/capgo backend and the supabase DB;

the capacitor-updater https://github.com/Cap-go/capacitor- updater native code for each of iOS and Android.

Then there is the encryption which also needs to be handled for this task to be complete.

For comparison, this issue #119 https://github.com/Cap-go/capacitor- updater/issues/119 requires significantly more effort than #72 https://github.com/Cap-go/capacitor-updater/issues/72 where only the capacitor-updater repo has to be updated but you priced both issues the same 🤷.

— Reply to this email directly, view it on GitHub https://github.com/Cap- go/capacitor-updater/issues/119#issuecomment-1671209104, or unsubscribe https://github.com/notifications/unsubscribe- auth/AA7FGL2UB5T5U5PM5D6G5OLXUN437ANCNFSM6AAAAAASAQXRBY. You are receiving this because you were mentioned.Message ID: <Cap- @.***>

tobihans commented 1 year ago

@riderx Haha, no problem. Thanks for the heads up. (Sounds like that person was subscribed to be notified of any activity on this issue 😀).

Quick question

When I tried to stand up the DB using supabase start as per your README instructions, I get the following error related to migrations:
Applying migration 20230321023005_create_base_db.sql...
ERROR: schema "supabase_migrations" already exists (SQLSTATE 42P06)
At statement 242: --
-- Name: supabase_migrations; Type: SCHEMA; Schema: -; Owner: postgres
--

CREATE SCHEMA supabase_migrations
Seesm like the latest Supabase docker image now ships with the supabase_migrations schema already in place which is why there's a conflict?

@ayewo I'm facing the same issue, would you mind giving me a tip on how you solved it? Thanks

AitorG commented 1 year ago

@AitorG yea it would nice, but I haven't been assigned to this issue, yet 😀.

Someone else proposed something similar above to which @riderx replied with the following comment:

the files are saved in a new folder and then the capgo define the capacitor web view new default location to it, so for your idea to work, it needs to copy previous app in new folder and then override certain files, the problem is: It's difficult to be sure to have one solution to fit all, since some people in web use unique file name to flush cache

Yeah I understand, thats why I proposed to use a config value. If that config is attached to the version, you would be able to sometimes upload an entire version and clean up previous version with some files repeated (files like JS with hash name to prevent cache)

ayewo commented 1 year ago

@tobihans There are some many little changes you'll need to make it work haha 😀. It was a lot of trial and error, reading lots of code, sprinkled with a ton of patience to get it working.

I'm busy now so can't really promise anything but I do hope to submit a PR -- separate from this issue -- that tidies up the documentation so it's easy to get started.

Perhaps take a look at other issues that are open?

@AitorG I'll have to defer to @riderx on that.

tobihans commented 1 year ago

@tobihans There are some many little changes you'll need to make it work haha grinning. It was a lot of trial and error, reading lots of code, sprinkled with a ton of patience to get it working.

I'm busy now so can't really promise anything but I do hope to submit a PR -- separate from this issue -- that tidies up the documentation so it's easy to get started.

Yes, I was working on https://github.com/Cap-go/capgo/issues/134, which requires me to do the setup as well.

Thanks for the reply. I'll dig into the migrations to see what's happening.

riderx commented 1 year ago

don't forget @ayewo to do the command / attempt #119 to show you started working on it. I will do a bounty for the dev setup as well

riderx commented 1 year ago

You have it here: https://github.com/Cap-go/capacitor-updater/issues/238

ayewo commented 1 year ago

@riderx sure 👍

/attempt #119

Options

Cancel my attempt

triforcely commented 1 year ago

Hey @ayewo ,

I was the "incognito" contributor that @riderx mentioned before, and no, I wasn't subscribed to this thread, I have emails with dates to prove, haha 😁

I won't be available for the upcoming couple of weeks, so the issue is fully yours. I thought that I would share some findings/ideas that we've come up with @riderx - it is how I was thinking about approaching the problem with some corrections from @riderx. Of course, you are free to do it your way; it would be a waste of time not to share my findings.

Good luck!

Notes

**New file format for OTA updates** Some resources about how others handle this issue · https://docs.expo.dev/technical-specs/expo-updates-1/ · https://docs.expo.dev/eas-update/how-it-works/ · https://docs.expo.dev/eas-update/deployment-patterns/ We need to introduce a new (optional) file format based on the bundle manifest to support incremental updates. The regular full-size bundle will be uploaded first when uploading the app update bundle - this is required to ensure that: · Regular, complete APP updates are continuing to work to support older clients. This ensures that the API changes that we make are not breaking. · Clients who do not enable incremental package updates are not interrupted. Bundle manifest file will be generated and uploaded to the Capgo (or generated by any other means when not using Capgo) This manifest is required so the clients can decide which files must be downloaded. This manifest could look like this: { "index.html": { "hash": "", "url": "" }, "js/main-18skch2.chunk.js": { "hash": "", " url": "" } // and so on }In case somebody is self hosting, they can provide their own urls in the manifest. Based on the manifest, the client can decide the following: · Which files should be kept (cloned) from the active bundle · Based on the file name · Based on checksum (to avoid re-downloading large static resources) · Which files are new and have to be downloaded The CLI will handle this new bundling behavior out of the box. https://api.capgo.app/updates endpoint is going to be extended with the „manifest” property but everything else is going to be kept the same to not break existing versions of the plugin. **Extending capacitor-updater** The new capacitor-updater will need to handle new, manifest base bundles gracefully. To support this case following changes should be implemented for Android and iOS: · When checking if updates are available, it should look at the „manifest” prop first · When exists: continue "incremental download" logic · When not exists: continue "full bundle download" logic · The "apply update" path is updated to handle both scenarios. · In case of issues with the bundle, e.g. checksums of downloads are different than what the manifest has stated, we should fall back to the full bundle download process. · There may be transport issues or a packaging mistake from the plugin users' side. const version = await CapacitorUpdater.download({ manifest: { // full manifest object here } }) await CapacitorUpdater.set(version); **Extending CLI** CLI should optional support incremental updates. To do: · Update the regular package bundle to support additional flags to push incremental update files. · Add new command to transform the raw web bundle to a structure compatible with capacitor-updater incremental updates, generate a file manifest and upload the manifest with separately encrypted parts of the bundle Extending Capgo Capgo backend should be updated in a way that is compatible with existing clients. To do: · Implement changes required to support new requirements of CLI and capacitor-updater

WcaleNieWolny commented 1 year ago

Hey, is this still up? I have some experience in setting up S3 with the CLI so I belive I can start by handling sending updates by the CLI and then move onto the updater plugin. I have some ideas how to make this work and I would love to work on this. Here are some of my ideas:

I believe setting up manifest as JSON is a bad idea. If we would to do the manifest something like this: app_versions	id	manifest
10	{"index.html": { "hash": "", "url": "" }, "js/main.js": { "hash": "", " url": "" } }

then diffing 2 versions would not be possible. How do you figure out what files have already been uploaded?

I believe a better solution would be to setup our mantifest like this:

app_versions	id	manifest
10	array of ids from manifest_entries

manifest_entries	id	file_name	checksum	url	size (?)	storage
10	main.js	something	URL	129	r2 (see note)

about size I am not sure, but I do think it is going to be needed to generate the metadata for the version and for storage, we might not need this column as I do not believe

Then when we attempt to upload new versions we would create a tree of files to upload like this:

main.js:
    checksum: something
dir/haha.txt:
    checksum: different
nested/deep/file/body.html:
    checksum: checksum

(This will be transformed into json)

and we send this to some edge function (either an existing one or a modified one) and the backed would attempt to construct the manifest by running the following steps:

Iterate over all of the files send by the CLI
check if checksum of said file exists in manifest_entries
if it does create an entry like this:
```
main.js:
needUpload: false
url: null
```
and it would add id of said manifest_entries into the manifest of the version
else if it does not:
```
main.js
needUpload: true
url: upload_url
```
then the server would create a new manifest_entry and insert it into the manifest

all of the entries are send to the client (CLI)
then the client uploads all the missing files into S3
after an upload to S3 we can have an edge fn run to set the size of a file, however we can also just trust the CLI to send both size and checksum

This however creates some problems:

Deleting version would not be easy. If the have version A and version B where version B depends on files from version A we need to make sure we do not delete these files from S3 We would achieve this by running an SQL query (as SQl is a very powerful tool) that selects all manifest_entries from version A that are used ONLY in version A After this we would either delete them from S3 and send another query to remove them from supabase or if possible we would select and remove them in the first query, essentially running 1 query to delete data from S3

For old clients (IOS/Android plugin) without progresive deploy we still need the zip file so we still likely need to upload it into S3 which sucks as it creates duplicates of the data

My solution is definitely not perfect, but nevertheless I would love to attempt this. Please let me know if someone is working on this.

ayewo commented 1 year ago

@WcaleNieWolny yes, I’m still working on this.

Your proposed solution is a bit too complicated though.

For instance,

For old clients (IOS/Android plugin) without progresive deploy we still need the zip file so we still likely need to upload it into S3 which sucks as it creates duplicates of the data

I’m proposing to solve this in a simple way: since this feature will likely require a new flag in capacitor.config.js which we could call ”enablePartialUpdates” = true, if this config is set, the plugin could use a newer version of the updateURL for updates.

So, existing apps will continue to work with the current updateURL of https://api.capgo.app/updates[^1] but new apps that enable the feature will use v2 of that URL i.e. https://api.capgo.app/v2/updates

[^1]: Typing on my phone so my recollection of the URL may not be accurate.

WcaleNieWolny commented 1 year ago

That does not change the fact that we still need to keep zip file and individual files. I understand that this is going to have to be enabled in the config. However how do we handle the migration to this feature? If there are already versions (of the updater plugin) deployed without this partial upadate and versions with them we need to have both files. Then when everyone migrates to the version in the app store that has this new feature you will be able to disable the old zip files

WcaleNieWolny commented 1 year ago

Imagine a situation like this:

This features go live
Someone enables it in an existing codebase
He/she uploads the new version to the app store/google play
He/she deploys a small change (for example a css change)
People that have still not migrated to the latest store version (ones without individual file download support) have no way to get the latest css change unless a zip file is provided. The old endpoint would essentially have to lie and tell that there are no updates available

WcaleNieWolny commented 1 year ago

I would be happy to work together on this. Let me know if that is okay with you

ayewo commented 1 year ago

Imagine a situation like this:

This features go live

Someone enables it in an existing codebase

He/she uploads the new version to the app store/google play

He/she deploys a small change (for example a css change)

People that have still not migrated to the latest store version (ones without individual file download support) have no way to get the latest css change unless a zip file is provided. The old endpoint would essentially have to lie and tell that there are no updates available

That’s easy, the CapUpdater plugin will continue to work as it currenty does today: it will download the full zip file from the updateURL. (Note I never said anything about retiring the current URL).

Newer apps OTOH can enjoy bandwidth savings on cellular by only downloading a partial update. Large updates tend to fail on cellular so bandwidth savings is what motiving this feature.

As I said, no need to overthink this when a simpler solution exists.

WcaleNieWolny commented 1 year ago

Okay, but are you satisfied with the rest of the proposal? How do you intend to implement this?

riderx commented 1 year ago

@ayewo i would prefer to have the same endpoint and just send in the body that you allow manifest in the request: likemanifest: true the the backend answer with manifest.

ayewo commented 1 year ago

@riderx yea, sure. I thought about it after I posted that.

ayewo commented 1 year ago

I would be happy to work together on this. Let me know if that is okay with you

@WcaleNieWolny I'm backed up a little bit so wondering: are you still game 😉?

If yes, I'll share a spec for the 3rd pull request needed to wrap up this feature.

riderx commented 8 months ago

/tip 100 @ayewo

algora-pbc[bot] commented 8 months ago

🎉🎈 @ayewo has been awarded $100! 🎈🎊

WcaleNieWolny commented 8 months ago

I will be attempting this

https://github.com/Cap-go/capgo/pull/601 https://github.com/Cap-go/CLI/pull/236

/attempt #119

Algora profile	Completed bounties	Tech	Active attempts	Options
@WcaleNieWolny	62 Capgo bounties + 2 bounties from 1 project	TypeScript, Java, Rust & more	Cap-go/capacitor-updater#266	Cancel attempt

riderx commented 1 month ago

It's finally done you can try it with version 6.3.0 and Capgo cloud, use the bundle upload with --manifest option

Cap-go / capacitor-updater

Download the OTA update as individual files #119

Part 1

Part 2

Immediate bottlenecks

Part 1

Part 2

Quick question

Quick question