search

redsolver / skydroid

A decentralized domain-based App Store for Android.
GNU General Public License v3.0
389 stars 21 forks source link

Suggestion: Use .well-known URIs #11

Closed allo- closed 3 years ago

allo- commented 3 years ago

Not everyone can control the TXT records of his domain. Using a .well-known URI more people will be able to setup auto-discovery of their repos.

redsolver commented 3 years ago

I see 3 major problems with this suggestion:

allo- commented 3 years ago

As far as I understood, the developer should host his own packages and provide his own index. So getting the user's IP is just one updated package away and something the user has to expect.

Currently it's possible to fetch 64 and more metadata updates at once, this is not possible if every metadata file is hosted on a different server.

I am not sure how often this is needed. But 64 seems like quite a big number of apps to install. When I really only need to subscribe to the repositories of the apps I am using, one HTTP request per index update should be feasible. Maybe we're talking about different updates here?

Single point of failure: If the website is down or blocked, it's not possible to fetch the app metadata anymore.

Just as fetching the update itself. This is a problem, but the whole downtime is a problem. I would not expect this to be a problem too often.

Privacy and centralization: One major advantage of using DNS is that it's propagated everywhere, so you can choose to use any DNS server of your choice. If the metadata would be loaded directly from a specific domain, the developer/website owner could collect IP addresses and more.

Same as above. When I am to fetch the update from the same server anyway, there is little privacy advantage. And in fact DNS is unencrypted while HTTP without TLS is rare.

To make sure we're talking about the same thing, because I learned of your project today and did not know it before. How I understood the concept is:

Sending, for example, 64 HEAD requests to check if the meta data is changed, should be fast enough in most situations. This can even be reduced by accounting for not all apps being installed at the same day. When you only fetch the meta data only when the last meta date update is older than one week, today will be a few updates to fetch and tomorrow a few other updates.

Now what my request is about: There are many offers for cheap webhosting with a domain, which both do not allow to add arbitrary domain records and are targeted at people who do not want to adjust their DNS records themself. Adding .well-known files is just an upload, changing DNS records would require to buy a hosting package that allows for updating the records and understanding a bit more about DNS.

redsolver commented 3 years ago

The F-Droid collection for example currently adds 700 apps at once and the app metadata is checked for changes for every single app every time a user opens the app. This way it's possible for developers to publish app updates in 10-20 minutes. Using the current implementation with the Multi-DNS-over-HTTPS server (https://github.com/redsolver/multi-doh-server), updating all of these apps (their metadata) only takes ~ 10 small JSON requests (which use HTTPS). Also, I recommend using the Skynet CDN (the SkyDroid CLI uses it by default) to upload metadata and APK files (https://siasky.net/). (From the SkyDroid README: This enables fully decentralized file-sharing, because the uploader can use any portal or even directly upload to Skynet and also the user can choose between any of the portals!). As for the cheap webhosting with a domain: You can buy a domain name for less that 1$/year on Namecheap or even get a free one (https://www.freenom.com/). I also think most developers who want to publish their own app should know how a TXT DNS record works.

redsolver commented 3 years ago

Skynet solves the downtime and privacy issue, because a user can select any Skynet portal they want and trust (for example https://skyportal.xyz/ or https://siasky.net) and still have access to everything while only revealing his IP address to this one portal.

allo- commented 3 years ago

The F-Droid collection for example currently adds 700 apps at once and the app metadata is checked for changes for every single app every time a user opens the app

Why? A Linux Distribution does at most one package list update per day. Some only update the security list daily and fetch the other lists once per week. And I would have thought when you fetch a package list, it only are a few kilobytes difference if it contains the meta data for 7 or 700 packages?

As for the cheap webhosting with a domain: You can buy a domain name for less that 1$/year on Namecheap or even get a free one (https://www.freenom.com/).

Good offeres aside. Software developers may have some hosting or have reasons to choose a specific one. All of them allow uploading package lists, but not all of them allow changing DNS records. And I would try not to have a hard requirement for developers to know DNS. I would be able to setup the record even in an own DNS server, but I would not want to force others to learn how to do it, when there is an easier way.

But I think I still not get exactly what you are downloading. Are you really checking for updates every time the user opens an app for that specific app? I thought about package lists, e.g., like the lists Debian uses (and probably F-Droid, even when I am not familiar with their meta data).

redsolver commented 3 years ago

You can't compare the package list of a linux distribtuion or F-Droid with SkyDroid, because on SkyDroid every single app is published independently. On the average linux distribution the package manager just downloads one index file which contains all packages and is coming from one source, this is not possible when every package's metadata is coming from a different source.

The process and what exactly is downloaded it described in the README of this project, for example:

When a SkyDroid user wants to check for updates of all apps, only the TXT records need to be checked. Only if a new metadata hash is found, the metadata file is downloaded again and applied.

All TXT records are fetched using https://github.com/redsolver/multi-doh-server (You can see some example requests in the README there)

redsolver commented 3 years ago

Good offeres aside. Software developers may have some hosting or have reasons to choose a specific one. All of them allow uploading package lists, but not all of them allow changing DNS records. And I would try not to have a hard requirement for developers to know DNS. I would be able to setup the record even in an own DNS server, but I would not want to force others to learn how to do it, when there is an easier way.

I agree, but the trade-offs (performance, privacy, scalability) I listed in this thread are a lot bigger than the advantage of allowing developers to use some cheap web hosting providers. So no, there is unfortunately not an easier way in my opinion.

It's however totally possible to create a third-party service for developers which simply provides developers with free subdomains for their apps and sets the DNS records automatically, so developers who don't know DNS can also publish their apps.

allo- commented 3 years ago

I understand, that you have for N apps up to N repositories. I only wonder, why I should have too many repositories, just because F-Droid includes a lot of apps.

Let's say I want to install 10 Skydroid apps. 2 from F-Droid, 2 from publisher A, 3 from publisher B and one each from C, D, E.

Then I would expect I need to fetch 6 indexes when updating the package lists. The one from F-Droid would probably be a few hundereds of kilobytes (uncompressed), the other ones less than 10 kb and in sum maybe 500 kb of traffic and connections to 6 servers. To me this seems like something that is reasonable both for me checking for updates and for the servers hosting the meta data (assuming that clients behave nicely and use Cache headers to avoid fetching unchanged files). Let's say I am an open source enthusiast and install 15 more apps from 10 more sources, then it may be 600 kb now and 16 servers, what is a matter of seconds on wifi and feasible on mobile data connections, when the user really wants to refresh the index while using mobile data.

Hosting my own app could then be literally by filling out some meta data template and just uploading it to the right location, when I am a novice programmer who just wants to host his app without thinking about more complicated processes which would allow to host hundreds of apps for thousands of users, what is not his use-case (yet).

I'd just would like to keep it simple, so the approach works for as many users and developers as possible by making it as easy as possible. And I still do not see why it is a problem to use more or less a "F-Droid with many repos" approach with an easy mechanism to add apps that have their own repo. On the other hand, you seem to try something much more ambitious what's a good thing, when it works out for developers and users.

redsolver commented 3 years ago

Hosting my own app could then be literally by filling out some meta data template and just uploading it to the right location, when I am a novice programmer who just wants to host his app without thinking about more complicated processes which would allow to host hundreds of apps for thousands of users, what is not his use-case (yet).

"uploading it to the right location" in this context means having a fully working web hosting package and knowing how to upload files there. If someone never did that before, I argue that only buying a domain and copy-pasting a DNS record is a lot easier. And when the developer wants to publish a new app, they have to get a new web hosting package with a new domain name - that's not really easy. (Every domain name MUST contain only one (1) app. If you want to offer multiple apps using one domain, you need to create subdomains)

allo- commented 3 years ago

Not only your installed apps are refreshed

Why?

I for example have a total of 1200 apps installed

On one Android device? I may have like 150 or less.

while being fully decentralized and fast

I guess this is the main point of your scheme. But decentralized as in "the developer hosts his app" seems (another) good approach.

SkyDroid wants to go away from the model where there are big repositiories which host a lot of apps from different developers.

I understand and fully support the point. That's why I am looking into your project.

You did not really address the issues with privacy and downtime, which are fully solved with the current architecture

Indeed. My personal opinion on this is mostly, that a developer hosts a repository and I load the meta data anonymously. Not as in anonymous IP (but nothing prevents me from using tor), but as in not needing to login to any service but just fetching data without leaving personal details. A mirror of a Linux distribution sees me doing this as well and people even argue over using https for mirrors of signed packages. Downtimes are downtimes. Who needs to serve many people or have high reliability should probably use a high-available setup for the web server serving both meta data and packages.

And when the developer wants to publish a new app, they have to get a new web hosting package with a new domain name

I think we're still not thinking about the same process.

This might not scale very well for 1200 apps from, e.g., 1000 sources. But for 150 from 50 sources it would probably work fine.

Every domain name MUST contain only one (1) app.

I don't understand why. Just show the user a list of all apps available from that domain.

redsolver commented 3 years ago

Every domain name MUST contain only one (1) app.

I don't understand why. Just show the user a list of all apps available from that domain.

It reduces complexity, because it allows you to refer to an app (for example skydroid.app) using its domain name and then directly link to it, for example: http://to.skydroid.app/skydroid.app It basically ensures a globally unique id for the app.

Advantages of the current approach (which don't work well with the .well-known method):

Disadvantages of the DNS-based approach:

In my opinion, the advantages clearly outweigh the disadvantages and I'm closing this for now. If you find more advantages, feel free to re-open.