search

termux / termux-packages

A package build system for Termux.
https://termux.dev
Other
13.23k stars 3.04k forks source link

End of service for Bintray in May, 2021 #6348

Closed imprakharshukla closed 3 years ago

imprakharshukla commented 3 years ago

Problem description Jfrog's Bintray which Termux uses as the primary hosting, is ending support in May of 2021. Here is the official announcement from the VP of dev relations for Jfrog https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/.

Bintray/JCenter users should start migrating to a new hosting solution.

What new hosting solution will Termux switch to?

RossComputerGuy commented 3 years ago

I've noticed Linode is a cheap solution for running a VPS, ~$5/mo but traffic may cause the price to go up.

imprakharshukla commented 3 years ago

@RossComputerGuy $5 isn't the actual price because Bintray had allocated around 32 TB / month, which would account to being approximately $350 in Linode's outbound transfer. This doesn't come with a CDN as opposed to Bintray which makes it terribly slow.

Using a dedicated object storage like S3, DO Spaces, wasabi or blackblaze will require an app update, therefore will render the Play Store version useless. Plus it will shoot the prices well beyond a $1000.

ghost commented 3 years ago

In Feb. 28:

No more submissions will be accepted to Bintray, JCenter, GoCenter, and ChartCenter The GoCenter and ChartCenter websites will be disabled (client requests will still work)

So this means that package updates will be stopped in Feb 28?

imprakharshukla commented 3 years ago

@WMCB-Tech That is very confusing. They could also mean that no new user would be able to register for these services or just that all the rest and client requests will be blocked. I think former. How unprofessional of them to give such a short window for the migration.

ghost commented 3 years ago

Termux has more than 1000+ packages available though, well it's not a big deal since we have mirrors

kcubeterm commented 3 years ago

By the way I got some good results with ipfs at some extent but don't know what happen if lots of user will be there. So I will share links once I managed to get atleast 10 node.

ghost commented 3 years ago

So this means that package updates will be stopped in Feb 28?

Yes, this means any data submissions (package uploads) will be stopped.

By the way I got some good results with ipfs at some extent but don't know what happen if lots of user will be there.

@kcubeterm I can give a "stress test" to your setup by sending traffic from https://main.termux-mirror.ml to IPFS repo for some time.

So I will share links once I managed to get atleast 10 node.

Hopefully these 10 nodes are not 10 (free) accounts on Pinata/Temporal... not saying that this is (likely) against their ToS but also effectively only 2 nodes.

ghost commented 3 years ago

Ok, then I will manage to upload all latest available updates and then disable Github Actions uploads on next week to prevent any potential inconsistent state on Bintray repo (so it will continue to be available in RO mode until shutdown). We still have mirrors as shown in https://github.com/termux/termux-packages/wiki/Mirrors, so hosting by itself is not a problem currently but package submission workflow likely will be switched to manual.

@a1batross Are you continue hosting Termux mirror at https://termux.mentality.rip? I have noticed that last sync was on 26 Dec 2020.

a1batross commented 3 years ago

@xeffyr sorry, I forgot to move mirror.list config from old server so I rewrote it from scratch. Now it's mirroring {termux,game,science,unstable,x11}-packages. :)

insign commented 3 years ago

Maybe we (community) can seed like a p2p network, but or create public mirrors based on checksum verifications?

Harshiv-Patel commented 3 years ago

Well if every node needs to have the whole 20-50gigs worth of data, the torrent like p2p would be quite expensive for end users, Though I'm interested in the idea. I guess it'd have to be something where people can join and leave seeding parts of it from time to time.

The whole thing looks far more complicated the more I think about it.

a1batross commented 3 years ago

@Harshiv-Patel termux repos are less than 10GB.

The problem is not the distribution though but publishing. Bintray was pretty good at this.

kcubeterm commented 3 years ago

Alright, here's IPFS repo. Based on P2P. Those who want to host few or whole repo can pin content from here. http://ipfs.io/ipns/k51qzi5uqu5dhght6oyh7c83cqfbssco7xx7yti0me15eoky0eyppd2go9scgw

ghost commented 3 years ago

As for testing, https://main.termux-mirror.ml is also on IPFS now. Can be accessed either directly (CloudFlare gateway) or with https://ipfs.io/ipns/main.termux-mirror.ml (IPFS default gateway).

Reverted due to issues:

ghost commented 3 years ago

Those who want to host few or whole repo can pin content from here.

And will need to re-pin it any time the repo is changes.

kcubeterm commented 3 years ago

Repository inconsistency: few deb files missing

Most probably, this issue should be fixed after fresh Installation. I have published newer cid on same ipns

Accessibility (stability?) issues: traffic has been reduced by 70% which means users have problems with downloading packages.

Not sure about it, I have installed many packages and there's no issues, latency is also better than before.

ghost commented 3 years ago

https://main.termux-mirror.ml/ is now served over IPFS (redirect to ipfs.io). Higher latency is expected.

Direct link: https://ipfs.io/ipns/k51qzi5uqu5dlolg4k8j0lyiuexr9tpmz4gy5a8w82czjdmr62h3cfw6mb2s8c

ghost commented 3 years ago

@kcubeterm I'm switching to custom key for IPNS which is detached from the node id. As result link has been changed now. Unstable and X11 repositories will soon be in IPFS and each will have own link.

If you decide to use custom keys instead of node-specific key pair, here is a quick tutorial:

# Generate new
ipfs key gen termux-main-repo

# Publish IPNS
ipfs name publish --key=termux-main-repo <CID>

Mirrors:

IPNS is expected to be permanent now.

INFO: I'm running only one node and host it on my PC (otherwise there would not be point in doing this hassle with IPFS). In order to deal with resource usage on my side (ipfs uses about 50% of CPU continuously!) and to ensure that most traffic is served by gateways mostly, node will be down for 11 hours each day which is a bit lower than lifetime of IPNS records. Obviously that only cached content will be served during downtime.

kcubeterm commented 3 years ago

Alright, I am aware of that tutorials, but what can I do here, since it's your key only you could publish unless you share that key. From now I will only pin repo with your CID (since it's change always so first my node will resolve ipns) So ensure don't change your ipns link next time otherwise I have to manually put it into all nodes.

ghost commented 3 years ago

@a1batross Your mirror looks messed up. There were reports about checksum mismatch from some users and some repositories are empty, e.g. https://termux.mentality.rip/termux-root-packages-24/

If you are using apt-mirror try to remove its cache (like /var/spool/apt-mirror/*, not repository data) and force re-sync.

a1batross commented 3 years ago

@xeffyr just did that and changed all unix permissions/ownership.

I joined #termux IRC channel, so if there will be any problems you can ping me there.

Will try to set up mirroring from IPFS this evening or tomorrow. Never worked with IPFS before.

finagolfin commented 3 years ago

Honestly, this is a good thing, as I never liked that Termux was so dependent on that company for free bandwidth. I agree with @insign, we should be able to set up a p2p network between the app installs, seeded by a single cheap VPS that provides the freshly built packages and checksums. The only question is whether we can easily repurpose some existing apt/torrent code out there for this or if it will require some custom code written by us.

kcubeterm commented 3 years ago

@buttaface We have tested ipfs and got good results. Right now(from yesterday) whole termux traffic are hosting from ipfs and no one reports any error regarding this. We have 12 nodes right now. And that is sufficient but since it's P2P more contributions are welcome.

finagolfin commented 3 years ago

I'm not familar with how ipfs can be used for this: who's running those 12 nodes and are they in data centers? In that case, it's federated and not true p2p like bit torrent, but that's fine for now if it works. If you put up instructions on setting up IPFS nodes for Termux, I'm sure more people will do it.

ghost commented 3 years ago

Everyone can set up a local ipfs node and use it as local gateway. Thus sources.list will look like

deb http://127.0.0.1:8080/ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx stable main

Local node will re-provide content which has been already cached.

How to set up a local IPFS node/gateway

  1. Install ipfs package 
    pkg install ipfs
  2. Initialize the node. This will create ~/.ipfs directory with basic data structure and configuration file. 
    ipfs init
  3. Start daemon. You can run it either in a separate session, in tmux, etc. 
    ipfs daemon

    Termux ipfs package currently lacks support of termux-services but this will be fixed.

Also you need to forward port 4001, both tcp and udp, or enable relay discovery:

ipfs config --json Swarm.EnableRelayHop false
ipfs config --json Swarm.EnableAutoRelay true

Better to visit https://docs.ipfs.io/ to learn what IPFS actually is and how to work with it.

For now, IPFS setup is centralized as everything goes through single gateway node (https://ipfs.io). If even 1/1000 of users will use a local gateway instead, this will make Termux less dependent on third-party gateway and will reduce load on our seed nodes.

Though it is better to make a full copy of seeded content.

ipfs pin add /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx

# Advanced variant, with garbage-collecting stale data.
ipfs pin rm /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx
ipfs pin add /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx
ipfs repo gc

and repeat this on periodic basis to ensure that pinned content is up to date.

kcubeterm commented 3 years ago

And those who is behind NAT and can't forward TCP/UDP port. Enable autorelay https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#autorelay

ghost commented 3 years ago

@kcubeterm autorelay instructions are already specified in my comment.

kcubeterm commented 3 years ago

@xeffyr 🤭🤭

ghost commented 3 years ago

There however is one issue with user setup (local gateway): ipfs daemon should run continuously. If user just using Termux for 5-10 minutes and then close, it will be useless. 2 - 3 hours minimum, with consistent Internet connection, especially if using relays.

ghost commented 3 years ago

There are variants with using apt-p2p but I don't think it will be better than IPFS.

kcubeterm commented 3 years ago

Running ipfs daemon as user perspective is not good idea. as you mentioned it gets killed and it will eat extra bandwith as well. Only thing I'm worrying about is gateway. By the way that was good idea as pkg choose mirrors in same way it will choose different gateway also

kcubeterm commented 3 years ago

Here's list of few public gateway https://ipfs.github.io/public-gateway-checker/

finagolfin commented 3 years ago

@xeffyr, thanks for the instructions on how to set up a true p2p ipfs node for the Termux packages. But who is running those 12 nodes now? I assume they're not running in Termux, as they wouldn't have enough bandwidth to take over for the bintray shutdown.

ghost commented 3 years ago

as you mentioned it gets killed

I have not mentioned that it "gets killed" exactly. But certain amount of cpu time, ram and bandwidth will be used.

P2P is useful when clients contribute too and there no way to avoid bandwidth use. Currently we pushed all traffic problems to ipfs.io which really isn't good.

Here's list of few public gateway

I know and will update pkg.

Not all gateways can be used. Some gateways looks like able to filter traffic somehow.

ghost commented 3 years ago

But who is running those 12 nodes now?

I run one node. Rest is by @kcubeterm and few other people.

landfillbaby commented 3 years ago

idea: what if each PPA repo, was hosted as a branch on the respective GitHub repo? would GitHub allow that? someone even wrote a little tutorial for it couple of years ago

ghost commented 3 years ago
  1. Repository size limit. It's soft, but going beyond will trigger interest of Gihub staff.
  2. Traffic limit of 100 GB per month. Again, soft.

Hosting binary files on Github can lead to terms of service violation.

Opengapps project had problem with Github due to these limits, https://opengapps.org/blog/post/2019/02/17/github-situation/.

landfillbaby commented 3 years ago

ah. that's a shame

Luflosi commented 3 years ago

I propose creating a collaborative IPFS cluster, similar to https://github.com/RubenKelevra/pacman.store. This would make it much easier for people to pin everything on their IPFS node without the frequent ipfs pin rm and ipfs pin add suggested in https://github.com/termux/termux-packages/issues/6348#issuecomment-778279980. It would also increase the visibility of this effort if it is listed on https://collab.ipfscluster.io.

SDRausty commented 3 years ago

free unlimited bandwidth web hosting debian

ajhalili2006 commented 3 years ago

Will package installs after May 1 on legacy environments break?

Grimler91 commented 3 years ago

@AndreiJirohHaliliDev2006 The main legacy repository is not hosted on bintray so will not be affected. The science and game-packages repos are though. I will set up a mirror for them elsewhere, but users might need to manually adjust the URL to be able to use them

m-p-3 commented 3 years ago 
ipfs pin add /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx

# Advanced variant, with garbage-collecting stale data.
ipfs pin rm /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx
ipfs pin add /ipns/k51qzi5uqu5dg9vawh923wejqffxiu9bhqlze5f508msk0h7ylpac27fdgaskx
ipfs repo gc

and repeat this on periodic basis to ensure that pinned content is up to date.

Okay, so you gotta make a cron job, as IPFS doesn't have any mechanism in place to monitor and keep an /ipns/ path synced for now? And there's no way to do it without purging the entire data and redownloading the whole repo each time, if I understood correctly?

Luflosi commented 3 years ago

If you do it this way, IPFS will only download new data but there will still be a lot of disk activity as IPFS will go through its tree of hashes and check if all the blocks are on disk.

m-p-3 commented 3 years ago

Got it, and thanks for letting me know about IPFS-cluster, that looks like a great system that would gracefully solve this problem.

ajhalili2006 commented 3 years ago

@AndreiJirohHaliliDev2006 The main legacy repository is not hosted on bintray so will not be affected. The science and game-packages repos are though. I will set up a mirror for them elsewhere, but users might need to manually adjust the URL to be able to use them

@Grimler91 asked about that in case the devs had any plans to permanently shut down the Android 5.x-6.x packages repo (thus break pkg installs for Play Store users, and also backporting to these unsupported versions is also a challenge), but since this stuff is on IPFS, I don't need to worry about it. I would like to maintain these packages to keep them up-to-date, as much as I can.

Sidenote: Since I'm new to deb packaging in general, this might give me some time to practice the packaging process.

Grimler91 commented 3 years ago

@Grimler91 asked about that in case the devs had any plans to permanently shut down the Android 5.x-6.x packages repo (thus break pkg installs for Play Store users, and also backporting to these unsupported versions is also a challenge), but since this stuff is on IPFS, I don't need to worry about it.

The main legacy repository is not on IPFS (as far as I know), but there are archives with the packages saved here. The main legacy repo is hosted on termux.net and controlled by @fornwall. I don't know if he has plans on continuing hosting it indefinitely or if he might sooner or later need the disk space for other things.

iBug commented 3 years ago

Thank you all for this amazing project for Android! Please allow me to share my $0.02 as a long-term Termux user.

As far as I know, Backblaze B2 provides affordable object storage for $5 per TB-month, and free outbound traffic if you place Cloudflare CDN in front of your buckets. This may be a good option for a small project, but with over 30 TB of monthly traffic, Cloudflare may complain about violating its Terms of Service, Section 2.8, which specifically calls out that excessive non-web content is prohibited, unless purchased separately as part of a Paid Service.

I would recommend giving it a try to contact Cloudflare to see if they can offer a good plan to deliver Termux APT/dpkg packages through their network, thus eliminating the traffic cost from Backblaze B2. In theory, if this pulls off, I would expect a monthly cost of less than US$50, which I sincerely hope is an affordable cost for Termux maintainers.

Disclaimer: I an NOT affiliated with either Backblaze, Inc. or Cloudflare, Inc.

Besides that, I'm also interested in any possible way public mirrors can assist Termux in delivering content. In my residual country (China), I am aware of multiple (6+) university-held public software mirror sites, including but not limited to those in Tsinghua University and University of Science of Technology of China*. Those mirrors are already serving TBs of packages for Termux, and their maintainers are eager to provide support should an open-source community need assistance.

If it's possible for mirror sites to provide additional help, please create a public thread and I can loop them in.

* I am one of the maintainers of the said mirror site.

fphammerle commented 3 years ago

@iBug, you could setup ipfs, pin the hashes posted in https://github.com/termux/termux-packages/issues/6348#issuecomment-776169136 and optionally serve a public ipfs gateway in your area. that's basically equivalent to hosting a repo mirror

vladimyr commented 3 years ago

Homebrew is also using Bintray and also in process of finding a new home for their artifacts. According to https://github.com/Homebrew/discussions/discussions/691#discussioncomment-469668 they are going with Internet Archive which seems like a very interesting possibility. Might be a good fit for Termux too?