search

regro / cf-scripts

Flagship repo for cf-regro-autotick-bot
Other
46 stars 70 forks source link

Add github releases for feedstocks #2828

Open ZzEeKkAa opened 2 days ago

ZzEeKkAa commented 2 days ago

Not sure where to ask this question, but I've found https://conda-forge.org/blog/2020/11/20/anaconda-tos/ post that references to regro/releases which is currently unavailable. I'm wondering what happen to this repo? Are there any plans to distribute builds as releases besides uploading to anaconda servers and what are the blockers? According to the https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases#storage-and-bandwidth-quotas there is no quotas on usage, except the size limit of 2GB per file, which should cover most of the conda-forge packages. It would be possible to update quartz to host repodata.json and redirect all the package traffic to the corresponding files in the release section of the github repo.

beckermr commented 2 days ago

This activity broke GitHub's backend code and so we had to stop.

ZzEeKkAa commented 2 days ago

Thank you! Are there any references or status update if it was fixed?

beckermr commented 2 days ago

It wasn't so much of a bug but actually they didn't want us uploading that much data to a repo.

ZzEeKkAa commented 2 days ago

How then homebrew works and it does not the case there? https://github.com/orgs/Homebrew/packages?ecosystem=container

image

beckermr commented 2 days ago

Those are container images and that is a separate issue. What are you looking to do here?

ZzEeKkAa commented 2 days ago

I'm looking for an option to use conda-forge without anaconda's servers

beckermr commented 2 days ago

Yeah that's something that is really hard to do. The amount of bandwidth and money they spend serving conda-forge is immense. We have hundreds of millions of downloads each month. They also pay for CDN services for conda-forge and some other large channels which is a significant performance enhancement.

We have been working with the conda community to upload artifacts to the github container site, but cannot yet point conda itself at the repo afaik.

Also, certain parts of the ecosystem have to be distributed on anaconda.org due to licensing restrictions. So any mirror is going to have that limitation.

ZzEeKkAa commented 1 day ago

Thank you!

The amount of bandwidth and money they spend serving conda-forge is immense. We have hundreds of millions of downloads each month. They also pay for CDN services for conda-forge and some other large channels which is a significant performance enhancement.

Having homebrew as an example it looks like github does not have any restrictions and may provide these capacities.. I've looked into how it works, and they are using OCI. It should be possible to pack conda packages with OCI (aka new conda package format) and teach conda cli to work with it. Then github registry will provide both hosting and open analytics on the usage... The technical challenge may be to provide different way of dependency resolution, cause there won't be repodata.json index.

We have been working with the conda community to upload artifacts to the github container site, but cannot yet point conda itself at the repo afaik.

So far any references to accomplished work?

Also, certain parts of the ecosystem have to be distributed on anaconda.org due to licensing restrictions. So any mirror is going to have that limitation.

What part of ecosystem? Is there any list to be aware of? And what kind of limitations are those?

beckermr commented 1 day ago

I'd ask @jaimergp snd @wolfv. They can let you know what the status is precisely.

beckermr commented 1 day ago

The limits in terms of licensing are nvidia / cuda and possibly intel packages.

wolfv commented 1 day ago

@ZzEeKkAa yes, you can already use a number of mirrors, with conda, or mamba. You can also use the OCI mirror on Github with pixi.

One mirror is for example on prefix, just use -c https://prefix.dev/conda-forge

If you want to use the packages on ghcr.io, you can configure pixi in a way to do that. Find the docs here: https://pixi.sh/latest/reference/pixi_configuration/#oci-mirrors

The packages are found under: https://github.com/orgs/channel-mirrors/packages

wolfv commented 1 day ago

The next version of mamba should also get support for accessing the OCI registry, by the way. If you want to help us make this nice, let me know! Easiest would be to continue chatting on our Discord server (you can find it on https://prefix.dev)

beckermr commented 1 day ago

Have we written a plugin to conda for this yet or is there more involved there?

wolfv commented 1 day ago

Jannis has some plans, but I don't know if that ever progressed anywhere :) But you can use all of this very nicely from rattler / py-rattler as well.

ZzEeKkAa commented 1 day ago

Wow, it looks like a lot of work was done! Let me check out pixi. Never heard about it. Thank you!

wolfv commented 1 day ago

yes, we are working hard to make sure that there is no vendor-lock-in for the conda-forge community :)

napetrov commented 1 day ago

yes, this is indeed interesting direction, even assuming associated costs.

On licensing for distribution - i also think this is feasible to discuss options there for vendor specific packages