Run a scalable mirror of conda-forge on GitHub packages.

Quansight-Labs / czi-conda-forge-mgmt

🚀 Top level project management for conda-forge CZI grant

https://github.com/orgs/Quansight-Labs/projects/10

BSD 3-Clause "New" or "Revised" License

5 stars 0 forks source link

Run a scalable mirror of conda-forge on GitHub packages. #23

Open jaimergp opened 1 year ago

jaimergp commented 1 year ago

📌 Summary

Create and maintain a mirror of conda-forge and bioconda on Github packages

📝 Background

Using the OCI capabilities of Github Packages, we can store conda artifacts. This can be used as a mirror for archival purposes, at the very least.

## 🚀 Tasks / Deliverables
- [ ] https://github.com/conda/ceps/pull/70
- [ ] Staging mechanism
- [ ] Repodata indexing mechanism
- [ ] https://github.com/channel-mirrors/mirrormirror/issues/11
- [ ] https://github.com/channel-mirrors/mirrormirror/issues/12
- [ ] https://github.com/channel-mirrors/mirrormirror/issues/13

📅 Estimated completion

This task should be finished in the first 12 months.

ℹ️ References

jaimergp commented 1 year ago

About direct uploads to ghcr.io, see this document.

jaimergp commented 1 year ago

xref https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/208

jaimergp commented 1 year ago

Posted one more comment with my thoughts in the issue above. See https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/208#issuecomment-1573586190

jaimergp commented 4 months ago

Current status:

Cronjob GHA workflow keeps running at https://github.com/channel-mirrors/mirrormirror. We recently fixed a problem with idle workflows that should be resolved now.
Small 404 problems with some marked-as-broken artifacts (see https://github.com/conda/infrastructure/issues/797 and https://github.com/conda-forge/admin-requests/pull/937)

What we need next:

CEP to document the OCI layering and naming conventions used at https://github.com/orgs/channel-mirrors/packages
A mechanism to stage and publish recently uploaded packages without Anaconda.org (see comments above for discussions)

jaimergp commented 4 months ago

Important CEP: https://github.com/conda-incubator/ceps/blob/main/cep-15.md

This allows the repodata and packages to live in separate locations.

jaimergp commented 4 months ago

There are some extra errors in osx-arm64 and bioconda caused by connection errors of different types. @goanpeca can you look into this?