Infrastructure update for Channel Cloning/CDN sync

morremeyer commented 2 weeks ago

Your question:

Hey everyone,

we (Anaconda) have been working on upgrades for the channel cloning/CDN sync infrastructure of anaconda.org, which is is used for conda-forge.

Time planning

We are planning to switch to the new infrastructure tomorrow, 2024-08-27 at 12:30 UTC. No downtime is expected for users.

Due to the amount of changes on the conda-forge channel, the initial sync after the switch to the new infrastructure will be a bit slower than the usual syncs - it is currently running in a "dry run" mode on the new infrastructure already to keep its caches up to date, and I expect the initial sync on the new infrastructure to take ~30 minutes.

We have already implemented this change for the r and anaconda channels to ensure that our maintenance process and the new infrastructure are working as desired.

What's changing?

The codebase has been updated to make the cloning more fault-tolerant and reliable
The new infrastructure is more tightly integrated into our observability stack, enabling us to detect problems earlier and more granularly (e.g. a specific package that fails to clone instead of just a general cloning error)
We're adding a last-updated.json file at the channel root, which contains the last update time both as UNIX and ISO8601 timestamps. We are using this directly to monitor the updates end-to-end. Check it out for the anaconda channel if you're interested!

Other notes

This topic was discussed in the conda-forge meeting on 2024-08-21 and it was decided to bring it up here to raise awareness.

If there's any questions, please ask!

h-vetinari commented 2 weeks ago

Thank you for all your work on this! 🙏

morremeyer commented 2 weeks ago

The initial clone on the new infrastructure started on 12:34 UTC and is currently ongoing.

I'll update here again when it is done.

morremeyer commented 2 weeks ago

The initial clone finished uploading and purging the Cloudflare cache at 13:18, we are now enabling the scheduled runs.

morremeyer commented 2 weeks ago

We are aware that the update frequency is lower than expected and are working to improve it.

morremeyer commented 2 weeks ago

We have improved the update frequency to be under 20 minutes now.

We will continue to look into improvements this week, as we already have ideas for that. For now, we will let the process run as-is to get more insights into where the performance can be further improved.

jezdez commented 2 weeks ago

Neat to be able to check https://conda.anaconda.org/conda-forge/last-updated.json now :)

jaimergp commented 2 weeks ago

Cool, I'll update the status page!

jakirkham commented 2 weeks ago

Am happy this went so smoothly! 😄

Thanks to everyone at Anaconda who deployed these updates! 🙏

Looking forward to hearing about additional planned improvements 😉

morremeyer commented 2 weeks ago

Yesterday, we improved the performance of the anaconda.org application, allowing us to put more load on it now with the channel cloning.

With this, we were able to parallelize more parts of the cloning process, so it now takes 13-14 minutes for each run.

There is some cleanup left for this migration (cleaning out unneeded cache files from S3), which we'll do early next week after reviewing the affected files carefully.

morremeyer commented 1 week ago

We have identified some package archives that would be cleaned up since they don't appear in any subdir's repodata.

To prevent the deletion of any archives that are still required, I will reach out to the maintainers of the affected packages in the upcoming days to clarify if the archives can be deleted or are still required.

I will link these discussions to this issue so they're easy to find.

beckermr commented 1 week ago

Can you cc core as well? It'd be good for us to review any decisions. We do not delete packages.

morremeyer commented 1 week ago

@beckermr Is it @conda-forge/core? I don't have access to the teams, looks like they are private? I'm not sure if I would even be able to @-mention them in this case.

beckermr commented 1 week ago

You'd need to join conda forge to ping us. Can you post a list of the packages here?

morremeyer commented 5 days ago

@beckermr Sure!

General

None of these archives appear in the repodata, so the cloning process considers them unneeded and would delete them once we activate the feature that deletes unneeded files.

Note that while these files would be removed from the clone, they would still be available to be downloaded from the anaconda.org frontend manually.

Files to be deleted

lightgbm

Notes: Version bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc is marked as broken.

win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h00ffb61_0.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h00ffb61_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h1afd202_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h91aab33_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h980d568_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h9de49d8_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310h9de49d8_1.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310hbb3657e_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310hbb3657e_1.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310hc6cd4ac_0.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310hc6cd4ac_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py310hc9d5dac_0.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h12c1d0e_0.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h12c1d0e_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h577f764_0.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h5939902_1.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h83cebed_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h83cebed_1.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h8715677_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h8715677_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h90d6d50_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311h93c27bc_0.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311hb755f60_0.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py311hb755f60_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312h10a32a6_1.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312h2aa54b4_1.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312h30efb56_1.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312h53d5487_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312h98d09ba_1.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py312hd38e099_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h0ed8989_1.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h17151c0_0.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h17151c0_1.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h2071588_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h2071588_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h332cf28_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38h8a13381_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38hb83fbf6_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38hb83fbf6_1.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38hd3f51b4_0.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38hd3f51b4_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py38hf5fd131_0.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h07d83e0_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h387a81e_0.conda
linux-aarch64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h387a81e_1.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h3d6467e_0.conda
linux-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h3d6467e_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h615f9d6_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h6bf436c_0.conda
linux-ppc64le/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h6bf436c_1.conda
osx-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h86ebb63_1.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h99910a6_0.conda
win-64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39h99910a6_1.conda
osx-arm64/lightgbm-bee59dd269a93b093f2c610d4a6683a7ea87c63d3ea35c622123ce2c020b2abc-py39ha04627c_1.conda

halide

Notes:

There are 16 archives for halide, 4 .conda and 12 .tar.bz2. The clone would delete all 12 .tar.bz2 archives.
Notable differences: Only the .conda archives are labeled with main.

osx-64/halide-2016.10.25-0.tar.bz2
linux-64/halide-2016.10.25-0.tar.bz2
osx-64/halide-2016.10.25-1.tar.bz2
linux-64/halide-2016.10.25-1.tar.bz2
osx-64/halide-2018.02.15-0.tar.bz2
linux-64/halide-2018.02.15-0.tar.bz2
linux-64/halide-2018.02.15-h0d7d2f0_1000.tar.bz2
osx-64/halide-2018.02.15-hbd2acd6_0.tar.bz2
linux-64/halide-2018.02.15-hbd2acd6_0.tar.bz2
osx-64/halide-2018.02.15-hca0541f_1000.tar.bz2
linux-64/halide-2019.08.27-ha4b6ee1_0.tar.bz2
osx-64/halide-2019.08.27-hee78dc8_0.tar.bz2

halide-python

Notes:

There are 49 archives for halide-python, 10 .conda and 39 .tar.bz2. The clone would delete all 39 .tar.bz2 archives.
Notable differences: Only the .conda archives are labeled with main.

osx-64/halide-python-2018.02.15-py27_0.tar.bz2
linux-64/halide-python-2018.02.15-py27_0.tar.bz2
osx-64/halide-python-2018.02.15-py27h5d47a0c_1.tar.bz2
osx-64/halide-python-2018.02.15-py27he4537c4_1001.tar.bz2
linux-64/halide-python-2018.02.15-py27he860b03_1001.tar.bz2
linux-64/halide-python-2018.02.15-py27he941832_1.tar.bz2
osx-64/halide-python-2018.02.15-py35_0.tar.bz2
linux-64/halide-python-2018.02.15-py35_0.tar.bz2
osx-64/halide-python-2018.02.15-py36_0.tar.bz2
linux-64/halide-python-2018.02.15-py36_0.tar.bz2
osx-64/halide-python-2018.02.15-py36h5d47a0c_1.tar.bz2
osx-64/halide-python-2018.02.15-py36he4537c4_1001.tar.bz2
linux-64/halide-python-2018.02.15-py36he860b03_1001.tar.bz2
linux-64/halide-python-2018.02.15-py36he941832_1.tar.bz2
osx-64/halide-python-2018.02.15-py37h5d47a0c_1.tar.bz2
osx-64/halide-python-2018.02.15-py37he4537c4_1001.tar.bz2
linux-64/halide-python-2018.02.15-py37he860b03_1001.tar.bz2
linux-64/halide-python-2018.02.15-py37he941832_1.tar.bz2
linux-64/halide-python-2019.08.27-py27he513fc3_0.tar.bz2
osx-64/halide-python-2019.08.27-py27hf0a70de_0.tar.bz2
osx-64/halide-python-2019.08.27-py36h0bd707b_0.tar.bz2
osx-64/halide-python-2019.08.27-py36h0bd707b_1.tar.bz2
linux-64/halide-python-2019.08.27-py36h7f0b59b_0.tar.bz2
linux-64/halide-python-2019.08.27-py36h7f0b59b_1.tar.bz2
linux-64/halide-python-2019.08.27-py36he513fc3_0.tar.bz2
osx-64/halide-python-2019.08.27-py36hf0a70de_0.tar.bz2
linux-64/halide-python-2019.08.27-py37h4ef193e_0.tar.bz2
linux-64/halide-python-2019.08.27-py37h4ef193e_1.tar.bz2
osx-64/halide-python-2019.08.27-py37hdbeccfc_0.tar.bz2
osx-64/halide-python-2019.08.27-py37hdbeccfc_1.tar.bz2
linux-64/halide-python-2019.08.27-py37he513fc3_0.tar.bz2
osx-64/halide-python-2019.08.27-py37hf0a70de_0.tar.bz2
linux-64/halide-python-2019.08.27-py38h31a667f_0.tar.bz2
linux-64/halide-python-2019.08.27-py38h31a667f_1.tar.bz2
osx-64/halide-python-2019.08.27-py38hd0b034b_0.tar.bz2
osx-64/halide-python-2019.08.27-py38hd0b034b_1.tar.bz2
linux-64/halide-python-2019.08.27-py38he513fc3_0.tar.bz2
osx-64/halide-python-2019.08.27-py38hf0a70de_0.tar.bz2
linux-64/halide-python-2019.08.27-py39hc2c29c0_1.tar.bz2

matplotlib

Notes: 29 .conda archives for version 3.9.0 would be deleted

osx-64/matplotlib-3.9.0-py310h2ec42d9_0.conda
win-64/matplotlib-3.9.0-py310h5588dad_0.conda
osx-arm64/matplotlib-3.9.0-py310hb6292c7_0.conda
linux-aarch64/matplotlib-3.9.0-py310hbbe02a8_0.conda
linux-ppc64le/matplotlib-3.9.0-py310hd032262_0.conda
linux-64/matplotlib-3.9.0-py310hff52083_0.conda
win-64/matplotlib-3.9.0-py311h1ea47a8_0.conda
linux-64/matplotlib-3.9.0-py311h38be061_0.conda
osx-64/matplotlib-3.9.0-py311h6eed73b_0.conda
linux-ppc64le/matplotlib-3.9.0-py311h957a2fd_0.conda
osx-arm64/matplotlib-3.9.0-py311ha1ab1f8_0.conda
linux-aarch64/matplotlib-3.9.0-py311hfecb2dc_0.conda
osx-arm64/matplotlib-3.9.0-py312h1f38498_0.conda
linux-ppc64le/matplotlib-3.9.0-py312h2344ed7_0.conda
win-64/matplotlib-3.9.0-py312h2e8e312_0.conda
linux-64/matplotlib-3.9.0-py312h7900ff3_0.conda
linux-aarch64/matplotlib-3.9.0-py312h8025657_0.conda
osx-64/matplotlib-3.9.0-py312hb401068_0.conda
win-64/matplotlib-3.9.0-py39h0d475fb_0.conda
linux-64/matplotlib-3.9.0-py39h4162558_0.conda
osx-64/matplotlib-3.9.0-py39h43b90dd_0.conda
osx-64/matplotlib-3.9.0-py39h6e9494a_0.conda
linux-aarch64/matplotlib-3.9.0-py39ha65689a_0.conda
linux-ppc64le/matplotlib-3.9.0-py39hb881913_0.conda
linux-ppc64le/matplotlib-3.9.0-py39hc1b9086_0.conda
win-64/matplotlib-3.9.0-py39hcbf5309_0.conda
linux-aarch64/matplotlib-3.9.0-py39hdc1b86f_0.conda
osx-arm64/matplotlib-3.9.0-py39hdf13c20_0.conda
linux-64/matplotlib-3.9.0-py39hf3d152e_0.conda

matplotlib-base

Notes: 29 .conda archives for version 3.9.0 would be deleted

linux-64/matplotlib-base-3.9.0-py310h0b1de36_0.conda
win-64/matplotlib-base-3.9.0-py310h37e0a56_0.conda
osx-64/matplotlib-base-3.9.0-py310h56dc6a7_0.conda
linux-ppc64le/matplotlib-base-3.9.0-py310hda06ede_0.conda
osx-arm64/matplotlib-base-3.9.0-py310heb73f16_0.conda
linux-aarch64/matplotlib-base-3.9.0-py310hf9f654d_0.conda
win-64/matplotlib-base-3.9.0-py311h8f1b1e4_0.conda
linux-aarch64/matplotlib-base-3.9.0-py311ha1793d2_0.conda
linux-ppc64le/matplotlib-base-3.9.0-py311hb9bd21d_0.conda
osx-arm64/matplotlib-base-3.9.0-py311hba6b155_0.conda
osx-64/matplotlib-base-3.9.0-py311hf31e254_0.conda
linux-64/matplotlib-base-3.9.0-py311hffb96ce_0.conda
osx-64/matplotlib-base-3.9.0-py312h0d5aeb7_0.conda
osx-arm64/matplotlib-base-3.9.0-py312h32d6e5a_0.conda
linux-ppc64le/matplotlib-base-3.9.0-py312h5ab8002_0.conda
win-64/matplotlib-base-3.9.0-py312h90004f6_0.conda
linux-64/matplotlib-base-3.9.0-py312h9201f00_0.conda
linux-aarch64/matplotlib-base-3.9.0-py312h97afc53_0.conda
osx-arm64/matplotlib-base-3.9.0-py39h1398496_0.conda
linux-ppc64le/matplotlib-base-3.9.0-py39h247f39b_0.conda
linux-64/matplotlib-base-3.9.0-py39h34e7624_0.conda
win-64/matplotlib-base-3.9.0-py39h5278f01_0.conda
win-64/matplotlib-base-3.9.0-py39h5376392_0.conda
linux-ppc64le/matplotlib-base-3.9.0-py39h6bea8b6_0.conda
linux-aarch64/matplotlib-base-3.9.0-py39hb0dea69_0.conda
osx-64/matplotlib-base-3.9.0-py39hb2d44f9_0.conda
osx-64/matplotlib-base-3.9.0-py39hc6bb3ea_0.conda
linux-64/matplotlib-base-3.9.0-py39hd75cb13_0.conda
linux-aarch64/matplotlib-base-3.9.0-py39hf3ba65a_0.conda

Other packages, labeled broken/corrupted

Notes: The following archives exist, but are labeled as broken or corrupted

noarch/cobra-0.25.0-pyhd8ed1ab_0.tar.bz2
osx-64/cross-python_osx-arm64-3.10-32_cpython.tar.bz2
osx-arm64/libcxx-19.1.0.rc1-h167917d_0.conda
win-64/vs2015_version-14.16.27033-hd19b579_12.conda
win-64/vs2015_version-14.28.29325-hca5e498_12.conda
win-64/vs2015_version-14.29.30139-h2746e5f_12.conda
win-64/vs2015_version-14.32.31332-h6c625ad_12.conda
win-64/vs2015_version-14.34.31931-hed1258a_12.conda
noarch/rattler-build-conda-compat-0.1.2-pyhd8ed1ab_0.conda

Everything else

Notes: These exist and would be deleted, nothing else about them stands out.

noarch/planet-2.0a6-pyhd8ed1ab_0.conda
osx-64/python-3.11.0-h4150a38_1_debug_cpython.conda
osx-64/python-3.11.0-he7542f4_1_debug_cpython.conda

beckermr commented 5 days ago

None of these archives appear in the repodata, so the cloning process considers them unneeded and would delete them once we activate the feature that deletes unneeded files.

What does this mean precisely? Do you mean the repodata_from_packages.json?

beckermr commented 5 days ago

Other notes. We never delete packages in conda-forge, so even packages with broken labels need to be retained.

conda-forge / conda-forge.github.io