conda-forge CDN cloning last updated ~869mins ago

conda / infrastructure

A repo to report issues and have discussions about the conda infrastructure

BSD 3-Clause "New" or "Revised" License

11 stars 15 forks source link

conda-forge CDN cloning last updated ~869mins ago #935

Closed jakirkham closed 2 months ago

jakirkham commented 2 months ago

Looks like we are running into an issue with CDN cloning for the conda-forge channel

According to the conda-forge.org/status page, it has been ~869mins since that last update (screenshot below):

xref: https://github.com/conda-forge/status/issues/176

jakirkham commented 2 months ago

A recent example of this is libtorch. The oldest package here was uploaded ~4hrs ago and still has a download count of 0

Edit: This can also be confirmed with conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_4'. Please details below

``` % conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_4' Loading channels: done No match found for: libtorch==2.1.2[subdir=osx-arm64,build=*_4]. Search: *libtorch*==2.1.2[subdir=osx-arm64,build=*_4] PackagesNotFoundError: The following packages are not available from current channels: - libtorch==2.1.2[subdir=osx-arm64,build=*_4] Current channels: - https://conda.anaconda.org/conda-forge/osx-arm64 - https://conda.anaconda.org/conda-forge/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org and use the search bar at the top of the page. ```

Note: conda search 'conda-forge:libtorch[subdir=osx-arm64]=2.1.2=*_3' (the previous build) works

h-vetinari commented 2 months ago

Another example is pyarrow-core / pyarrow-all which saw their first builds a few hours ago.

jakirkham commented 2 months ago

The conda-forge status page is now showing the CDN as having synced recently

Do see some improvements with libtorch, but still a lot of packages are being missed

IOW this is still a live issue, but it does seem like the CDN is trying to do something

h-vetinari commented 2 months ago

IOW this is still a live issue, but it does seem like the CDN is trying to do something

I'm cautiously optimistic. Basically my guess is that the cloning is operational again, and now it takes "just" the usual 1-2 hours (or sometimes 5, if a clone fails), for the packages to show up in the CDN.

h-vetinari commented 2 months ago

All the pyarrow-core packages have now made it through the CDN.

jezdez commented 2 months ago

Yes, this was a temporary problem that @dholth has fixed within the cloning system.

jakirkham commented 2 months ago

Thanks Jannis! 🙏

Do we know what the cause was?

dholth commented 2 months ago

We are working on a permanent fix that does not fill the disk. When the clone process is working, it takes about 8-12 minutes per iteration. There may be delays in other parts of the process like anaconda.org or the CDN. Do we have a way to measure the time that individual packages take to clone?

jakirkham commented 2 months ago

Thanks Daniel! 🙏

Ok gotcha. So this is the disk overfilling issue we have seen before. Good to know it's a familiar issue

I don't think we have measurements collected external to Anaconda. Though I would be happy to be wrong. @wolfv or @beckermr is this something either of you have looked at before? No worries if not

Is this measured on the Anaconda side somehow?