[bug] Virtual repository breaks when remote repository becomes unreachable (and conan center is down)

[rconde01]

Environment Details (include every applicable attribute)

• Operating System+version: all
• Compiler+version: n/a
• Conan version: 1.55
• Python version: 3.11

Steps to reproduce (Include if Applicable)

Conan center appears to be down at the moment. I have a virtual repository on Artifactory that uses conan center as a remote repository. I thought some of the point here was to have a cache that was under our control, so if conan center goes down or goes away, we can continue to operate. Shouldn't this work?

[blackliner]

wow. This. Is. Bad... @memsharded

Removing the remote from the virtual repo in Artifactory did not help either, although the package is definitely available in the local repo. This is a death sentence for conan, seriously!

[memsharded]

Yes, seems ConanCenter is down, team is investigating.

[memsharded]

https://status.cloud.google.com/ it seems GoogleCloud (used by ConanCenter) is having issues, a bit outside of our reach.

[rconde01]

That's fair - but to the larger issue, shouldn't this have worked when setup via Artifactory as a remote repository?

[memsharded]

That's fair - but to the larger issue, shouldn't this have worked when setup via Artifactory as a remote repository?

Yes, we need to follow up with the Artifactory team to better understand this issue. Thanks very much for reporting it.

[memsharded]

The incident with ConanCenter seems to be resolved. Please try again and let us know.

[kymikoloco]

https://www.jfrog.com/confluence/display/JFROG/Remote+Repositories#RemoteRepositories-SingleRepositoryOffline

Manually setting the remote to Offline may be a requirement to bypass the query and allow the caching to work in these situations, albeit temporarily. Not ideal, I would also have assumed it would use the cache as a fallback, especially if there are local packages in the virtual repo unrelated to the remote, but I'm not intimately familiar with the backend of Artifactory in these situations.

[rconde01]

any feedback here from the artifactory team?

[blackliner]

They reached out to me through their commercial support channel, but other than a few misplaced explanations, no solution.

[memsharded]

I am discussing the topic with the Artifactory teams. At the moment it seems that it is in fact necessary to go to Artifactory and activate "offline mode", but there is no such automatic activation of "offline mode" when the proxied repo (ConanCenter in this case) is down.

I will keep pushing this internally, but at the moment the alternatives could be:

• Live with the potential occasional disruptions of ConanCenter, they are unlikely but they might happen from time to time, for example sometimes we are being affected by other cloud providers outages or issues. Have some admin that can quickly activate "offline mode" when necessary.
• Decouple the ConanCenter remote via the Conan client: define 2 repos in the client and do conan download -r=conancenter + conan upload -r=myrepo of the desired artifacts when desired, populate your own myrepo remote that is the one used in production. This approach needs onboarding new packages that might be necessary explicitly, and also explicitly doing updates.
• Decouple the ConanCenter remote by building your own binaries from conan-center-index repo and uploading to your own remote. This approach is the recommended one for most enterprises, as it allows full control of what goes into the company binaries, full ownership of binaries, full isolation of automatic updates in the upstream with much better robustness and security. This can be achieved quite easily with some simple scripts like the ones I have in my fork: https://github.com/memsharded/conan-center-index/tree/mycustom (mybuild.py).

[memsharded]

Team is trying to reproduce this issue, but no success. It seems that latest Artifactory version is doing the expected and is able to serve cached packages even hen the proxied repository (ConanCenter is down). Can you please report which Artifactory versions are you running? Thanks.

[blackliner]

How did you reproduce the issue?

[memsharded]

Fetching packages from a remote repo then disconnecting the network to the proxied repo, re-installing from that remote repo.

[rconde01]

I’m not in a position test at the moment…but I wonder what happens if it’s reachable but unresponsive. In my case it shouldn’t have needed anything from the actual conan remote.

[memsharded]

I’m not in a position test at the moment…but I wonder what happens if it’s reachable but unresponsive. In my case it shouldn’t have needed anything from the actual conan remote.

Yes, this is what I am suspecting, that the behavior might not be exactly the same if it is a "clean" disconnection or if there are other kind of issues. Specially if it not responsive, that might be definitely a very complicated issue to resolve.