lasote
commented
5 years ago Hi, I understand the use case but I'm afraid that would require huge changes and complexity, that eventually will increase the unexpected errors and the complexity to debug them.
Also, this idea could be valid when using the revisions mechanism correctly, otherwise a package could be overwritten and upload so you would need to invalidate the cache, deal with the concurrency, etc.
Probably it would be, better than having a "double" local cache, a better idea to introduce a dummy downloads cache, but only if the server allows to check the checksum of an artifact, (Artifactory can do that) before downloading it. That way you could manage a little bit easier a "shared" folder with the downloads cache. But of course, the package should be uncompressed also, so it is expensive too. I think that could be doable and not excessively complex.
We are currently trying to define how the CI/CD flows should work and any idea is welcome, of course, but we have a priority of optimizing the parallelism when building the dependency tree, in addition, to know which packages have to be built when some of them change. We would like to improve that but it is not a priority right now.
Thanks!
memsharded
Going beyond the basic package consumer use case, there are two basic ways in which a package may make its way into the conan cache:
These two variants have basic differences:
The core suggestion would be to create a two-tier package cache. Anything that is downloaded from a remote should be placed in a separate location than the locally built packages. If such functionality existed, I think it's naturally to expect that users could share the same downloaded package cache location.
A motivating factor for this request is how CI/CD systems may operate with conan. We'd like to set each build job to be independent from all other jobs. As such, we'd like for each job to have its own build package cache. If each job is independent though, we incur the cost of downloads for each job. If we try to solve the problem with a common conan cache, we've run into issues with collisions. Basically someone will try to build a package with version X, then someone will try to build a different commit with the same version X (or someone will try to consume version X in a different build), and that leads to unexpected behaviour. We can fix this problem by enforcing a better versioning system, but as a more general alternative, we'd like to eliminate duplicate downloads, so we'd like for the build jobs to share a download package cache.