Open johschmitz opened 2 years ago
I'd also like to see something a bit more explicit about this point for 2.0. But the first point is try to define how a succesful proof of concept looks like, define the basic requirments, UI, etc. For example:
As we are already approaching 2.0 GA (already in beta), it doesn't look like that such a new feature will be in 2.0 scope but in 2.X. Looking forward to getting more detailed feedback from users.
I will link this old tickets #1597, #7693 as a starting point. Will need to look more into this and experiment to see how to combine build caching with Conan. I was actually hoping to find an opinionated solution within the Conan ecosystem which works well enough. As of now it still seems to be secret sauce knowledge though..
Hi @memsharded!
I'm currently working with remote ccache and conan2 and have encountered an issue related to the calculation of cache hash using include paths. Some conan packages seem to generate different include paths on separate machines, even when everything else remains the same.
For example, the include path for OpenSSL 3.1.0 might vary like this: ~/.conan2/p/opens738d1e90ec137
vs ~/.conan2/p/opensb892b04917b0
. Moreover, every time I reinstall the package, the path changes. However, other libraries, such as zlib, consistently generate the same include paths.
I am unsure how conan calculates these paths. If the package version and toolchain are the same, having consistent include paths would greatly simplify the integration with remote ccache. Could you please provide some insight into this issue or suggest possible ways to ensure consistent include paths across different machines and installations?
Thank you for your assistance!
Hi @pwqbot
Thanks for your report. I have been having a look, this is the related code:
@staticmethod
def _short_hash_path(h):
""":param h: Unicode text to reduce"""
h = h.encode("utf-8")
md = hashlib.sha256()
md.update(h)
sha_bytes = md.hexdigest()
# len based on: https://github.com/conan-io/conan/pull/9595#issuecomment-918976451
# Reduce length in 3 characters 16 - 3 = 13
return sha_bytes[0:13]
@staticmethod
def _get_path(ref: RecipeReference):
return ref.name[:5] + DataCache._short_hash_path(ref.repr_notime())
As you can see, this should be a perfect mathematical, constant hash of the recipe/package reference. That means that the folder should always be exactly the same one for the same recipe (same revision) and the same package (same package revision)
Could you please double check that you are installing exactly the same recipe and package revision in both machines? A log of the conan install
in the 2 different machines could help, it should contain everything there, including the revisions and the different folder names.
Install in the same machine twice will get different result. conanfile.py
from conan import ConanFile
from conan.errors import ConanInvalidConfiguration
class test_ccache(ConanFile):
name = "contek_adapter"
version = "0.0.0"
settings = "os", "compiler", "build_type", "arch"
generators = "CMakeToolchain", "CMakeDeps"
def requirements(self):
self.requires("openssl/3.1.0")
first time.
openssl/3.1.0: package(): Packaged 3 files: c_rehash, openssl, LICENSE
openssl/3.1.0: package(): Packaged 1 '.txt' file: LICENSE.txt
openssl/3.1.0: package(): Packaged 135 '.h' files
openssl/3.1.0: package(): Packaged 2 '.a' files: libcrypto.a, libssl.a
openssl/3.1.0: package(): Packaged 1 '.cmake' file: conan-official-openssl-variables.cmake
openssl/3.1.0: package(): Packaged 2 '.dylib' files: legacy.dylib, fips.dylib
openssl/3.1.0: Created package revision 7c0c796248b58e869035b45d1aa35e34
openssl/3.1.0: Package '144c355fd5d237d22e14c77819fe31f3933f7e4c' created
openssl/3.1.0: Full package reference: openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#7c0c796248b58e869035b45d1aa35e34
openssl/3.1.0: Package folder /Users/benben/.conan2/p/opens7252b9b44ae00/p
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: 'cpp_info.names' used in: openssl/3.1.0, zlib/1.2.13
WARN: 'cpp_info.build_modules' used in: openssl/3.1.0
WARN: 'env_info' used in: openssl/3.1.0
======== Finalizing install (deploy, generators) ========
conanfile.py (contek_adapter/0.0.0): Writing generators to /Users/benben/tmp/build
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeToolchain' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (contek_adapter/0.0.0): Preset 'conan-release' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-release'
conanfile.py (contek_adapter/0.0.0): If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/Users/benben/tmp/build/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: CMakePresets.json
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeDeps' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): Generating aggregated env files
conanfile.py (contek_adapter/0.0.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully
second time.
openssl/3.1.0: package(): Packaged 3 files: c_rehash, openssl, LICENSE
openssl/3.1.0: package(): Packaged 1 '.txt' file: LICENSE.txt
openssl/3.1.0: package(): Packaged 135 '.h' files
openssl/3.1.0: package(): Packaged 2 '.a' files: libcrypto.a, libssl.a
openssl/3.1.0: package(): Packaged 1 '.cmake' file: conan-official-openssl-variables.cmake
openssl/3.1.0: package(): Packaged 2 '.dylib' files: legacy.dylib, fips.dylib
openssl/3.1.0: Created package revision a480b3336aa4e36f9e4011ab7435d4dc
openssl/3.1.0: Package '144c355fd5d237d22e14c77819fe31f3933f7e4c' created
openssl/3.1.0: Full package reference: openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#a480b3336aa4e36f9e4011ab7435d4dc
openssl/3.1.0: Package folder /Users/benben/.conan2/p/opens1a90f1ceb4970/p
WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: 'cpp_info.names' used in: openssl/3.1.0, zlib/1.2.13
WARN: 'cpp_info.build_modules' used in: openssl/3.1.0
WARN: 'env_info' used in: openssl/3.1.0
======== Finalizing install (deploy, generators) ========
conanfile.py (contek_adapter/0.0.0): Writing generators to /Users/benben/tmp/build
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeToolchain' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (contek_adapter/0.0.0): Preset 'conan-release' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-release'
conanfile.py (contek_adapter/0.0.0): If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/Users/benben/tmp/build/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.py (contek_adapter/0.0.0): CMakeToolchain generated: CMakePresets.json
conanfile.py (contek_adapter/0.0.0): Generator 'CMakeDeps' calling 'generate()'
conanfile.py (contek_adapter/0.0.0): Generating aggregated env files
conanfile.py (contek_adapter/0.0.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully
the reference is
openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#7c0c796248b58e869035b45d1aa35e34
vs
openssl/3.1.0#25925a18588e030e406a15da96c4d32b:144c355fd5d237d22e14c77819fe31f3933f7e4c#a480b3336aa4e36f9e4011ab7435d4dc
Thanks for the feedback.
I see what is happening: you are building openssl
from source twice. The build of openssl
is not deterministic/hermetic/reproducible. Each time it will create a different binary, with a different checksum, and that produces a new revision, so it will be in a different folder (Just in case, achieving deterministic builds is super challenging, and outside of Conan scope, nothing that Conan can do there: https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html)
If the second time, instead of re-building from source, you do conan .... --build=missing
, as the previous binary is there, it will have the same revision and be in the same folder.
Thanks for the feedback. I see what is happening: you are building
openssl
from source twice. The build ofopenssl
is not deterministic/hermetic/reproducible. Each time it will create a different binary, with a different checksum, and that produces a new revision, so it will be in a different folder (Just in case, achieving deterministic builds is super challenging, and outside of Conan scope, nothing that Conan can do there: blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html)If the second time, instead of re-building from source, you do
conan .... --build=missing
, as the previous binary is there, it will have the same revision and be in the same folder.
Thanks! So if I download binary from conan center the folder will be the same? I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center.
Thanks, I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center?
I guess not.
But in any case, it shouldn't be an issue of existing binaries in ConanCenter, but an issue that the second build from source shouldn't be done at all, instead using the --build=missing
will avoid it.
Also a note that in production, relying directly on ConanCenter binaries is not recommended in general (not just for Conan and C++, but for all package management in general). If you are using Conan in production, it is strongly recommended to use your own server to store your packages and dependencies. This way it doesn't matter if ConanCenter doesn't have the gcc12 binaries, you build them once, store them in the server, and don't need to re-build it from sources again. Also, it will make you way more robust against possible changes in ConanCenter recipes, network outages, etc.
Thanks, I am building with gcc12, it seems like gcc12 binary package is not yet supported in conan center?
I guess not. But in any case, it shouldn't be an issue of existing binaries in ConanCenter, but an issue that the second build from source shouldn't be done at all, instead using the
--build=missing
will avoid it. Also a note that in production, relying directly on ConanCenter binaries is not recommended in general (not just for Conan and C++, but for all package management in general). If you are using Conan in production, it is strongly recommended to use your own server to store your packages and dependencies. This way it doesn't matter if ConanCenter doesn't have the gcc12 binaries, you build them once, store them in the server, and don't need to re-build it from sources again. Also, it will make you way more robust against possible changes in ConanCenter recipes, network outages, etc.
Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?
Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?
Yes, this is what many other Conan users are doing, and it seems to be a super reliable and convenient way for production, with reasonable effort. Actually, there are ongoing and planned features for making this even easier:
conan-extensions
repo is getting commands that could help to build things from a conan-center-index
fork: https://github.com/conan-io/conan-extensions/pulls Very preliminary, but expected to keep improving in the following weeks and months.Thanks! What is the best practice to store conan packages on local server, should I fork the conan-center-index and rebuild the packages I need and upload them to my server?
Yes, this is what many other Conan users are doing, and it seems to be a super reliable and convenient way for production, with reasonable effort. Actually, there are ongoing and planned features for making this even easier:
- The new
conan-extensions
repo is getting commands that could help to build things from aconan-center-index
fork: conan-io/conan-extensions/pulls Very preliminary, but expected to keep improving in the following weeks and months.- A new idea about using a fork as a Conan remote directly: POC: Remotes from local file tree #10433
Hi @memsharded, thank you for your suggestions. Currently, I have set up a CI machine and to upload binary files to Artifactory, I execute conan install --requires={pkg_name}/{version} --build=missing
, followed by conan upload {pkg_name}/{version} -r =my_server}
. This method is convenient as it eliminates the need to clone and update the conan-center-index, but I am unsure if it qualifies as a good practice.
Not terrible, but not great either :)
As usual, it depends on your needs, resources, constraints...
Not terrible, but not great either :)
- You are mixing binaries, created in ConanCenter, and your own binaries. You will have a mixture of binaries created in your own machines and in ConanCenter, and this is not great. It would be better if you also build the binaries for existing configurations in ConanCenter, just make sure that all binaries are yours. It is a little bit more of extra CI time, but totally worth it.
- If you depend on those ConanCenter recipes and do that continuously, you will be getting a continuos flow of recipe updates from ConanCenter too, new binaries, etc. While there might be interesting updates, it can also introduce some inestability, some occasional breaking changes or issues, etc. Building from your own fork guarantees complete immutability and full control of the dependencies without having to use extra things as lockfiles.
As usual, it depends on your needs, resources, constraints...
Thanks! I'll consider using conan create instead. However, I still face an issue with uploading all dependency packages to the server. I've found a solution at https://github.com/conan-io/conan-package-tools#upload-dependencies-237, but it seems to be applicable only for Conan 1.x versions.
What issue exactly? We are working on pkg-lists, maybe this is what you need? Check: https://github.com/conan-io/conan/pull/13928
What issue exactly? We are working on pkg-lists, maybe this is what you need? Check: #13928
The issue I'm encountering is that uploading a package doesn't upload its dependencies along with it. This is inconvenient, as it requires me to upload the dependencies explicitly.
The issue I'm encountering is that uploading a package doesn't upload its dependencies along with it. This is inconvenient, as it requires me to upload the dependencies explicitly.
Perfect, this flow is the one that the "package lists" implements:
The issue I'm encountering is that uploading a package doesn't upload its dependencies along with it. This is inconvenient, as it requires me to upload the dependencies explicitly.
Perfect, this flow is the one that the "package lists" implements:
- conan create ... --format=json > graph.json
- conan list --graph=graph.json --graph-binaries=* --format=json > pkglist.json
- conan upload --list=pkglist.json -r=myremote
That's great!!!
@memsharded Returning to the issue at hand, is it possible for Conan to use ccache/scache in CI to speed up the build process in CI?
@memsharded Returning to the issue at hand, is it possible for Conan to use ccache/scache in CI to speed up the build process in CI?
Yes, it is definitely possible, we have seen quite a few Conan users doing it in production and at scale.
A different story is that if it needs extra configuration, adapters, etc, we don't know because it has never been contributed back (most companies are not allowed), so this issue here is for us to investigate, test, and document possible approaches. We haven't had the time to do it yet, we will try to address this when possible, but so far there have been other higher priorities.
@uilianries linked this issue on Discord, but I'm not sure it is exactly about the same topic (how to enable ccache when you build conan recipes).
Anyway, here is a proposal:
conf
called tools.build:compiler_launchers
, similar to tools.build:compiler_executables
. It would be a dictionary (empty by default). Allowed keys could be c
& cpp
to begin with.CMakeToolchain
, define CMAKE_<LANG>_COMPILER_LAUNCHER
with value of corresponding key (if defined) in tools.build:compiler_launchers
. Since CMake 3.4, injecting CMAKE_<LANG>_COMPILER_LAUNCHER=ccache
externally is the transparent way to enable ccache. In the meantime CMAKE_<LANG>_COMPILER_LAUNCHER
are env vars listened by CMake, so technically ccache can be enabled for CMake based recipes without any code manipulation, just by defining CMAKE_C_COMPILER_LAUNCHER
& CMAKE_CXX_COMPILER_LAUNCHER
in buildenv
of profile.
For other build systems, I don't know.It is not necessary to define a new conf
for every CMake variable that we want to inject.
There is now a clean way to inject any arbitrary CMake toolchain file: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/inject_cmake_variables.html
Until there is a known way to add this to other build systems, this wouldn't be necessary.
@uilianries linked this issue on Discord, but I'm not sure it is exactly about the same topic (how to enable ccache when you build conan recipes).
Anyway, here is a proposal:
* Add a `conf` called `tools.build:compiler_launchers`, similar to `tools.build:compiler_executables`. It would be a dictionary (empty by default). Allowed keys could be `c` & `cpp` to begin with. * In `CMakeToolchain`, define `CMAKE_<LANG>_COMPILER_LAUNCHER` with value of corresponding key (if defined) in `tools.build:compiler_launchers`. Since CMake 3.4, injecting `CMAKE_<LANG>_COMPILER_LAUNCHER=ccache` externally is the transparent way to enable ccache. In the meantime `CMAKE_<LANG>_COMPILER_LAUNCHER` are env vars listened by CMake, so technically ccache can be enabled for CMake based recipes without any code manipulation, just by defining `CMAKE_C_COMPILER_LAUNCHER` & `CMAKE_CXX_COMPILER_LAUNCHER` in `buildenv` of profile. For other build systems, I don't know.
Hi there ! You're probably talking about uilianries' answer to my question and I would say that, yes, you're right, I am looking for the "proper way" to use ccache, firstly, for the packages, created for the projects of my team, but secondly, if it's available for packages downloaded from conan-center-index (or any other public or private index server), why deprive ourselves of it ?
As uilianries recommended, here is our situation : Before integration of Conan 2 :
# Based on https://crascit.com/2016/04/09/using-ccache-with-cmake/
cmake_minimum_required(VERSION 3.4)
option(MY_PROJECT_USE_CCACHE "Use the compilation cache system" ON)
if(MY_PROJECT_USE_CCACHE)
find_program(CCACHE_PROGRAM ccache)
if(CCACHE_PROGRAM)
# Set up wrapper scripts
set(C_LAUNCHER "${CCACHE_PROGRAM}")
set(CXX_LAUNCHER "${CCACHE_PROGRAM}")
set(CUDA_LAUNCHER "${CCACHE_PROGRAM}")
file(WRITE "${CMAKE_BINARY_DIR}/launch-c" ""
"#!/usr/bin/env sh\n"
"\n"
"# Xcode generator doesn't include the compiler as the\n"
"# first argument, Ninja and Makefiles do. Handle both cases.\n"
"if [ \"$1\" = \"${CMAKE_C_COMPILER}\" ] ; then\n"
" shift\n"
"fi\n"
"\n"
"export CCACHE_CPP2=true\n"
"exec \"${C_LAUNCHER}\" \"${CMAKE_C_COMPILER}\" \"$@\"\n"
)
file(WRITE "${CMAKE_BINARY_DIR}/launch-cxx" ""
"#!/usr/bin/env sh\n"
"\n"
"# Xcode generator doesn't include the compiler as the\n"
"# first argument, Ninja and Makefiles do. Handle both cases.\n"
"if [ \"$1\" = \"${CMAKE_CXX_COMPILER}\" ] ; then\n"
" shift\n"
"fi\n"
"\n"
"export CCACHE_CPP2=true\n"
"exec \"${CXX_LAUNCHER}\" \"${CMAKE_CXX_COMPILER}\" \"$@\"\n"
)
# Cuda support only added in CMake 3.10
file(WRITE "${CMAKE_BINARY_DIR}/launch-cuda" ""
"#!/usr/bin/env sh\n"
"\n"
"# Xcode generator doesn't include the compiler as the\n"
"# first argument, Ninja and Makefiles do. Handle both cases.\n"
"if [ \"$1\" = \"${CMAKE_CUDA_COMPILER}\" ] ; then\n"
" shift\n"
"fi\n"
"\n"
"export CCACHE_CPP2=true\n"
"exec \"${CUDA_LAUNCHER}\" \"${CMAKE_CUDA_COMPILER}\" \"$@\"\n"
)
execute_process(COMMAND chmod a+rx
"${CMAKE_BINARY_DIR}/launch-c"
"${CMAKE_BINARY_DIR}/launch-cxx"
"${CMAKE_BINARY_DIR}/launch-cuda"
)
if(CMAKE_GENERATOR STREQUAL "Xcode")
# Set Xcode project attributes to route compilation and linking
# through our scripts
set(CMAKE_XCODE_ATTRIBUTE_CC "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_CXX "${CMAKE_BINARY_DIR}/launch-cxx")
set(CMAKE_XCODE_ATTRIBUTE_LD "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_XCODE_ATTRIBUTE_LDPLUSPLUS "${CMAKE_BINARY_DIR}/launch-cxx")
else()
# Support Unix Makefiles and Ninja
set(CMAKE_C_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/launch-c")
set(CMAKE_CXX_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/launch-cxx")
set(CMAKE_CUDA_COMPILER_LAUNCHER "${CMAKE_BINARY_DIR}/launch-cuda")
endif()
message(STATUS "Using compilation cache: CCache")
endif()
else()
message(STATUS "Not using compilation cache")
endif()
to define the launchers for each used languages.
After integration of conan 2.0 :
conan create . -b missing
(where conanfile defines self.tool_requires("ccache/[~4]")
)I was thinking about creating a package option with_ccache: True/False
to define the CMake variable MY_PROJECT_USE_CCACHE
, but not sure if:
So... yeah, looking for advice on the subject, or to help on the matter ! Thanks in advance !
@memsharded Returning to the issue at hand, is it possible for Conan to use ccache/scache in CI to speed up the build process in CI?
Yes, it is definitely possible, we have seen quite a few Conan users doing it in production and at scale.
A different story is that if it needs extra configuration, adapters, etc, we don't know because it has never been contributed back (most companies are not allowed), so this issue here is for us to investigate, test, and document possible approaches. We haven't had the time to do it yet, we will try to address this when possible, but so far there have been other higher priorities.
Since build systems are heterogeneous regarding ccache, a consistent abstraction & behavior ensured by conan recipes would be nice.
CMake: opt-in, set CMAKE_<LANG>_COMPILER_LAUNCHER
to ccache
Meson: opt-out. Automatically enabled if ccache found unless CC and CXX explicitly set
Autotools: opt-in. You have to explicitly set CC
& CXX
to ccache <compiler>
Because conan may override CC and CXX in several helpers/toolchains based on tools.build:compiler_executables, there is no way for users to enable ccache irrespective of underlying subsytems, since it could break for cryptic reasons.
A topic that was always quite a weak point with Conan 1.x is the topic of build caching support. Obviously since Conan is not a full blown build system one may claim that Conan is also not fully responsible for this topic. Nevertheless it does have a very strong impact on the file system layout during build time and therefore should be held responsible for at least not preventing build caching or making it very difficult to implement/configure. The minimum I would expect/wish for is a description and some recommendations in the documentation how to use Conan together with ccache locally. A more advanced requirement would be to better support shared build caches like sccache or similar alternatives.