User friendly integration with ccache

[ogrisel]

Originally discussed here: https://github.com/conda-forge/conda-forge.github.io/issues/389#issuecomment-797642282

When installing the ccache and compilers from conda-forge in a given conda env, it seems that to enable ccache one has to manually set the CC and CXX env variable:

% conda create -n cppdev compilers ccache
% conda activate cppdev
(cppdev) % echo $CXX
arm64-apple-darwin20.0.0-clang++
(cppdev) % ls -l `which $CC`
lrwxr-xr-x  1 ogrisel  staff  8 Mar 12 18:45 /Users/ogrisel/miniforge3/envs/cppdev/bin/arm64-apple-darwin20.0.0-clang -> clang-11
(cppdev) % file `which $CC`
/Users/ogrisel/miniforge3/envs/cppdev/bin/arm64-apple-darwin20.0.0-clang: Mach-O 64-bit executable arm64

so to use ccache one would have to set:

(cppdev) % export CC="ccache $CC"
(cppdev) % export CXX="ccache $CXX"

whenever one conda activates that env which is quite a hassle when working on multiple conda environments.

Wouldn't it be possible to have some kind of hook or toolchain package to do that automatically up conda activate?

[isuruf]

Tried conda env config? See https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables

[jakirkham]

The compilers packages are already setting environment variables. AIUI this request is just asking about setting a couple more for ccache

Edit: Scratch that. Didn't realize a lot of the environment variables have been moved out of these packages. Nice!

[jakirkham]

Should add we might even be able to fold these into the existing c-compiler and cxx-compiler packages by checking for ccache and including it if present automatically. Not sure if that is too magical

[ogrisel]

Tried conda env config? See https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#setting-environment-variables

This works for my use case:

conda activate myenv
conda install compilers ccache
conda activate myenv   # to make sure the $CC $CXX are well defined for the `compilers` package
conda env config vars set CC="ccache $CC"
conda env config vars set CXX="ccache $CXX"
conda activate myenv   # to actually export the new ccache-enabled variables
echo $CC
ccache arm64-apple-darwin20.0.0-clang

This is a bit involved but it works if I deactivate / reactivate environments as long I do not uninstall the compilers package, but this is fine.

Should add we might even be able to fold these into the existing c-compiler and cxx-compiler packages by checking for ccache and including it if present automatically. Not sure if that is too magical

Not sure either. Maybe it's a bit too magical.

Maybe we could just better document that you can use conda env config vars set CC="ccache $CC" when installing the ccache package in the README of the ccache-feedstock repo? Or somewhere else, maybe on the conda-forge doc or on the compilers-feedstock README?

[ntorresalberto]

I'm trying what you describe but it seems to create 2 optional variables CONDA_BACKUP_CC and CONDA_BACKUP_CXX instead of CC and CXX which have the full path of the gcc and g++ binaries.

This is using gcc_linux-64 and gxx_linux-64 instead of compilers package, is that important?

[ogrisel]

This is using gcc_linux-64 and gxx_linux-64 instead of compilers package, is that important?

If you want to hard code those compilers, then the problem is easier. The following should work, no?

conda activate myenv
conda install compilers ccache
conda activate myenv   # to make sure the $CC $CXX are well defined for the `compilers` package
conda env config vars set CC="ccache /path/to/gcc_linux-64"
conda env config vars set CXX="ccache /path/to/gxx_linux-64"
conda activate myenv   # to actually export the new ccache-enabled variables

[nvaytet]

Say I would like to use ccache from conda-forge, but not the compilers from conda-forge. I have a project that uses cmake to find the system compiler. So at the time when I invoke the cmake command, the CC and CXX compilers are not known/set (it's cmake's job to find them). Hence I cannot use

conda env config vars set CC="ccache $CC"
conda env config vars set CXX="ccache $CXX"

because my compilers will then be ccache instead of ccache /usr/bin/cc. Any ideas how I could fix my conundrum? Many thanks!

[ogrisel]

I think the above solution described in https://github.com/conda-forge/compilers-feedstock/issues/31#issuecomment-800393655 is good enough.

It's easy enough to find it when googling.

[lesteve]

Reviving this old discussion, I somehow think that if I install ccache and compilers in the same environment, I very likely want ccache to be setup out of the box without having to tweak environment variables.

In other words, I would expect a variable like CC to be something like:

CC="ccache /home/lesteve/micromamba/envs/scikit-learn-dev/bin/x86_64-conda-linux-gnu-cc"

whereas currently it is:

CC=/home/lesteve/micromamba/envs/scikit-learn-dev/bin/x86_64-conda-linux-gnu-cc

It is not clear to me from the above discussion and linked issues whether this could be done in a straightforward manner. In particular, there is a concern about it maybe being too magical in https://github.com/conda-forge/compilers-feedstock/issues/31#issuecomment-797703779.

I have to say I looked a bit but I am not too sure where the CC variable gets set, whether it is in a particular feedstock or somewhere else. I was maybe expecting an activate script in compilers-feedstock but I could not find any.

One potential downside that was reported is an issue with scipy in https://github.com/conda-forge/conda-forge.github.io/issues/389#issuecomment-307187798 but I am pretty sure this works fine now that scipy has switched from numpy.distutils to Meson.

If I can help in any way this issue moving forward, let me know!

[lesteve]

I guess one of the activation script is https://github.com/conda-forge/ctng-compiler-activation-feedstock/blob/main/recipe/activate-gcc.sh and similar in the same folder.

Maybe there is a way to check if ccache is in the env (or even in the path although maybe it could pick up a system ccache which maybe could lead to some issues?) and set CC="ccache $CC" if that's the case. To be continued ...