Add an environment variable pointing to the .conan folder

[MathPlayer]

I know about the CONAN_USER_HOME variable pointing to a folder in which conan will create the .conan folder. This allows for multiple .conan folders, especially useful for CI systems where one might want to build everything from scratch, maybe even based on a different settings repository.

Is there a reason against having a CONAN_HOME variable to point to the exact .conan folder? This seems pretty similar with the idea behind CONAN_USER_HOME, but with a better directory structure on disk.

I am using multiple conan config settings which are installed in different folders; I am switching between them by adjusting the CONAN_USER_HOME variable. The directory structure I currently have looks like this:

~
├── .conan_user_home_1
│   └── .conan
│       ├── remotes.json
│       └── settings.yml
│       ...
└── .conan_user_home_2
    └── .conan
        ├── remotes.json
        └── settings.yml
        ...

After having CONAN_HOME I can switch to something like this:

~
├── .conan_1
│   ├── remotes.json
│   └── settings.yml
│   ...
└── .conan_2
    ├── remotes.json
    └── settings.yml
    ...

[danimtb]

Hi @MathPlayer,

yes, I think this totally makes sense and in some cases the behavior of CONAN_USER_HOME is a bit strange. At least it was confusing for me at first.

I don't know if there is a reason behind this CONAN_USER_HOME instead of just a CONAN_HOME, so I will ask the team for feedback.

Thanks!

[danimtb]

Hi @mathieu,

I checked this with the team and we think the current CONAN_USER_HOME covers the use case you propose and the CONAN_HOME could be confusing, as usually, this points to the folder that contains the executable of the program (for example JAVA_HOME, https://confluence.atlassian.com/doc/setting-the-java_home-variable-in-windows-8895.html).

So finally, I agree, a direct variable to the CONAN_CACHE_FOLDER, or something like this may be useful, but it is already covered by CONAN_USER_HOME. We could consider this change for Conan 2.0 maybe

[memsharded]

Actually I did an attempt to remove the extra .conan folder, but it was a bit breaking. The internal code is prepared, it might be possible to make it the default in Conan 2.0.

[iiknd]

@memsharded is there any rough idea when the 2.0 should be released?

[danimtb]

@unzap I am sorry but the 2.0 version is still not in development, so we do not have an estimation for it

[memsharded]

Actually, the kick-off for starting to work on 2.0 will be in https://conandays.conan.io/ "hearing the tribes" session, to get initial feedback from the community to prioritize things. Conan 2.0 will be big, so expect at least 6 months of development.

[memsharded]

This has been already implemented in develop2 branch, will be in next 2.0.0-alpha2 (alpha1 already released). The variable (renamed) CONAN_HOME will point directly to the .conan (temporary assigned to .conan2 while the migration) folder, not its parent. So defining other value with ending different to .conan, will not create such child .conan folder, and will respect the given folder.

[iiknd]

Thanks for the update!

[hoyhoy]

What would really be good is if I could switch the configuration storage.path per conan profile. I have asan and msan builds that must have their dependencies instrumented with certain compiler and linker flags. As it is now, there is no way to handle this with conan. Why not just allow setting the storage.path in the profile? All I need is .conan/data_asan, .conan/data_msan, .conan/data_debug and so on. It would save hours of compile time. All I can do now is manually rename the data directories and rename it back when I use the profile I need.

[memsharded]

Hi @hoyhoy

You probably want to create a new ticket to discuss this.

Handling compiler options should be managed in a different way, not by changing the storage path. There are mechanisms in Conan to handle this like:

• Inject the flags with tools.build:cxxflags conf
• Define tools.info.package_id:confs=["tools.build:cxxflags"] to make the cxxflags part of the package_id

And voila, you have different package_ids for different compiler flags, with everything modeled and tracked correctly in Conan, also in the server side (note that the solution you are suggesting would be a disaster in the server side, basically overwriting the previous binaries of the same package_id without any control of the binary compatibility or the model, not knowing if the binaries in the server are those instrumented with sanitizers or not)

[hoyhoy]

AFAIK, my instrumented binaries in .conan/data aren't being uploaded to the remote, and our remote doesn't have these instrumented binaries cached. We're only using our artifactory remote for release builds with jenkins. These local *san builds are for devtest only.

[memsharded]

AFAIK, my instrumented binaries in .conan/data aren't being uploaded to the remote, and our remote doesn't have these instrumented binaries cached. We're only using our artifactory remote for release builds with jenkins. These local *san builds are for devtest only.

This is still a risk. At some point it might be necessary to upload the binaries, simply because the sanitized build is too long, the caches are invalidated too often, and it doesn't make sense to re-build all the sanitized builds from source for the whole dependency graph even. Or some bug will upload the binaries by accident.

Managing binaries is one of the things that Conan does automatically. If there are different binaries because of different builds, those binaries should be correctly modeled, there are different ways to do it, from defining custom settings (settings_user.yml) or adding options to the recipes, to defining conf and the effect of the conf on the package_id. You can even invent your own conf and add it to tools.info.package_id, but the important thing is that when you do a conan list mypkg:* you will see listed there the different package_ids of all the binaries and you will see there their settings, options, dependencies and conf that define such package_id and define that specific binary being different from other binary.

This is the recommended approach for managing binary variability, defining different storage.path would be against the core modeling of Conan package design, in which the location (cache, server, etc) of a package should never define the package contents.

[hoyhoy]

Yeah, the other issue we have is we're using a massive dependencies.yml file. I'd have to modify all 65 package_ids with something like this in the package id?

So, the package_id would change from:

redis/6.2.11@xxxxx/patched

to:

redis/-glldb-fPIC-Wl,-rpath,$CLANG_ROOT/lib/msan/patched-fsanitize=memory -fsanitize-recover=memory -fsanitize-memory-track-origins -fsanitize-memory-use-after-dtor -fno-common -fno-omit-frame-pointer -fno-optimize-sibling-calls -fuse-ld=lld -fsanitize=memory -L$CLANG_ROOT/lib/linux/ -L$CLANG_ROOT/lib -fsanitize-ignorelist=$PROFILE_DIR/msan_ignore_list.txt@xxxx/patched

Sixty-five times in our case.

?

[memsharded]

dependencies.yml

Sorry, I am afraid I don't know what you are taking about.

redis: full_name: redis/6.2.11@xxxxx/patched

No idea about this format/file either.

[hoyhoy]

This is a production vs devtest issue. What you're saying makes sense for a release build, but I've never uploaded a single binary to production because nobody cares about macOS Ventura clang ASAN nor Linux GCC 13.1 MSAN binaries in our organization. So it makes sense for me just store them locally. A coworker here actually wrote a script to create conan.conf and remotes.json on the fly because the only facility for managing local builds is $CONAN_USER_HOME which requires us to have to copy our ~/.conan directories over and over again. This is what his ~/.conan directory looks like.

_build.profile.clang.libcpp.sys.asan/conan
└── .conan
_build.profile.clang.libcpp.sys.debug/conan
└── .conan
_build.profile.clang.libcpp.sys.msan/conan
└── .conan
_build.profile.clang.libcpp.sys.release/conan
└── .conan
_build.profile.clang.libcpp.sys.tsan/conan
└── .conan
_build.profile.clang.libcpp.sys.ubsan/conan
└── .conan
_build.profile.clang.libstdcpp.sys.asan/conan
└── .conan
_build.profile.clang.libstdcpp.sys.debug/conan
└── .conan
_build.profile.clang.libstdcpp.sys.release/conan
└── .conan
_build.profile.clang.libstdcpp.sys.tsan/conan
└── .conan
_build.profile.clang.libstdcpp.sys.ubsan/conan
└── .conan
_build.profile.gcc.sys.asan/conan
└── .conan
_build.profile.gcc.sys.debug/conan
└── .conan
_build.profile.gcc.sys.release/conan
└── .conan
_build.profile.gcc.sys.tsan/conan
└── .conan
_build.profile.gcc.sys.ubsan/conan
└── .conan

I just self manage my data directories by mving them to-and-fro manually in one ~/.conan directory rather than rebuilding 65 dependencies from scratch.

[hoyhoy]

dependencies.yml

Sorry, I am afraid I don't know what you are taking about.

Another engineer here wrote some code to store dependencies in a yml file. He copied this from github. We're using that for dependencies rather than inlining over a thousand lines of Python in our conanfile.py.

We're pretty new to conan here. It's been just over a year since we switched from a self-managed ruby build system. It's likely we're living our lives wrong in multiple ways.

[memsharded]

This is a production vs devtest issue. What you're saying makes sense for a release build, but I've never uploaded a single binary to production because nobody cares about macOS Ventura clang ASAN nor Linux GCC 13.1 MSAN binaries in our organization. So it makes sense for me just store them locally. A coworker here actually wrote a script to create conan.conf and remotes.json on the fly because the only facility for managing local builds is $CONAN_USER_HOME which requires us to have to copy our ~/.conan directories over and over again. I just self manage my data directories by 'mv`ing them manually.

I am afraid that the fact that you manage to workaround it doesn't make it a recommended practice, and as such, we will not consider implementing such proposed feature in Conan. This approach of managing the binaries by putting them in different folders, even if they are not uploaded, is not correct. There are already built-in mechanisms in Conan to correctly model the binaries that are both built-in, but also more robust and future-proof.

I just self manage my data directories by mving them to-and-fro manually rather than rebuilding 65 dependencies from scratch.

This is an indicator that you might not be leveraging the system as much as you could. You insist on not uploading the sanitized builds, but at some point it is necessary. Otherwise it means that you need to rely only on things that live in a cache, by definition, ephemeral storage. Then unnecessarily rebuilding from source from scratch over and over, just because you didn't store the compiled libraries in permanent storage (the server). With growing projects, not uploading things become more and more problematic, because you can't either distribute builds in different servers.

It's been just over a year since we switched from a self-managed ruby build system. It's likely we're living our lives wrong in multiple ways.

Don't hesitate to ask! We try our best to help and support and advice users. You can create new Conan tickets for new questions and discussions. We also some times do zoom call with users (our availability is very low, but we try) to discuss approaches and give advice. We are here to help 🙂

[hoyhoy]

Yeah, I agree with what you're saying in theory. I could just store the instrumented binaries on the remote. But putting eleventy bajillion compiler options in the package_id seems insane too. I guess I could just make separate user and channel for the instrumented build types, but then I'd also have to maintain a dozen dependencies files (or update them on the fly from conanfile.py).

[memsharded]

But Conan offers different levels of abstractions. If you don't want to model at the compile flags level, that is perfectly understandable, then you should be modeling at the settings level.

We even have a howto for this: https://docs.conan.io/1/howtos/sanitizers.html (it is for Conan 1.X, but 2.0 should be pretty the same, just improved by the fact that you can maintain your own settings with settings_user.yml which is more convenient)

Do not use user/channel to separate based on the instrumentation, that is not a good approach either, as it requires to clutter the conanfile.py with extra stuff just to handle the channel variability.

[hoyhoy]

I already configured our settings.yml with sanitizer options per the doc a while back. (BTW, why not just include these sanitization options in the default conan.io settings.yml?)

compiler:
    gcc: &gcc
        address_sanitizer: [None, True]
        hwaddress_sanitizer: [None, True]
        kernel_address_sanitizer: [None, True]
        thread_sanitizer: [None, True]
        undefined_behavior_sanitizer: [None, True]
        pointer_compare_sanitizer: [None, True] # only valid with address or kernel_address
        pointer_subtract_sanitizer: [None, True] # only valid with address or kernel_address
        shadow_call_stack_sanitizer: [None, True]
        leak_sanitizer: [None, True]
        stack_protector: [None, True]
        stack_check: [None, True]
    clang:
        address_sanitizer: [None, True]
        hwaddress_sanitizer: [None, True]
        thread_sanitizer: [None, True]
        memory_sanitizer: [None, True]
        undefined_behavior_sanitizer: [None, True]

But, I still don't know how to manage my .conan/data output. As it is now, conan install just overwrites the dependency binaries per sanitization setting. How does a settings_user.yml fix this? Are you saying to upload different binaries to artifactory on a per-settings.yml-basis? What does that look like? I could in-theory upload the binaries to artifactory, but nobody else needs them other than me.

Oh, and CONAN_SETTINGS_COMPILER_SANITIZER seems not to be propagated for the conan 1.60 generator. Is there something else I have to do to make that work?

[hoyhoy]

Even if I could do something with settings.yml, uploading and then downloading 65-package binaries that I already compiled on my macbook just because I changed a compiler flag seems wrong too.

[memsharded]

I have created new issue for this: https://github.com/conan-io/conan/issues/13981, it is better not to hijack other issues if the conversation deviates from the original topic, specially closed issues.

[hoyhoy]

Oh, I see what you're saying. I didn't know that the package_id is automatically regenerated per conan profile. I'm testing it now with two different profiles in the same ~/.conan. It appears to be working. The individual package directories are different UUIDs. Presumably, our project will pick up the right the binaries. I actually have a separate conan profile per sanitizer because most of the packages don't support build_type=Debug nor verbose. Nearly all of them observe CXXFLAGS and the like though (sans openssl-fips).

[hoyhoy]

😂 It all works. I didn't need to do anything. The packages are all re-id'd automatically if I change the sanitizer. I thought there was just one package per dependency under data. That was our mistake. In conclusion, I don't need different ~/.conan/data directories at all. I just switched back-and-forth instantaneously from what was already built.

[hoyhoy]

Sorry, I didn't understand what you meant by the package_ids being unique per compiler settings.

[memsharded]

I created a new ticket in https://github.com/conan-io/conan/issues/13981 to follow this conversation there. Please post your messages in that new ticket, this one is already closed.