[question] Parse buildenv variables in conanfile.py

[donlk]

What is your question?

Environment

• OS: Debian 11
• Python version: 3.9.2
• Conan version: 2.0.0

Hi there! Scenario: Let's say there is a library with the autotools build system, particularly FFmpeg. Now, FFmpeg has some interesting quirks regarding it's build system, as it ships with a rather odd configure file. Whenever the user cross-compiles, it always uses the system-wide compiler binaries, if not told otherwise explicitly. Say we want to cross-compile from Linux to OSX using an osxcross toolchain. Having CC=<toolchain>/bin/arm64-apple-darwin21.4-clang is not enough, we have to supply it with: --cc=<toolchain>/bin/arm64-apple-darwin21.4-clang

This is particularly problematic, as conan altered the environment variable setting in 2.0 amongst many other things, so instead of defining a simple [env], we have to use [buildenv] and [runenv]. This is fine in itself, however, the access to these specific env variables are quite tricky. There is not documentation on this, so I had to come up with this solution:

CC = self._conan_buildenv.get_profile_env("").vars(self).get('CC')

However, this will result in the following value in our case (see toolchain profile below): $toolchain_path/bin/$target_host-$cc_compiler The variables are not expanded. I either have to do manual parsing by iterating over all variable references, or I'm missing something. I also cannot burn-in the full toolchain path variables, as these will be parsed from a property-like file from the client side. Any help is appreciated.

Ps: The official conanfile for ffmpeg is outdated, many functions (including the env variable handling) are deprecated, I cannot use that unfortunately for conan 2.0.

Cross toolchain profile (osxcross.jinja):

[buildenv]
...
CHOST=$target_host
CC=$toolchain_path/bin/$target_host-$cc_compiler
CXX=$toolchain_path/bin/$target_host-$cxx_compiler
cc_compiler=clang
cxx_compiler=clang++
toolchain_path=<path>/macosx-12.3-cross-toolchain
target_host=aarch64-apple-darwin21.4

(The reversed ordering is intentional, as conan exports the last entries first)

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @donlk

Thanks for your question. It is possible to access values of env-vars in your conanfile with https://docs.conan.io/2/reference/tools/env/envvars.html#applying-the-environment-variables, so you can use apply() to activate them in Python environment or you can iterate them with items(), or get the value with get() The VirtualBuildEnv() can also return vars() https://docs.conan.io/2/reference/tools/env/virtualbuildenv.html#conan-tools-env-virtualbuildenv

So all together:

def generate(self):
    buildenv_vars = VirtualBuildEnv(self).vars()
    cc = buildenv_vars.get("CC")

This is better than the above, because it also takes into account possible per-package env-vars (it is possible to define env-vars in profile per-package if necessary too)

[donlk]

Yes, that is much nicer, thank you. However, this did not solve the lack of variable expansion. What solved it is to use jinja profile rendering at the beginning of the toolchain profile:

{% set toolchain_path = "<toolchain>" %}
{% set target_host = "arm64-apple-darwin21.4" %}
{% set cc_compiler = "clang" %}
{% set cxx_compiler = "clang++" %}
...

Thank you again, closing the issue.

[donlk]

Actually, one more question: How do I acquire VirtualBuildEnv() for the host and build toolchain profile separately? I assume the default VirtualBuildEnv() object corresponds to the host configuration. Is it the scope parameter that controls this?

[memsharded]

Yes, the above is right. Textual variable replacement has been dropped in 2.0, it is necessary to use jinja

How do I acquire VirtualBuildEnv() for the host and build toolchain profile separately? I assume the default VirtualBuildEnv() object corresponds to the host configuration. Is it the scope parameter that controls this?

In theory you don't. It is the responsibility of the recipes to define the self.buildenv_info and self.runenv_info correctly. Some recipes might add information that is necessary for building them, even if it is the "host" scope. Not common but possible. So the environment separation is not by "build" and "host", but environment needed for build and environment needed for run.

[donlk]

I see your point, however, there are a number of libraries that produces binaries (protobuf files, shaders, etc) mid-build that is used to create target binaries (VTK, QT just to name a few). In these cases, we need to do a pre-step of compilation that takes care of building these binaries with the build toolchain, then move on to producing host (or target) binaries with the other. How should we handle these cases with conan?

[memsharded]

Not sure if I understood your case.

In the protobuf case, for example, lets say that protobuf is being used in a cross-compilation scenario cross building from Windows to Linux. This requires 2 different binaries for the protobuf package, with 2 different package_ids:

• The Windows package_id (lets call it pkgidwin), builds the libprotobuf library with MSVC, then the protoc compiler also with MSVC, linking with the previous libprotobuf and package both in the binary identified with pkgidwin.
• Same for the Linux binary, lets call it pkgidnix, builds both the library and the protoc with gcc for Linux.
• Then, some library mylib, that implements some protobuf protocol will do a tool_requires = "protobuf/version" and requires = "protobuf/version". When the graph is resolved, the tool-requires will resolve to protobuf/version:pkgidwin in the build context and the requires will resolve to protobuf/version:pkgidnix in the "host" context.
• The "build" context protobuf will put itself in the PATH environment variable via VirtualBuildEnv, so the mylib can easily run protoc to compile .proto and generate the sources. The "host" context protobuf will be the one that creates the protobuf-config.cmake that find_package(protobuf) will find the linux "host" library to link with it.

[donlk]

Perhaps protobuf was not the best example in my case, as it is quite more straightforward as you described. However, in the case of Qt, things are a bit more sketchy, as I need to build certain binaries for the build platform that only Qt uses in the host build process. If I'd apply the same principle as you described for protobuf, I'd have to put tool_requires = "qt/version" in the Qt conanfile. I would rather not do that, and I think it would create a circular dependency issue anyway. Not to mention that I would need to specify the exact arch, march and platform attributes for this requirement to match the build machine's ABI.

Could I perhaps use two different conanfiles for the host and build separately? These would have to be in the same repo in my case.