[question] Request clarification on builds using multiple "sets of settings"

[gdray-dev]

What is your question?

I looked at the blog posting about building with two different compilers and this is some what relevant to my question.

https://blog.conan.io/2016/12/22/Building-multi-compiler-C_and_C++-applications-with-conan.html

I did a simple test case where recipe "module" depends on "xilinx" which depends on "component". When I specify different settings for "xilinx" in the profile used to build "module", then conan attempts to build "component" with the default settings in the profile, conan does not propagate the "xilinx" settings to "component".

The use case here is that I want to build a XILINX FPGA binary using GCC (baremetal, Microblaze arch) and then embed that binary in the "module" component which is an ARM Linux process build.

So the package "component" is a library that get linked with xilinx into an FPGA and would clearly need to have the xilinx settings propagated to it. But instead, the "default" ARM Linux settings are propagated.

So the documentation for specifying changed settings using the format below kind of implies that it would only apply to that package. That's why I did the quick test.

My question is: What would be the best way to achieve my goal of doing a "sub build" of a package with different settings in the best way that Conan would understand dependencies. I could clearly just build the two as separate builds and somehow "paste" the XILINX image into the ARM Linux build as a data file but if Conan was aware that the XILINX build was actually a full C build it could better track dependencies. Am I simply off base or asking too much? While I'm sure this is not a mainline use case, I could see this being needed for quite a few situations. I would even say that propagating the settings down should have been the default behavior. In the example provided of building openssl with a different compiler (clang), it would be a reasonable idea that I would also want dependencies of openssl built with clang.

Thanks for any ideas or suggestions on how to proceed.

Example Profile:

[settings]

os=Linux arch=armv7 compiler=gcc compiler.libcxx=libstdc++11 compiler.cppstd=14 compiler.version=6.2

xylinx:os=BareMetal xylinx:arch=Microblaze xylinx:compiler=gcc xylinx:compiler.libcxx=libstdc++11 xylinx:compiler.cppstd=11 xylinx:compiler.version=6.2

Have you read the CONTRIBUTING guide?

• [ ] I've read the CONTRIBUTING guide

[memsharded]

I did a simple test case where recipe "module" depends on "xilinx" which depends on "component". When I specify different settings for "xilinx" in the profile used to build "module", then conan attempts to build "component" with the default settings in the profile, conan does not propagate the "xilinx" settings to "component".

Yes, this is true, settings are not propagated from one recipe to another, they are global they go directly from profiles to every package recipe.

So the package "component" is a library that get linked with xilinx into an FPGA and would clearly need to have the xilinx settings propagated to it. But instead, the "default" ARM Linux settings are propagated.

If you need component to have the xilinx configuration (settings, options), you need to define that in the profile too.

In your case, it would mean adding to your profile:

xylinx:os=BareMetal
xylinx:arch=Microblaze
xylinx:compiler=gcc
xylinx:compiler.libcxx=libstdc++11
xylinx:compiler.cppstd=11
xylinx:compiler.version=6.2

component:os=BareMetal
component:arch=Microblaze
... # etc

If the issue is that you will have more than 2 packages there could be different approaches:

• You can use profiles that end with .jinja and are jinja templates, so you can easily do for loops, something like:

  {% for pkg in ("xylinx", "component", ...)  %}
  {{pkg}}:os=BareMetal
  {{pkg}}:arch=Microblaze
  {{pkg}}:compiler=gcc
  {{pkg}}:compiler.libcxx=libstdc++11
  {{pkg}}::compiler.cppstd=11
  {{pkg}}::compiler.version=6.2
  {% endfor %}

  You can also try another approach, like relying on patterns, lets say that all your packages that needs to be built with the xilinx settings are called pkg/version@company/xilinx then:

*/xilinx:os=BareMetal
*/xilinx:arch=Microblaze

[gdray-dev]

Thank you for your response. The possible use of the jinja feature in profiles had not clicked with me. This is a really good feature and I will have to look at it more. Your ideas will probably mitigate my issue for the most part but it will still leave the situation that client users of XILINX builds will have to "track" package changes in these builds. I also wonder about the case where the XILINX build and the Linux build both use the same package (say a shared communication package compiled on both sides). I will deal with that if/when the case arises.

I had also looked at a couple of posts with respect to invoking Conan from inside a Conan build. This actually seems to be closer to what I need than to try and mix architecture builds under one profile. My concerns when reading about this were that the Conan Python API is apparently not "publicly supported" until the 2.0 release. Would doing a "nested build" through the API seem more appropriate to you in this case? My only concern is that the top level build would need access to the package variant that matches the XILINX build, not one that matches the client Linux build. Does such an ability to access a package variant defined by different settings exist.

I have also tried really, really hard not to post this following question but I am concerned to move our build environment to 2.0 until it formally moves out of Beta. Without asking for any commitment, would you have any hint of when I might want to plan to reserve time to do my migration from 1.X? There is so much good stuff in 2.0 that seems like it would really help.

Thanks for your help.

[memsharded]

Thank you for your response. The possible use of the jinja feature in profiles had not clicked with me. This is a really good feature and I will have to look at it more. Your ideas will probably mitigate my issue for the most part but it will still leave the situation that client users of XILINX builds will have to "track" package changes in these builds. I also wonder about the case where the XILINX build and the Linux build both use the same package (say a shared communication package compiled on both sides). I will deal with that if/when the case arises.

It feels that the best approach could be to manage the 2 dependencies graph separately, as the case where mixing completely different subgraphs of same packages but built with radically different compilers and architectures together is not built-in in Conan.

I had also looked at a couple of posts with respect to invoking Conan from inside a Conan build. This actually seems to be closer to what I need than to try and mix architecture builds under one profile. My concerns when reading about this were that the Conan Python API is apparently not "publicly supported" until the 2.0 release. Would doing a "nested build" through the API seem more appropriate to you in this case? My only concern is that the top level build would need access to the package variant that matches the XILINX build, not one that matches the client Linux build. Does such an ability to access a package variant defined by different settings exist.

Yes, the good news is that the capabilities in Conan 2.0 are beyond a public API. There are custom commands, and the API is not centered around commands, but low level features. So it is possible to implement a custom command that builds 2 different subgraphs independently, each one with a different profile, and then finally combining everything with a deployer or something (It would be interesting to know how the final thing is connected. How are the binaries built for xilinx work together with the host binaries? Are they really linked together at some point? How do they execute together?)

I have also tried really, really hard not to post this following question but I am concerned to move our build environment to 2.0 until it formally moves out of Beta. Without asking for any commitment, would you have any hint of when I might want to plan to reserve time to do my migration from 1.X? There is so much good stuff in 2.0 that seems like it would really help.

We are aiming at mid Feb for Conan 2.0 GA. Still not set on stone, but this is our current thinking. It is important to know that most of the migration work (recipes) can already be done in 1.X. I would start the work now, setting up a pipeline with the Conan 2.0 beta and seeing what breaks and what not, so smaller, incremental changes can start happening already. Also the sooner we get feedback of something not working during the beta, the better to fix it early and have a more stable 2.0.

[gdray-dev]

Thank you, these are all great answers. While I've been holding off, it's been clear to me for a while that 2.0 will be a great step forward. This issue has provided the extra motivation to move.

We're building embedded applications on a XILINX 7020 ARM SOC so we have a built in FPGA. The FPGA supports a MicroBlaze soft processor core so the overall applications can be split between the ARM core and the Microblaze soft core. The "joining" of the two is rather low tech as the MicroBlaze code is seen as a blob of data to the ARM software. But once loaded to the FPGA, ARM and Microblaze communicate by registers, shared memory and interrupts. So no true "compiler linking" or direct function calls between the two sets of code but we still need everything to be a consistent build. We're trying to convert our build over from some pretty out of control makefiles. I'm hoping and it certainly appears like Conan can bring some sanity to the situation. If I see anything unexpected, I will certainly provide feedback on what I see. Thanks again for your help and comments.

[memsharded]

We're building embedded applications on a XILINX 7020 ARM SOC so we have a built in FPGA. The FPGA supports a MicroBlaze soft processor core so the overall applications can be split between the ARM core and the Microblaze soft core. The "joining" of the two is rather low tech as the MicroBlaze code is seen as a blob of data to the ARM software. But once loaded to the FPGA, ARM and Microblaze communicate by registers, shared memory and interrupts. So no true "compiler linking" or direct function calls between the two sets of code but we still need everything to be a consistent build. We're trying to convert our build over from some pretty out of control makefiles. I'm hoping and it certainly appears like Conan can bring some sanity to the situation. If I see anything unexpected, I will certainly provide feedback on what I see. Thanks again for your help and comments.

Super interesting feedback, thanks very much for the detailed explanation. At the light of this, yes, I think a custom command that builds 2 subgraphs would be the right approach. If further consistency checking of versions is necessary, then it would also be doable in the custom command. Thanks again for all the feedback!