[question] modify dependencies settings in build_requirements

[Johnnyxy]

hi there, I have a situation where I have to change the build_type for one or more dependencies dynamically in the build_requirement function. I found a thread https://github.com/conan-io/conan/issues/3000#issuecomment-405596294 where someone asked something similar and commented in that thread.

My question is, has there been some progress in solving such issues without statically describing the dependencies in a profile file? If not, how to distribute a profile file with a recipe?

Best regards

[memsharded]

HI @Johnnyxy !

No, there has not been progress in such issue, because we are not sure about such approach. The idea of having settings is exactly that they are configuration defined by the user. One package should not be able to modify the settings of another package, actually it has to be agnostic about it. Doing it other way is a path for lots of troubles and pain.

The only way that one package can provide information for its dependencies is via options, and even though we recommend not to hardcode it in recipes, leave the defaults as-is, if possible, then let the final user via profiles and command line to select their own configuration if they want to change it.

Lets say you do what you suggest in the other #3000 issue, and have a PkgA with:

def configure(self):
    if (str(build_type) == "RelWithDebInfo"):            
        self.settings["PkgB"].build_type = "Release"

Now you have hardcoded your dependency graph. If now you create RelWithDebInfo packages for "PkgB", you cannot use them, because "PkgA" will be hardcoding the "Release" one. Imagine the source of frustation for a developer that created "PkgB" with RelWithDebInfo because they need to debug something in "PkgB", and the system insists in using the "Release" version.

Also, it is another source for conflicts, now every diamond in the dependency graph can generate a conflict also in the settings.

If RelWithDebInfo is an important configuration for you I would recommend:

• Create RelWithDebInfo binaries for all packages
• If you have some very specific package that you want only Release binaries for it, write a package_id() method in it that makes it usable for RelWithDebInfo too, in a way that consumers doesn't have to manage it.

Managing profiles shouldn't be an issue. You can put profiles (they are just text files) in a folder in a repo, or you might use the conan config install feature to share profiles with your team and CI machines: https://docs.conan.io/en/latest/reference/commands/consumer/config.html#conan-config-install Please check if this would help. Thanks!

[Johnnyxy]

Hi, thanks for the explanation. I understand your concerns. I think it is my background with C where you can almost do anything if you want/has to but also have the freedom to make all mistakes :).

Let me clarify if I followed you correctly with the profiles proposal as conclusion of this situation.

1. I create a standalone profile for my piece of software with specific dependency version/options/settings.
2. Put the profile into the same repo with the conanfile or any other persistent storage.
3. With copying/checkout of the conanfile I receive the profile too.
4. Now I run "conan install" with (the hardcoded settings in) the profile.

[memsharded]

thanks for the explanation. I understand your concerns. I think it is my background with C where you can almost do anything if you want/has to but also have the freedom to make all mistakes :).

Yes, conan also gives a lot of flexibility, and gives a lot of freedom to make many mistakes. But in this exact case we didn't implement it, because of the subsequent problems it will lead to, and also because of the overhead of support tickets, debugging broken things... we prefer not going that way :)

Yes, your flow looks good, you can definitely put the profiles for the configurations you want to build inside of the repo with your piece of software. However, consider the case when you have multiple packages, all of them build with the same profiles, so you don't want to put the profiles in all the repos, that would easily lead to maintenance problems. In that case you can:

1. Put the profiles in a dedicated repo. In that repo you can also put remotes.txt file (to define your remotes), and your custom settings.yml file, for example, if you need it.
2. When you want to get them, use conan config install <url-repo.git>. That will install the profiles and the other files in your conan installation. You can use it for CI machines, and developers in their machines.
3. To update and get the latest profiles, you can use conan config install, the url is cached

You can also create a zip instead, and put it in a http server, but I find the git repo more convenient.

[Johnnyxy]

Sorry to interfere your day again. Now I have created some profile files.

• conan_profile_dependencies lists all dependencies

  
  include(default)

[options] dep1:option1=option1value

[build_requires] boost/1.67.0@conan/stable

- conan_profile_debug, conan_profile_release, conan_profile_relwithdebinfo
sets the build_type for project (and dependencies if necessary)
``` ini
[settings]
build_type=Debug

• conan_profile_shared, conan_profile_static

  [options]
  shared=True
  boost:shared=False

• conan_profile_shared_debug, conan_profile_shared_release, conan_profile_shared_relwithdebinfo

  include(conan_profile_dependencies)
  include(conan_profile_shared)
  include(conan_profile_debug)

On my CI system (Jenkins) I need to build with serval versions of a dependency (e.g. boost in this example). How can I do a batch build with different versions of one or serval dependency versions or dependencies? With profiles this will be a huge amount of serval files with many includes that build up specific build configurations or one file for each build configuration with serval redundant descriptions (settings, options, etc.) in it.

Now my followup question that there is no misunderstanding: Was it intended like that? Or did I misunderstood and took a wrong approach?

thanks for your time :)

[memsharded]

May I ask why are you using boost as build_require and not as a normal requirement? You might be abusing build requirements. They are intended mostly for tools (e.g. cmake), which do not have an effect on the binary ID of the packages being built. Linking with different versions of boost can produce different binaries, so you might want to take that into account, declaring it as a requirement in your recipes and maybe defining a package_id() method to customize the logic you want.

Then, build-requirements typically doesn't make sense to build with many different versions of it. When building with different versions of a normal requirement in conan recipes there is no explicit mechanism other than changing the version number of the conanfile.py and creating it again.

Typically this is not something that is done explicitly, but when the version is overriden by downstream consumers, new binaries can be created. (with --build=missing or similar). This is what I would do now:

• Use the test_package/conanfile.py in the conan create command.
• Put in the root conanfile.py the "default" or most used boost version, you want to use.
• Use requirements in test_package/conanfile.py, with a requirement "override" to boost, like:

requires = ("boost/%s@user/channel" % os.getenv(BOOST_VERSION), "override"),

• Fire it with

  BOOST_VERSION=1.65 conan create . user/channel
  BOOST_VERSION=1.66 conan create . user/channel
  BOOST_VERSION=1.67 conan create . user/channel

  You might need to use (set BOOST_VERSION=... and then call conan create later, for Windows)

Please try this approach and tell me. Thanks a lot for your feedback.

[Johnnyxy]

First of all shame on me. I misplaced boost and the other "prebuilt-libary" I have in build_requires not in requirements. Fixed that.

For your comments on boost, it was misused as example for my prebuilt-library I have to link to.

I will clarify. I am using boost but this requirement is static. But for a prebuilt third party library that comes from a third party vendor who supplies DLLs and Libs on a request basis for only specific versions, platforms (Windows, ARM, etc.), architectures (x86, etc.) and only as debug and release I have a conan recipe made for. This recipe resides with binary packages in the conan repo with the version number in the reference string "name/version@...".

Now my own software has to link against it. I have to be able to link my software against different versions of that third party library. That is the reason why I tried to parametrize this library requirement in my software's recipe that lists the library as requirement.

prebuilt library build_type issue

My first problem was that I have to solve the situation where I have a software.build_type of RelWithDebInfo. This build_type cannot be given to the prebuilt-library-recipe as I do not have "RelWithDebInfo"-variants for it. I tried to set the prebuilt-library.built_type in the "configure" function of my software's recipe and failed as the settings object of dependencies is not "subscriptable".

prebuilt library version issue

The second issue is that I have to be able to choose which version of the prebuilt-library will be used to link against.

solution

I the end your solution was to use profiles for that where I can set the build_type as well as version of requirements. I have to say this works. What makes me not so happy is that I have now 16 profiles for my software just to link against one version of the prebuilt library. When I want to support another version of the prebuilt-library I will get additional 16 profiles. I could optimize the redundant informations in the profiles by using includes. But that would not reduce the amount of profile files as I cannot set multiple profile files to combine them per command line (maybe a proposal/feature :)).

Content of a typical profile file: profile1.txt

List of profile files: windows_x86_64_vs14_lib531_shared_debug windows_x86_64_vs14_lib531_shared_release windows_x86_64_vs14_lib531_static_debug windows_x86_64_vs14_lib531_static_release windows_x86_64_vs15_lib531_shared_debug windows_x86_64_vs15_lib531_shared_release windows_x86_64_vs15_lib531_static_debug windows_x86_64_vs15_lib531_static_release windows_x86_vs14_lib531_shared_debug windows_x86_vs14_lib531_shared_release windows_x86_vs14_lib531_static_debug windows_x86_vs14_lib531_static_release windows_x86_vs15_lib531_shared_debug windows_x86_vs15_lib531_shared_release windows_x86_vs15_lib531_static_debug windows_x86_vs15_lib531_static_release

[SeanSnyders]

Are there any improvement on this situation with the profile proliferation? It seems to be an issue we would very likely run into as well, supporting many target platforms, versions of compilers, and c++ standards.

[memsharded]

https://github.com/conan-io/conan/issues/10995#issuecomment-1092830058 summarize updated thoughts about this. In short: recipes will not implement functionality to define settings for themselves or for their dependencies. Lots of experience with the options show this is a complicated thing.

On the other hand, profiles have gained some interesting functionality. Besides being able to be composed, both in command line and with include(otherprofile) in the profiles, and with single arguments in command line, now profiles can be jinja2 templates, which allows dynamic definition of values and python-like syntax. That can really help to reduce the number of profiles if necessary/wanted.

So I think this question can be closed.