[question] Testing methods for conan config

[michxymi]

What is your question?

Just wondering what approaches people use for testing a centralized conan config. Mainly about profiles....how can you verify that an exe or lib has been compiled as per instructed by the conan profile?

The approach I followed was to build the package you get from conan new and slightly modify the source to output to a file instead of cout, that I then parse and assert using python.

This mostly works, however, certain compilers have defs defined for compatibility purposes and that can ruin testing of certain features. For example, msvc has the cpp98 macro defined all the time so even if you are tell it to build with cpp17 for example, your test will fail, even though it has probably done it correctly.

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[memsharded]

Thanks for your question @michxymi

Just wondering what approaches people use for testing a centralized conan config. Mainly about profiles....how can you verify that an exe or lib has been compiled as per instructed by the conan profile?

The truth is that there are not absolute guarantees, even if you test things with a conan new, we have seen for example open source libraries that will hardcode some things like the CXX_STANDARD on their CMakeLists.txt, so no matter what Conan defines in the toolchain, and no matter what the conan new testing shows, that specific library might still be built with a different C++ standard than the intended one. This is a difficult thing from the outside of the source build scripts, and in general it relies on the CMakeLists.txt writers to follow good practices.

The approach I followed was to build the package you get from conan new and slightly modify the source to output to a file instead of cout, that I then parse and assert using python.

This is definitely something useful, I use conan new a lot to check things like this, but mostly when testing and debugging, not something automatic.

Another approach that you might want to explore is using hooks. You can write a hook that for example inspect binaries, and binaries can be inspected for certain properties using objdump or dumpbin, like the runtime linked and things like that. You might also use hooks to try to inspect the build logs to check for certain patterns (CMake for example outputs the current compiler).

The "metadata files" feature might be useful to store the build logs, to have them there for posterior analysis if necessary.

[michxymi]

Thanks @memsharded, that makes sense.

Maybe the best we can do is verifying that at least our config has installed the expected profile, and just verify the content, which I can do with conan profile show -pr:b foo -pr:h bar --output=json < foo.json and parse/assert this with python.

[memsharded]

Maybe the best we can do is verifying that at least our config has installed the expected profile, and just verify the content, which I can do with conan profile show -pr:b foo -pr:h bar --output=json < foo.json and parse/assert this with python.

but this seems a bit redundant? You are the one providing the profile, and defining it in conan install ... -pr=myprofile, so checkin that the output of the profile is the expected profile with a Conan command is kind of just verifying itself? I am not sure if I get the concern leading to this testing/verification? Sounds like verifying after a git clone that the code retrieved is the code that is in the git server?

[michxymi]

I basically want a way to verify that a profile can be used to compile a package. I also wanted to verify that the compiled package obey what the profile told but as we said that can't be guaranteed. We're also slowly migrating to conan 2.x and another thing to check is that the profiles work with a conan 2.x client after I do the necessary alterations.

[memsharded]

I see. So if you intend to kind of close the loop about the profile definition and the build system integrations, then yes, probably the conan new + conan create is a possibility. Also recall that you can have your own custom templates for conan new, if you need to customize them.

[michxymi]

I see. So if you intend to kind of close the loop about the profile definition and the build system integrations, then yes, probably the conan new + conan create is a possibility. Also recall that you can have your own custom templates for conan new, if you need to customize them.

That's good to know, at a later stage I might customize one, but for now I'll just use the cmake_lib option. Thanks for all your help!

[memsharded]

Thanks to you for the feedback!