Detect should show a warning about libcxx compatibility

[lasote]

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [x] I've specified the Conan version, operating system version and any tool that can be relevant.
• [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion. Read:

https://github.com/bincrafters/conan-boost_generator/pull/1#issuecomment-362856847

[a-teammate]

Okay, so its not only me? I was trying to debug these today, but hadn't enough time yet.

My setup is: Linux Mint, gcc 5.4, conan 1.0.4 (via pip) I am trying to build https://bintray.com/inexorgame/inexor-conan/Protobuf%3Ainexorgame#files/inexorgame%2FProtobuf%2F3.5.1%2Fstable%2Fpackage for Debug and Release. Neither is using the new cxx abi. When I build the project using that package with the exact same compiler, i get linker errors since it is expecting the new cxx abi. Seeing this ticket I checked whether using conan build for the consuming project fixes this error and it does. It is just problematic when manually executing cmake and make install for the consuming project.

The quickfix is to use the old cxx abi for our project when compiling with gcc 5, see https://github.com/inexorgame/inexor-core/pull/550/commits/76d3edcab46e93b48bd4c3b92b827c97e85732da. I actually though this was due to our protobuf recipe being corrupted.

Appending -D_GLIBCXX_USE_CXX11_ABI=1 or -fabi-version=0 to the environment variable CXXFLAGS didn't seem to work. I will try to modify the protobuf recipe now, to find out whether bypassing the cmake.command_line or build_config arguments fixes it. And maybe the options grafikrobot mentioned work better :)

To add some more information: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html is where one can find out about the ABI options. Actually gcc 5 should use the new abi by default, while it seems to using the old one when using conans cmake tool

[lasote]

Reading all the docs again and recalling all this stuff I think that probably we are not doing it very wrong. The libcxx values libstdc++ and libstdc++11 are exactly to model which ABI to use with gcc > 5.

The default profile when detected, set libcxx to the old compatible ABI, even with gcc >5 but it is only a default, you can change it to default your compiler defaults. We prefer to default to the most compatible mode.

So, why setting for libcxx are you selecting?

When I build the project using that package with the exact same compiler, i get linker errors since it is expecting the new cxx abi.

Is maybe your project selecting (or your compiler defaulting) the new ABI and you are linking with packages generated with old ABI?

What I think we could be doing wrong is using the options mentioned by @grafikrobot instead of the -D_GLIBCXX_USE_CXX11_ABI but it's probably working good anyway.

[a-teammate]

The libcxx values libstdc++ and libstdc++11 are exactly to model which ABI to use with gcc > 5.

Uhm there is a libstdc++11 option? :o didn't know about that.. well that seems to be the problem then. I thought it was just libstdc++ vs libc++ :/ hm okay

Is maybe your project selecting (or your compiler defaulting) the new ABI and you are linking with packages generated with old ABI?

Yes that's exactly it: I do conan install .. --build and it builds them with the old ABI, because we override the compilers default value in the cmake generator. But when I afterwards do not use the conan CMake() generator, i.e. by doing cmake .. && make install it will use the compilers default value, which is the new ABI. Hence I get linker errors afterwards.

Well.. if we would not make the libstdc++ and libstdc++11 differentiation, it would take the default settings of the compiler. So maybe we should adapt, which setting it takes by default, to mimic the behavior of the compilers defaults? So the user would need to explicitly opt-out?

Because if a person really wants to link to a binary with a different ABI they usually know what they are doing. Since it means you did not build all your dependencies with conan, but some with an older compiler. While they do not know what breaks when they use conan just for installing the dependencies, but not for building the project.

And well.. there are not many solutions to pick for users here:

• hardcode into the build system to also always use the old ABI
  • does not really seem like a good solution, since the break of the ABI was for the sake of some features probably (probably performance wise, but still)
• tell everyone to never use cmake manually but always conan build
  • probably gonna be hard to accomplish, since some IDEs like CLion execute cmake by default
• tell everyone to change their conan settings
  • or provide scripts for executing conan

[lasote]

Personally I think that we need to encourage this idea: the default settings are just a detection following a criteria that can be "wrong" for you, and you need to change them. Conan try to do nothing too smart, just act following the introduced settings.

For this special case of libcxx I commented in some other similar issue that we should print a warning when we detect gcc > 5, something like:

Conan detected gcc 7, we set for compatibility the compiler.libcxx=libstdc++ to be compatible with old ABI. But the default of your compiler can be different, please change the *.conan/profiles/default* file and adjust compiler.libcxx=libstdc++11 if you want to use the new ABI.

[a-teammate]

Hm yeah I see You try not to add extra complexity where it is avoidable, that is a good point, too.

Conan try to do nothing too smart, just act following the introduced settings

Could it be that the libstdc++11 option was just recently introduced and the "error" happened migrating the old profiles/default ? Maybe since my old settings were introduced in some early versions, the "libstdc++11" option just wasn't there yet and the "settings upgrader tool" didn't want to change on its own behalf "libstdc++" to "libstdc++11"? So people who download 1.0 do not have these issues? Since their standard settings are correct by default?

the default settings are just a detection following a criteria that can be "wrong" for you, and you need to change them

Yes I totally support that! That is the real decision to make here: what is the criteria and why?

Sidenote: The ABI option is just to keep compatibility between different compiler versions when using a library which is using the stdlib in its interface, but is compiled with an older compiler.

I mean there are three options: Clarification: For gcc >= 5!

1. Make it default to the old ABI
   • Pro: If the project they are working on has such a library (stdlib in the interface, compiled with an older compiler) they would profit from conan settings being working by default.
   • Pro: No code adjustments needed
   • Con: They would still need to hardcode in the projects compiler flags that they want to use the old ABI
2. Make it default to the new ABI
   • Pro: If people start with a minimal setup, it will just work
     • Minimal "criticial" setup: barebone 3 lines CMake file and only one dependency, which happens to use the stdlib in its interface, but is build using conan (hence with the same compiler)
   • Con:
     • Some complexity setting the std default
       • probably it is only a minor change: GCC manual chapter 6 if gcc >= 5 then libstdc++11
3. Do not set a default ABI
   • Do not override anything. Always use what the people set in their CXXFLAGS if they want a behavior different of their compilers default.
   • Pro: same as 2.
   • Con: more complexity as 2. because generating the package SHA you still want to include which libstdc++ it is using

Imo the novice user needs special treatment here, not the experienced one. Hence I would argue for making the setting defaulting to the new ABI. If someone is looking for getting libraries compatible which were not built with conan, but a legacy compiler -> awesome, they got their switch! Good idea to make the distinction between libstdc++ and libstdc++11! They can opt-out

Another note: if you switch on "compatibility" for the reason that a project with conan dependencies should be ship-able to old distros, then that's another Pro argument for case 1. But thats a really tough area and the correct solution for users would be to either make a self-contained binary (link everything statically, including the libstd), use a container software like docker, or create a native package (using the exact same compiler the distro provides). I don't think conan should aim for that, shipment is user territory. :)

[weatherhead99]

I may be very much in the minority here, but for the systems I largely use conan on, switching the default ABI would break pretty much everything. Of course, by now I'd know enough to change it back. But it's a very big thing to assume everyone is on the latest compilers etc.

[lasote]

@a-teammate that setting exist for many many months. I don't know where these problems could came from.

@weatherhead99 Don't worry, we are not changing the default. First because the commitment of stability of 1.0 and second because we don't think there is any problem with the current approach. The most important thing is that there is a 1 to 1 relationship between the setting chosen by the user and the ABI compatibility.

By the way, this issue was not opened to discuss the default ABI, but to investigate if we were adjusting it correctly. I really appreciate all the feedback anyway.

I will talk with @memsharded about some warning and better docs. After review it, I think the implementation of Conan is ok.

[grafikrobot]

Here's the basic question.. What is libcxx supposed to indicate for any compiler?

1. The ABI version to use when compiling?
2. Which standard library to use?
3. Both?
4. Something else?

If it's (1) then there are various places in conan and in outside recipes that are using it to signify the standard library to use. For instance in autotools_envrinment, and cmake_comon.

If it's (2) then there are places in conan and in outside recipes that are using it to signify the ABI version to use. For instance in autotools_environment, cmake_common, and boost_generator.

If it's (3) then there are places in outside recipes that are using it incorrectly, like boost_generator. But the more important aspect is that you are using one setting to signify two orthogonal aspect of compiling. This is, to say the least, confusing and limiting.

If it's (1), (2), (3), or (4) the behavior should be documented clearly some place. I failed to find if there's any existing documentation.

[a-teammate]

@weatherhead99

Of course, by now I'd know enough to change it back. But it's a very big thing to assume everyone is on the latest compilers etc.

A sorry I think I didn't put that quite clear: for me it is only about the default value for gcc >5. Never meant to make a suggestion to break any old compiler. It is only the decision: Should the value of the libc++ setting for a freshly installed conan be set to the value the users compiler has as default?

@lasote

By the way, this issue was not opened to discuss the default ABI, but to investigate if we were adjusting it correctly. I really appreciate all the feedback anyway.

Ups :D okay well maybe I'll open another ticket for that than. Or did you already decide against it? :) If yes, what was your reasoning for not using the compilers default? I mean I see that

we set for compatibility the compiler.libcxx=libstdc++ to be compatible with old ABI

but why? Is there any other reason other than the ones I mention below? Maybe I'm totally missing sth here

Although I could understand that you may not want to change that behavior now, being just arrived at 1.0.

A warning in the docs for new users that they absolutely need to change the libc++ version if they want to use a newer gcc than 4 would still be very needed (except if they always build their project with conan build or if they explicitly add -D_GLIBCXX_USE_CXX11_ABI=0 to their cxx flags)

@grafikrobot Your question is good!

If you want to mimic the cxx abi, it should be 1) ( and only 1))

To quote from https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html

In the GCC 5.1 release libstdc++ introduced a new library ABI that includes new implementations of std::string and std::list. These changes were necessary to conform to the 2011 C++ standard which forbids Copy-On-Write strings and requires lists to keep track of their size. [..] Although the changes were made for C++11 conformance, the choice of ABI to use is independent of the -std option used to compile your code, i.e. for a given GCC build the default value of the _GLIBCXX_USE_CXX11_ABI macro is the same for all dialects. This ensures that the -std does not change the ABI, so that it is straightforward to link C++03 and C++11 code together.

Means: it does not affect what c++ version the user wants. It is only about the compiled binaries compatibility. I.e. you build your project with gcc 6.0. You have a dependency which was built using gcc 7.2. You have another dependency which was built using gcc 5.4. It will magically still work, since their ABI is compatible. (Some may don't find this too special, but i.e. Visual Studio did/do break their ABI for every version -> you can't link your lib built with vs 2013 into an application built with vs 2015)

BUT from gcc 4 to gcc 5 they needed to break this awesome compatible ABI. Hence you can't use your dependency compiled with gcc 4.3 in the project using gcc 5.0. But the gcc stdlib people were nice people and put in a special switch: You can still use the old ABI with gcc >= 5.0 and still link that old dependency of yours. (the "OLD ABI" switch)

I didn't exactly grasp what you are sacrificing switching that switch, but Imo it is not necessary for the "default value" decision. It is just and only where conan wants to keep compatibility: between a project built with your compiler's default settings and the dependencies built with conan's default settings? Or between a project built with the legacy ABI enabled and the dependencies built with conan with default settings and possibly other legacy binaries built with older gcc versions?

The latter may be needed for companies having some closed source binaries they can't rebuild, but want to link into their project. While the former is what people expect conan to do imo: you execute conan install, you execute g++ or make -> it will just work.

However I do understand that you may not want to change that now in 1.0, and maybe never. (to not disturb companies which maybe already happen to have tons of computers set up with conan, but who are still needing the old ABI since they have one of these black-box dependencies) EDIT: Since the computers which are already set up are not affected by a default-value change, it will only mean that the companies maintainers remember to change the default value back to "libstdc++" if they set up new work places. Still a thing though..

Note: when I'm talking about "dependencies" it is always about "dependencies which use the stdlib in their interface", i.e. a logging library which takes a std::string as input. If they always use const char *, they won't be affected.

[lasote]

@grafikrobot Thanks for your answer!

The answer is 3) We are using libcxx to model the standard library, being the value libstdc++11 a specialization of libstdc++ to enable the modern ABI.

Totally agree that it can be confusing, especially if we don't document it properly, but limiting? in which sense? We definitely have to explain it better, at least in the docs.

[SSE4]

so far, my understanding is that compiler.libcxx setting actually models 3 orthogonal things at time (as for conan 1.0.4):

• C++ standard library (like libc++ VS libstdc++, or -stdlib compiler flag)
• libstdc++ ABI (which is that _GLIBCXX_USE_CXX11_ABI controls)
• C++ standard (which is not actually controlled, because it's different compiler flag -std, but name libstdc++11 suggests this) that causes a lot of confusion, misunderstanding and debates. the fact that setting is not well documented only makes everything worse, and confusing name forces people to misunderstand and misuse it.

so, my suggestion is to split this setting into 3 different settings, let's name em cxxstdlib, cxxstandard and libstdcxxabi (the latter applies only to libstdc++, obviously) but what should we do with numerous existing packages and recipes, and the fact that conan 1.0 must maintain backward compatibility and avoid breaking changes as much as possible?

I suggest to use settings preprocessor for that (the same entity which automatically defines compiler.runtime for Visual Studio) - settings preprocessor can convert legacy libcxx setting to the corresponding values of 3 new settings suggested. that way direct usage of libcxx will be still supported, but strongly discouraged for new recipes (conan may produce some human-readable warning).

the next step, is implementing helpers to abstract compiler specific details, so recipe and generator authors can re-use the same codebase to convert high-level conan settings to low-level compiler flags to be passed directly into build systems. PR #2424 basically implements this idea.

[memsharded]

compiler.libcxx doesn't model the C++ standard, or at least it doesn't try to do such. It models the C++ standard library. As the dual-abi is specific of libstdc++, I think that is nicely addressed with the current model, specifying libstdc++/libstdc++11. It is like it was a different library.

Regarding the C++ standard, it is ongoing work. There are currently 2 proposals to model them, one using settings and another using options. Both have important issues regarding compatibility and usability, so we are still working on them.

• Options: https://github.com/conan-io/conan/pull/2042
• Settings: https://github.com/conan-io/conan/pull/2030

What you propose of using a new settings has important challenges: it can easily break header-only and pure C libraries, that will get a new setting that is totally irrelevant to them. This cannot be solved with the settings-preprocessor, and requires explicit action from package creators modifying recipes, which cannot be mandatory until 2.0

[mropert]

@SSE4 The setting indeed is which runtime and which version. But instead of having a sub setting version, there are simply different names used to differentiate. I agree with you that this is not the prettiest and could be enhanced, but I don't feel like it's the most pressing issue.

As far the -std flag, the setting has actually no bearing to that, it just happens to have c++11 in its name.

So far I think (as you said) that the most important point would be to document which values Conan can handle and how they are translated.