search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
945 stars 1.72k forks source link

[service][rfc] Configurations built by the service #4761

Open danimtb opened 3 years ago

danimtb commented 3 years ago

In the latest release of changes to the pipeline of the service, we added support for new compilers, gcc 10 and clang 10 and 11 in Linux and apple-clang 12.0 in Mac OS. As stated in the supported platforms and configurations, due to the frequent updates of the apple-clang compiler, we decided (right from the beginning of the conan-center-index) to rotate the latest 3 versions of it to keep the pace of new versions and avoiding scalability issues spending too much time generating many binaries that will not be used. Therefore, we will stop generating packages for the apple-clang 9.1 compiler soon.

To better serve the users and to have a fast CI for contributors, it is necessary to be efficient and select the configurations built by the center carefully. Otherwise, generating more and more configurations will get very challenging over time. We are working on getting some stats about the usage of the packages in ConanCenter so we can have a better perspective around this.

However, we wanted to ask you about it to know what do you think might be the considerations to stop supporting old compiler versions and, for example, move all the compilers to that same rolling version policy, meaning that we could only generate the latest 3 versions of each compiler or maybe keep generating binaries for the oldest compiler version and the two latest (again, these are only examples).

With that, the time to build packages in PRs could be reduced significantly and would open the possibility to support other configurations that might be of interest to more ConanCenter users without facing enormous scalability challenges in the future due to the number of packages to be generated.

Thanks

Croydon commented 3 years ago

Here is my personal take on it.

macOS Apple Clang

This is probably the easiest one. macOS developers seem to be used to targeting newer compiler versions. We also cut old Apple Clang versions rather regularly in Bincrafters and I never heard anyone complain. I think the current policy of only generating the latest three compiler versions is fine.

Windows MSVC

CCI supports 2015, 2017 and 2019 currently. I wouldn't be entirely surprised if VS 2019 becomes an evergreen IDE similar like Windows 10 has become evergreen. VS is now much more modular and ships regularly new features.

Thus said, the discussion is more about the compiler versions and not the IDE version (which is also better reflected in the new MSVC compiler setting in Conan).

I would say that supporting the three latest MSVC versions and rotate out the oldest one if a new one gets released, should be fine. Identical to the macOS situation in its handling, but with the huge difference that there are not that often new MSVC versions (if you don't count minor releases). So, if you compare the years of support for an individual MSVC version, it would be much longer supported than the support for an individual Apple Clang version, which is probably fitting, given that Windows development is more conservative.

Linux Clang

As far as I know, Clang is not the default compiler on any major Linux distribution right now. Hence, I would not mind a schema with a fixed number of supported versions and rotating out the oldest ones.

Currently, Clang 3.9+ is supported. With 9 compiler versions in total.

Clang 3.9 got released in Sep 2016. Clang 4 in March 2017 and Clang 5 in September 2017 - just to name the oldest (Source).

They are releasing every 6 months a new version which does strengthen the argument to only support a fixed number of compiler versions. Basically, we can think in years of support which makes it less abstract.

Example: We want to support all Clang versions of the last 4 years, hence we support the latest 8 Clang versions.

This is somewhat arbitrary and more a feeling, but 4 years of support would sound good to me.

Linux GCC

Given that GCC is the default compiler in all major Linux distributions (of which I am currently aware), this might be the most complicated case regarding which versions are meaningful to support.

Currently, GCC 4.9+ is supported. With 7 compiler versions in total.

GCC 4.9 got releases in April 2014 and since then one release per year, which is their release schedule (Source).

Meaning, if we support the same amount of GCC compiler versions as we do with Clang versions, we cover twice as many years of support for an individual compiler release. Cutting off the old GCC versions is probably the most painful, hence supporting GCC versions longer than Clang versions might make sense.

Again somewhat arbitrary and more a feeling, but 8 years of support a.k.a the latest 8 GCC versions might sound good - for instance, RHEL versions are also supported for 8 years, if you don't count the extended support - and it would be the same amount of supported compiler versions than for Clang.

The issue description goes more in the direction of reducing what we already have, while my suggestion goes more in the direction of not letting it grow further.

Another thing which might be considered in this regard: Maybe it does not have to be an all-or-nothing thing. Example: We want to support the newest 8 GCC versions, but only for the newest 4 versions we build all configurations and for the others only a subset of those. This would reduce the number of total builds further without cycling out older compilers (too) quickly.

Reducing the number of builds also ties in with #2813. With the differences, that this issue is about the general configuration support and #2813 would allow recipe specific limitations in cases it makes sense, i.e. something like #2813 would also help to bring CI time down for the heaviest recipes we have.

When it comes to new potential configurations, I'm most interested in Clang for Windows, but at this point, I'm not even sure if having binary packages for this makes much sense as it aims at being ABI compatible with MSVC. Having test coverage via CI might be great though.

a4z commented 3 years ago

Since packages can be rebuild and cached and archived, removing exotic compiler is a good idea.

I think everyone in professional environment ends up to rebuild everything them self anyway. (and if they don't they should)

So the binary packages from the cci are maybe only a time to hello world reducer for new conan user and hobbyist. This is important, but I think even for new users it is understandable that software needs to be build. (possible adopting the docs to use packages that build fast can do a fast time to hello world for others)

Therefore, imho, everything beside most recent/ common compiler can be removed, since I think most new users will either have current compiler, or understand that software needs to be compiled 1 time.

So maybe keep the last 2 of every kind (MSVC, XCode/Clang, latests clang on Linux) and on Linux the 2 most popular, might be current ubuntu and latest ubuntu LTS

jgsogo commented 3 years ago

I'll try to highlight one of the main issues here (from my perspective): this is not a matter of storage, if that was the problem we could remove packages from older revisions and we will save many gigabytes.

The problem is that we can only be sure that the package builds for a given compiler if we actually build it (once we build, we will store). I totally agree that building from sources is not a big handicap... if the recipe works.

dmn-star commented 3 years ago

Would you like to have packages generated for other compilers or architectures?

Yes Android & iOS

a4z commented 3 years ago

Yes Android & iOS

for this we might first have this https://github.com/conan-io/conan/issues/7876 resolved

and then, there are so may settings, like os.api_level/ Android os.version/ iOS , where it might be good to agree, so that looks a bit more complicated

prince-chrismc commented 3 years ago

Would you like to have packages generated for other compilers or architectures?

I would also really appreciate Alpine binaries but https://github.com/conan-io/conan/issues/3972 🤷 Is it doable for only CCI? https://github.com/prince-chrismc/user-management/commit/881b28df49f267d8e176b81b1e703503688cda94#diff-981783d07c49301cd9b135809fe286aaed8d8c2e2dcfe391663bb4f1c37f0721R53-R54

n-bes commented 3 years ago

I think that main goal of cci is contain verified recipes. Binaries are optional. So toolchains (environments) count should be maximal. Recipes can have different support level:

package \ toolchain gcc 4 gcc 10
boost recipe only recipe and binaries
chgans commented 3 years ago

What are the plans regarding android? Most (all?) the libs i tried to install had to be built from source

jgsogo commented 3 years ago

Hi, @chgans . We are not generating binaries for Android (yet), so it is expected that you need to build from sources all the libraries. We probably will, but there are some steps to take before, this is the plan:

It will take time, but we have already started 😊

BTW, can I ask you about your experience building the libraries for Android? Do they work (most of them)? Are you using our android_ndk package as a build-requires? Thanks!

chgans commented 3 years ago

Hi,

We would like to switch our deps management from a completely inhouse adhoc system to conman w/artifactory. So i'm only investigating for now.

I'm using the official Conan docker container for Android. Took me some time to get my setup right (build/host profiles), but i'm finally able to get this working on armv8:

[requires]
boost/1.76.0
benchmark/1.5.3
mosquitto/1.6.12
protobuf/3.15.5
rapidjson/1.1.0
sqlite3/3.35.5
# libtorrent/1.2.3
openssl/1.1.1k
gtest/1.10.0
# libusb/1.0.23
# qt/5.15.2
# To be created:
# libs3
# libmarnav/0.10.0

[options]
qt:opengl=es2
libusb:enable_udev=False
boost:i18n_backend_iconv=libiconv

Not bad! libtorrent problem is #5666, libusb problem is #5675, trying to get Qt now!

If we do the switch to conan, then we'll have to maintain our own repo as we target custom embedded Linux, Android, Windows and Linux 64 bits. all of these in 32 and 64 bits! (:cry:)

I haven't attempted to do actual builds for Android, that will be next step in my investigation.

We'll use our own container, eg. Full Android SDK/NDK, conan, jfrog, .. with prebuilt Qt (and other monsters).

Update: Qt fails, as it needs the SDK, I wonder if Conan's Qt for Android supports multi-arch (Qt does)

dmn-star commented 3 years ago

Android builds on CCI are long overdue, almost every recipe update ends in tedious troubleshooting. For example now again boost #5747

jgsogo commented 3 years ago

FYI: https://github.com/conan-io/conan-docker-tools/pull/267, it's also relevant for this issue.

jgsogo commented 3 years ago

Hi! After a long journey, we have prepared some new docker images that we are planning to use to generate the packages in ConanCenter (it won't happen soon, we need to define how to transition to those images). You can read more about it here:

Main implications:

I wanted to share this with you here first, because we are considering removing NOW those compilers and configurations that won't be generated in the future. Also, it is more or less aligned with previous comments. Does anyone sees a big problem if we remove those old compiler versions?

Thanks for your feedback.... and please, spread the word about this topic.

ericLemanissier commented 3 years ago
  • we won't generate GCC 4.9 and Clang versions older than 10.0.0.

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

SSE4 commented 3 years ago

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

this is about regular clang. that's all about Linux docker images right now.

jgsogo commented 3 years ago
  • we won't generate GCC 4.9 and Clang versions older than 10.0.0.

Do you mean apple-clang older than 10.0.0 ? Or will there really be only 2 versions of clang (10 and 11) ?

For macOS we will keep generating the latest three major versions (10, 11 and 12). We have started to collect metrics and we will have information about the configurations being requested, that will help us to decide if we should stop generating some of them... or if we should add others. For gcc-4.9 and linux-clang it is more a technical reason given the docker images we will be using.

jgsogo commented 3 years ago

We already have some data from the previous 7 days in ConanCenter. These are install requests for the compilers we are building right now:

Windows:

compiler
Visual Studio 14 0,81%
Visual Studio 15 11,06%
Visual Studio 16 88,13%

Linux

compiler
GCC 4 0,81%
GCC 5 4,59%
GCC 6 0,57%
GCC 7 26,39%
GCC 8 15,30%
GCC 9 35,81%
GCC 10 16,53%
compiler
clang 3 0,19%
clang 4 0,05%
clang 5 0,39%
clang 6 4,70%
clang 7 9,96%
clang 8 3,65%
clang 9 20,60%
clang 10 32,24%
clang 11 28,22%

Macos

compiler
apple-clang 10 1,11%
apple-clang 11 21,95%
apple-clang 12 76,94%

Given these numbers IMHO we can start thinking about deprecating some old compilers and be a bit more aggressive with it. Wdyt?

jgsogo commented 3 years ago

(Linux context) My biggest concern (given the technical reasons for docker containers) would be clang 9 (20,60%), why is it so common? Is it the default for Linux distributions?

a4z commented 3 years ago

the Linux GCC Number is slightly surprising, but on the other hand, gcc 7.5 is the default compiler on ubuntu 18.04 LTS , and gcc 9 on 20.04 , so no big surprise either ....

I do not think there is a Linux dist with default clang, so that is surprising

ericLemanissier commented 3 years ago

clang9 is the default clang on openSUSE Leap 15.2

dmn-star commented 3 years ago

I do not think there is a Linux dist with default clang, so that is surprising

OpenMandriva :-)

Llvm Download for Linux ( clang version <-> distro ) https://pkgs.org/download/llvm

jgsogo commented 3 years ago

Checked clang versions (by default I meant the one that is installed with apt-get install clang):

dmn-star commented 3 years ago

@jgsogo see my comment

chgans commented 3 years ago

We have a massive code base stuck on clang-9. Linux people interested in building their own stuff with conan are IMHO unlikely to use default distro compiler. If you like default stuff, then you would use apt instead of conan to get your lib and dev packages installed. As well, the compiler survey doesn't include Android. Do you guys have any figure regarding Android?

jgsogo commented 3 years ago

Hi @chgans, we are trying to get information (numbers) and reaching out so we know better the impact that any decisions could have.

We will try to keep building configurations as long as they are used, but on the other hand, we should consider resources and the time it takes to build each pull-request and provide feedback to contributors. We also need to take into account human resources and our own technical limitations (we have found problems to build the new docker images for clang 9).

As well, the compiler survey doesn't include Android. Do you guys have any figure regarding Android?

(This is not from a survey, these are actual package installations - Sorry if I misunderstood and it was a false friend)

About new configurations (Android, iOS,...), it is something we are considering and it is very related to that resource-configurations balance. Some people think that it is more valuable to have configurations far away from each other than to have every single GCC version, and probably recipes themselves are more valuable if they manage properly different cross-building scenarios,... although they are not tested for some compiler versions or flags. Of course, here there are different opinions and necessities, and we cannot satisfy them all.

I don't think there is such a strong correlation between using the default compiler in the distro and Conan usage. Conan lets you manage the graph of dependencies, change requirements, versions, options,... from one project to another. It decouples the build from the underlying system libraries. ConanCenter probably provides libraries that are not available via the system package manager (if there is a system package manager available)...

About your clang-9, @chgans , are you (your company) using the packages directly from ConanCenter? We tend to think that companies build from sources their own packages and upload them to an internal Conan server, because they usually need to apply some additional patch. I'm glad to hear it is not your case and you can use packages directly (probably you are also contributing whenever you need a new version o some build to fix).

chgans commented 3 years ago

Our big project stuck on clang9 is not using conan, but we would like to move to conan. And yes we will have to use our own conan repo since we do cross-builds for a in-house embedded Linux OS. And we have started a new line of Android based products. We actually have to support a big range of products, some stuck with (very) old gcc. Nonetheless we do Linux, MacOSX and Windows build of this software, mainly for convenience of developers.

eigenwhat commented 3 years ago

Regarding additional platforms, I'm biased since I don't need to support Android in my work projects, but compiling to iOS is not much different than compiling for Mac OS, especially when it comes to CMake projects. If I were to pick one or the other, I'd look at iOS first since it's already better supported in other parts of the ecosystem. Mobile development is huge and there is plenty of demand for cross-platform C++ development, so I think it's a strong candidate. It also covers some important cross-compilation corner cases, like a library that has both build-time and run-time dependencies on the same package but with different settings and options. (e.g. proj requires the binary from sqlite in the build context and the library in the host context.)

I don't know how C3I currently handles building for MacOS, but if there are Mac agents it shouldn't be much trouble to include iOS as well. Since these are libraries and not apps you shouldn't need to deal with all the code signing hell. That's all one step further down the line. For the most part we're accustomed to building dependencies from source anyhow (e.g. CocoaPods) so for me at least my principle concern is that the recipes actually work moreso than having binaries available on ConanCenter. It may even be best not to publish iOS binaries on ConanCenter because there are some unique ABI concerns with bitcode on ARM that Conan currently doesn't handle all too well but which can be mitigated in an in-house CI environment.

PolyPik commented 2 years ago

Hi I would like to ask why are Windows MinGW GCC build configurations not supported by the Conan Center build service?

I know that Visual Studio may be an industry standard for Windows C and C++, but MinGW is also significant for providing the open source GCC toolchains to Windows developers.

Also I know that one can provide the --build option to conan install to get one's needed packages built using MinGW, but certain packages can have a lengthy build process and it will be the downstream project developer who will be bearing the burden of that wait time. So for the benefit of the developers who use the MinGW toolchains, I would like to see Windows MinGW GCC added as a configuration supported by the build service.

fuchse-ego commented 1 year ago

Hi I would like to ask why are Windows MinGW GCC build configurations not supported by the Conan Center build service?

I know that Visual Studio may be an industry standard for Windows C and C++, but MinGW is also significant for providing the open source GCC toolchains to Windows developers.

Also I know that one can provide the --build option to conan install to get one's needed packages built using MinGW, but certain packages can have a lengthy build process and it will be the downstream project developer who will be bearing the burden of that wait time. So for the benefit of the developers who use the MinGW toolchains, I would like to see Windows MinGW GCC added as a configuration supported by the build service.

Especially since it is not free anymore for companies (which want to distribute internal or closed source software using conan). Are there technical advantages using only msvc under windows?

bldrvnlw commented 4 months ago

@Croydon mentioned that the policy on macos os is the latest threes versions.

As of May 2024 this would imply that there should be apple-clang 13, 14 and 15 for an artifact such as zlib. In fact there is on apple-clang 13. Is there a reason for this?

Croydon commented 4 months ago

The policy changed.

Docs are currently not really up to date, but for macOS it already shows that only apple-clang 13 is supported https://github.com/conan-io/conan-center-index/blob/master/docs/supported_platforms_and_configurations.md

bldrvnlw commented 4 months ago

Was this change made because apple-clang 13 is ABI compatible with apple-clang 14 and 15? If so is it possible to alter the compiler version for the dependencies? (That's probably a question for the conan forum I realize)

bldrvnlw commented 4 months ago

To answer my own question James indicates a method to solve this