search

conan-io / meta-conan

Yocto layer for the Conan project
22 stars 28 forks source link

yocto/openembedded vs. conan - Workflow? #12

Open RobertBerger opened 2 years ago

RobertBerger commented 2 years ago

My understanding is that this layer just grabs some binaries from somewhere and adds them magically to some yocto project/oe build.

Right?

conan seems to do similar things to BitBake, so both seem to be kind of interfering with each other.

I would distinguish between a "developer workflow" and for this conan seems OK, and a "release workflow", where I don't quite see how conan would be adequate. (but I don't know conan well enough, hence I am asking for comments here).

What I am missing, I guess, is building via the yocto project/oe and adding those binaries/packages to some conan package, or a workflow where developers use conan and releases are built from source via BitBake. So maybe a conan to BitBake or BitBake to conan converter? That kind of sucks as well, since most likely different binaries will be generated.

As far as I see only a Yocto/OE generated toolchain can be used together with conan, but many cool/important things, which are available via a real BitBake build are lost like that:

And you could add additional meta-layers which provide:

What I am after is everything built from sources via Bitbake for a release and still having the possibility to use for application/application developers conan, I guess.

... and/or a reasonable workflow how to integrate conan with yocto/oe.

derived-coder commented 2 years ago

My understanding is that this layer just grabs some binaries from somewhere and adds them magically to some yocto project/oe build. Right?

No, when you want you can have source builds too. But it can provide you all binary artifacts for what ever platform you need it.

How to use conan best with yocto? I dont have any practical experience with mixing them. What is also possible is: Yocto builds a base image, with base software (X11, drivers, etc) and then you use conan to deploy the binaries to your different images (https://docs.conan.io/en/latest/reference/generators/deploy.html#deploy-generator)

From my experience regarding using yocto in real world project with 100+ custom components build via yocto recipes... Yocto does not scale at all. Building the image on a developer laptop is impossible...the problem is the monolithic approach of itself.

However conan can solve all the problems of yocto by introducing this new layer above the components, decentralizing the component packaging and creating binary artifacts directly in the repository where it happens.

Yocto has created a package manger for C/C++, because there wasn't one available in the past. This was a valid solution in the past. However this situation has now changed and modern decentralized tools like Conan or Vcpkg are introduced and will become the standard for C/C++ applications.

So it is better to keep up with this new trend and getting first class support for consuming conan or Vcpkg packages, there are 1000 of up to date recipes ready to be deployed on an image...

RobertBerger commented 2 years ago

My understanding is that this layer just grabs some binaries from somewhere and adds them magically to some yocto project/oe build. Right?

No, when you want you can have source builds too.

From what I see here[1] I don't get the impression that things are compiled. Would you please show me how to do that? This would help me a lot. I mean if I would be able to BitBake with conan from sources it would solve a lot of my problems.

[1] https://github.com/conan-io/meta-conan/blob/master/classes/conan.bbclass

But it can provide you all binary artifacts for whatever platform you need it.

This is exactly what I want to avoid in a "release build". Everything should be BitBaked. One "blob" which is flashed into a system and ideally can be reproduced/rebuilt bit identical.

How to use conan best with yocto? I dont have any practical experience with mixing them. What is also possible is: Yocto builds a base image, with base software (X11, drivers, etc) and then you use conan to deploy the binaries to your different images (https://docs.conan.io/en/latest/reference/generators/deploy.html#deploy-generator)

I understand your view as a "developer" if you want to do something like this. In my mind the workflow for a "developer" and a "release manager" is totally different.

Let me tell you my issues 1) If I look at the link you provided above I see openssl and zlib provided by conan. Both of them will be provided by Yocto the base software image. And not because I am a nasty person and want to make your life difficult, but because they are needed by the base image. 2) deploy seems to deploy those libraries as well to the target system. I am not sure if it's only the stripped libraries or also source code. That's not obvious to me from the doc. This means we have now different versions of openssl and zlib on the target system. One from conan and one from the base system. How is this supposed to work? Do you need to use tricks like LD_PRELOAD with conan to load specific versions of libraries and not the system libraries? 3) If you follow this paradigm and deploy the app in your embedded system via conan (not as a developer, but in the real product) what happens if there is a power failure while deploying? Is it possible to do incremental updates?

From my experience regarding using yocto in real world project with 100+ custom components build via yocto recipes... Yocto does not scale at all. Building the image on a developer laptop is impossible...the problem is the monolithic approach of itself.

And here we kind of disagree in various ways ;)

1) If a developer really wants to build on a laptop with Bitbake an image, or packages via BitBake of course they can. It's just a matter of a proper setup. Not sure what you mean by "it does not scale". 2) Having said that. If I would be a developer I would not use BitBake for my "developer" workflow. I would use an SDK or an eSDK which can be built with the Yocto Project and this provides all the header files and libraries I need to link against. 3) Now you say something interesting above - "100+ custom components". Maybe that's the use case? Big projects with custom C++ components. Of course, you could write Yocto recipes + Cmake for this as well. All the "non custom" stuff like zlib and openssl and so on should not be built by conan.

However conan can solve all the problems of yocto by introducing this new layer above the components, decentralizing the component packaging and creating binary artifacts directly in the repository where it happens.

The problem here is, that this is most likely non-reproducible. What I mean is, that if you build the same thing from sources 2 times you get different binaries. Or if you make a change to something you built 20 years ago you don't see only the change you made, but also different timestamps and so on...

Yocto has created a package manger for C/C++, because there wasn't one available in the past.

No. Yocto bundles whatever it builds into rpm, deb or ipk. Those are package managers and they do not care about the programming language. And rpm, deb, ipk existed long before Bitbake/Yocto/OE.

This was a valid solution in the past. However, this situation has now changed and modern decentralized tools like Conan or Vcpkg are introduced and will become the standard for C/C++ applications.

This is what somehow blows my mind. I somewhat see that C++ developers like those, but I don't quite see what is so special about them compared to the classic ones. Also, and that's my main problem, what you suggest is to mix package managers. One package manager (rpm, deb, ipk) for the base system and conan for the app. This sounds very dangerous since 1) the one does not know about the other. 2) duplicated work (recipes that exist both for Yocto and conan) 3) multiple versions of the same libraries 4) How to deploy at the customer/product without leaving a dead embedded system behind with conan?

So it is better to keep up with this new trend and getting first class support for consuming conan or Vcpkg packages, there are 1000 of up to date recipes ready to be deployed on an image...

I am trying to understand this and trying to find a way to do it. If you are willing to help we can do a few experiments. I guess I can cover the Yocto side of things if you help with conan.

Actually, I am after a solution/workflow which can make Yocto and conan people at least somehow happy.

And keep in mind in my opinion there should be a different workflow for a developer and a release. There are other things required. Of course, you might not want to BitBake every time you make a change as a developer. But you might want your CI system to make automated checks and start a BitBake once you check your code in. Ideally, those workflows don't have too much difference, but I am happy to explore. Also just upgrading the application at the embedded target is something people want. But we need a way that is reliable and does not brick the device.

derived-coder commented 2 years ago

You are right is it not enabled by default.

You need to add here: https://github.com/conan-io/meta-conan/blob/master/classes/conan.bbclass#L71 the build policy --build needs to be added. See also: https://docs.conan.io/en/latest/mastering/policies.html

Did you already checked this page: https://docs.conan.io/en/latest/integrations/cross_platform/yocto.html

regarding RPM see here: https://docs.conan.io/en/latest/integrations/deployment/system_package_manager.html

deploy seems to deploy those libraries as well to the target system. I am not sure if it's only the stripped libraries or also source code. That's not obvious to me from the doc. This means we have now different versions of openssl and zlib on the target system. One from conan and one from the base system. How is this supposed to work? Do you need to use tricks like LD_PRELOAD with conan to load specific versions of libraries and not the system libraries?

Conan helps here with activation scripts. But anyway, you can basically create one big deploy recipe with conan, which you can imagine as a Layer in yocto. Then ALL dependencies will be installed into one directory (zlib, openssl, Qt, etc...). Just to give an idea when you go the binary only approach.

Now you say something interesting above - "100+ custom components". Maybe that's the use case? Big projects with custom C++ components. Of course, you could write Yocto recipes + Cmake for this as well. All the "non custom" stuff like zlib and openssl and so on should not be built by conan.

No there are 100 yocto recipes created for creating all the components....this does not scale at all. Because the recipes are working only with yocto and not the local development workflow so creating and maintaining the recipes i violating DRY.

This is what somehow blows my mind. I somewhat see that C++ developers like those, but I don't quite see what is so special about them compared to the classic ones. Also, and that's my main problem, what you suggest is to mix package managers. One package manager (rpm, deb, ipk) for the base system and conan for the app. This sounds very dangerous since

Anyway, this is the future of C/C++ package management, like CMake is the defacto standard build system, conan or vcpkg will make the race for defacto standard packagemanager. To give you one reason why they are so successful: they are working cross-plattform and are only a thin layer above the build script, which can be integrated into the repository. So no matter what comes, with these package manager you can have nice reproducible builds. One of the key features!

Maybe this will give you some answers regarding: "Why conan?": https://www.youtube.com/watch?v=jKG6cETLN3M

Feel free to ping me in the offical conan channel for more questions.

bou6 commented 2 years ago

I am also kind of forced to use conan packages inside my yocto project (backward compatibility of some conan packages), Actually my opinion is that both of them are really great if used separately, Conan is really good on CI (It makes life easy) and Yocto is really stable on building Linux Images and handling the dependencies, The better in my opinion is to have only one Bandmaster in each step of the workflow, so Yocto could do the building step and then after the image (kernel, rootfs,...) is ready then a conan layer could be Implemented above to deploy the whole image and make the CI Task much easier . Now to my concrete questions:

1- I followed the steps described here , but that seems not to work. $ source oe-environment-setup- $ conan create . user/channel --profile armv8 If I go to the conan cache than CMAKE_CXX_COMPILER is always pointing to the host toolchain instead of the yocto toolchain. So is there something missing in the Documentation.

2- This is a remark: As a user of Yocto, I dislike the fact of configuring conan again, because I already configured Yocto for my target, so I would suggest kind of wrapper layer that takes the Yocto configuration and creates the profile needed, and then in the end user will no more call conan but only bitbake.

RobertBerger commented 2 years ago

I am also kind of forced to use conan packages inside my yocto project (backward compatibility of some conan packages), Actually my opinion is that both of them are really great if used separately, Conan is really good on CI (It makes life easy) and Yocto is really stable on building Linux Images and handling the dependencies, The better in my opinion is to have only one Bandmaster in each step of the workflow, so Yocto could do the building step and then after the image (kernel, rootfs,...) is ready then a conan layer could be Implemented above to deploy the whole image and make the CI Task much easier .

I believe there should be a separation between the developer workflow and the ci/release workflow, since when you start using conan for the app and BitBake for the release you produce a non-reproducible release and loose many other goodies available by the yocto project when you BitBake and don't just use the SDK and need to deal with versioning, license compliance, off-line builds and so on on 2 different places, which is not manageable.

Developer: use conan if there is some advantage. CI/Release: everything BitBaked

Now to my concrete questions:

1- I followed the steps described here , but that seems not to work.

Yes they don't work. Besides I think it's a bad example, since mosquitto is already available (like many other recipes) by meta-openembedded: http://cgit.openembedded.org/meta-openembedded/tree/meta-networking/recipes-connectivity/mosquitto/mosquitto_2.0.11.bb?h=master

$ source oe-environment-setup- $ conan create . user/channel --profile armv8 If I go to the conan cache than CMAKE_CXX_COMPILER is always pointing to the host toolchain instead of the yocto toolchain. So is there something missing in the Documentation.

In my case it's enough to source the environment script to cross compile without a special profile, at least for the hello-world example.

With the help of people on the conan chat I managed to do something like this:

https://gitlab.com/exempli-gratia/conan-yocto-hellocppcmake/-/blob/master/conanfile.py

https://gitlab.com/exempli-gratia/conan-yocto-hellocppcmake/-/blob/master/build.sh

2- This is a remark: As a user of Yocto, I dislike the fact of configuring conan again, because I already configured Yocto for my target, so I would suggest kind of wrapper layer that takes the Yocto configuration and creates the profile needed, and then in the end user will no more call conan but only bitbake.

I really would like to find use cases and proper integration between conan and yocto, but this seems to be tricky. Still trying to find out if the integration issues are technical or "philosophical".