search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.14k stars 970 forks source link

[question] conan build and conan export-pkg #15939

Closed maitrey closed 5 months ago

maitrey commented 5 months ago

What is your question?

Hi! I have been using conan build and conan export-pkg both for local developemnet workflow and in CI/CD.

conan build . -of kdp --deployer=full_deploy --update --user autosar --channel featuresupport -pr:b=default -pr:h=kdp
conan export-pkg . -of kdp --user autosar --channel featuresupport -pr:b=default -pr:h=kdp

I donot run these commands concurrently rather sequentially. I have 5 such variants to build. In some cases, I noticed that only one variant gets uploaded although all 5 variants are build and exported to the cache. All the 5 exported packages have different package revisions. I am clueless how to debug this further. Could you please leave me some hints what I could be doing wrong? Additional Info: When I run conan list I always see only the last variant that was built not all the 5 variants although they do exist in the cache.

Thanks! Br, Maitrey

Have you read the CONTRIBUTING guide?

memsharded commented 5 months ago

Hi @maitrey

Thanks for your question.

We would need the output of at least 2-3 of the builds an export-pkg of those variants (also the Conan version you are using). I suspect that the package_ids might be the same, if you can share the conanfile that would help a lot too.

maitrey commented 5 months ago

No the package ids are not the same.. But what I noticed is that the recipe revision is different in each of the variants. Also I generate code in the build method from simulink models before compilation. The code generation is done in the source folder , does this impact how recipe revision is calculated. Its the same recipe used for compiling all the different variants. Interesting for me , would be to know if generating code in the source folder could lead to different recipe revisions. I am later packaging the generated sources, header and the generated libraries.

memsharded commented 5 months ago

No the package ids are not the same.. But what I noticed is that the recipe revision is different in each of the variants.

That could be because the exported source files are different. This shouldn't happen, the exported files must be the same for all different variants/binaries. Please make sure that you only exports_sources files common to all different configurations, like the common source code. Generated files that are specific for a configuration are not "sources" to be exported, those are "build" artifacts, and should be directly packaged as the binary, not as input sources.

maitrey commented 5 months ago

Does it matter if for example source/.gitkeep and include/.gitkeep are checked in to the repository. Code generation happens at source/myfile.c and include/myfile.h . Both source and include are not listed under export_sources.

memsharded commented 5 months ago

Does it matter if for example source/.gitkeep and include/.gitkeep are checked in to the repository.

It depends on what you exports_sources. If you do exports_sources = "source/*" and the .gitkeep is exported too and it is variable, you will get different recipe revisions, and that is incorrect. So please make sure that the exports sources don't include any variable files.

The exported files are easy to check, doing a conan cache path pkg/version#recipe_revision you will get the path to the recipe and the conanmanifest.txt there will get you the exported sources. You can diff them and you will see which files are causing the difference that causes different recipe revisions.

maitrey commented 5 months ago

I have two additional questions:

memsharded commented 5 months ago

I have used launch.json in visual studio code with conan commands to debug recipes. Is this the correct way to debug recipes?

You mean the python code of the recipe? Yes, that can be debugged with any technique that you like, even adding self.output.info() messages to the recipe can help. It is python, so any strategy to debug python code would be valid.

I have noticed one more behaviour: For packages that have generated content : conan build + conan export-pkg (after fixing export_sources), if I use the same combination for triggering tests it overwrites the contents of the packages leading to no libraries getting copied. lib is empty although it was generating the libraries before triggering the tests Commands that I use is:

It should be necessary to have something that can be reproduced. I guess it could be something related to the layout, the conan build is nothing but a call to the build system, it doesn't clean the local build folders. If doing builds without cleaning the folders, or without a better modeling of the binary variants (like build_folder_vars definition), then libraries can accumulate in the local build folder, and then export-pkg will just pick the accumulated libraries.

Side note about --channel featuresupport. Please recall that the usage of channels to represent binary variants is no longer recommended. Feature/stage of development is better approached by using different server repos and lockfiles in majority of cases.

maitrey commented 5 months ago

Hi @memsharded , Thanks for your guidance. I have it in my list to remove channel. But what I find contradicting is most of the commands do support channel: https://docs.conan.io/2/search.html?q=channel&check_keywords=yes&area=default . Is it for providing backward compatibility? Why was channel discontinued? Also, I have a question when I run conan build with full_deploy lets say more than one time, it does the dependency graph computation even though I have not changed anything in the dependencies. Could this be somehow made faster or avoided? Also when I run the tests , I use the same folders this also does the same computation multiple times.

memsharded commented 5 months ago

Thanks for your guidance. I have it in my list to remove channel. But what I find contradicting is most of the commands do support channel: https://docs.conan.io/2/search.html?q=channel&check_keywords=yes&area=default . Is it for providing backward compatibility? Why was channel discontinued?

It is ok to use the channel as a fixed, immutable part of the name. And yes, very intended for backward compatibility, you will see in many command inline help, but you will probably see almost any real example or explanation in which --channel=xxxx is used.

What is completely discouraged is to use channel as a variable or mutable "state" of the package, and try to use it to rename a package from pkg/0.1@team/testing to pkg/0.1@team/stable. This rename shouldn't be done, and packages shouldn't be copied, promoted or moved or anything like that in any way that tries to change the user and channel identifiers. The recommendation is that once a user/channel is given to a package, it should be that way forever.

Also, I have a question when I run conan build with full_deploy lets say more than one time, it does the dependency graph computation even though I have not changed anything in the dependencies. Could this be somehow made faster or avoided? Also when I run the tests , I use the same folders this also does the same computation multiple times.

The conan build does a conan install + build() everytime, because knowing the dependency graph is needed in many cases for the build() method to execute.

The recommendation is not try to make it faster, but not to use it at all. It is just a convenience thing, but in most cases the recommended flow is conan install + cmake ... (or native build system calls, including ctest or whatever...).

maitrey commented 5 months ago

We are using the channel as the same name as the feature branch name. And we donot rename packages. We have channel as feature branch or pull request for distinguishing between branch builds and tag builds. For the tag builds we use same version as the git tag. channel in this case is always release. The branch builds are in a seperate artifactory repo and the tag builds as well. Do you see any problems with the way we are using channel?

memsharded commented 5 months ago

We are using the channel as the same name as the feature branch name. And we donot rename packages. We have channel as feature branch or pull request for distinguishing between branch builds and tag builds. For the tag builds we use same version as the git tag. channel in this case is always release. The branch builds are in a seperate artifactory repo and the tag builds as well. Do you see any problems with the way we are using channel?

If you are not renaming then there shouldn't be any big inconsistency or core issue.

The major downside might be that this approach might require way more CI power, because way more packages need to be built that wouldn't be necessary to build otherwise. If you have 100 packages, and you do 1 change in one of them for a feature/tag/branch "myx", then will you be creating 100 new packages all of them with pkg/version@user/myx? Are you doing a self.requires(f"dep/version@user/{self.channel}") in your recipes to propagate and require the packages in the same channel?

maitrey commented 5 months ago

Normally, for the dependencies lets say i have 100 packages that depend on package A. If package A needs to be changed , then I follow the usual git workflow and once a tag from A is released I use it in X, Y, Z + 97 other packages. Would not this workflow be the same with/out channel? I do specify dependencies in the 100 packages as: self.requires(f"dep/version@user/{self.channel}")

memsharded commented 5 months ago

Normally, for the dependencies lets say i have 100 packages that depend on package A. If package A needs to be changed , then I follow the usual git workflow and once a tag from A is released I use it in X, Y, Z + 97 other packages. Would not this workflow be the same with/out channel? I do specify dependencies in the 100 packages as:

The issue is that when you have app -> pkg1 -> pkg2 -> pkg3 -> pkg4 ....... -> pkg100

Then pkg10 changes in branch "mypkg10fix". You don't need to create new packages for pkg11-pkg100 by forcing a new channel with that branch name into them. Those are totally unaffected, and it should only be necessary to process app and pkg1-pkg9. And in many cases it might not even necessary to rebuild them, depending on the package types (static libraries) and changes done in pkg10 (for example if it is only an implementation .cpp bugfix and it is a static library too), it might be only necessary to rebuild/relink app.

But if you are forcing the channel thing always to the 100 packages, you are likely wasting a ton of CI that would be unnecessary.

maitrey commented 5 months ago

Ok Understood! Thanks for explaining!All questions have been answered! Ticket can be closed!

memsharded commented 5 months ago

So, what would be the status of this question? Can the ticket be closed, or is there still something pending? (for some of the above we would need a reproducible example, otherwise it is almost impossible to know) Thanks!

maitrey commented 5 months ago

No with your hints and guidance, I could fix the issue!

memsharded commented 5 months ago

Ok, closing the ticket now, thanks!