[bug] conan install deploy cannot be used to deploy a "development" recipe

[hpe-ykoehler]

Describe the bug

My goal is to create an application using conan which depends on many other libs. I then would like to extract all the runtime files and copied them into a Dockerfile

myapp mylib

when I build myapp, I want to generate a docker image contains the runtime of both myapp and mylib.

For that, it appears that conan install --deploy is the best tool.

But, conan install --deploy only work for dependencies of a recipe (mylib), and not the recipe you execute it itself (myapp).

For example the graph bin directory for dependencies contains an absolute path, while the root level of the graph binder contains only a folder "bin", without saying it is relative to what other folder.

This results in not being able to write a deployer script that extract runtime not only for dependencies for for the top-level package itself unless the package is store in the local repo using "conan create", but in my case, I need to test the Dockerfile, before submitting it, so I need to be able to deploy the recipe after a conan build.

Need some advice here.

I try the following:

   # deploy runtime of all direct dependencies
    $CONAN install . --deployer=strip_and_deploy --deployer-folder=myfolder/
   # deploy runtime of this package itself
    $CONAN install --requires=$PROJ_NAME/`cat version`  --deployer=strip_and_deploy --deployer-folder=myfolder/

which works only if the package is in the local repo, but doesn't work for package in development.

I then thought to do a sub-package kind of like test_package, and use the self.tested_reference_str as a way to pull in the current under dev build package, but realized that the tested_reference_str require conan to participate in that model.

How to reproduce it

No response

[memsharded]

Hi @hpe-ykoehler

Thanks for your report.

But, conan install --deploy only work for dependencies of a recipe (mylib), and not the recipe you execute it itself (myapp).

Deployers work over packages, their responsibility is to copy things from Conan packages from the Conan cache and deploy them in some user folder to extract them "out of conan packages" for final app deploy.

If app is a package and you do a conan install --requires=app/1.0 --deployer, then, it will work and deploy also app/1.0.

The confusion comes when app is not really a package, but it is just a local "consumer" recipe. When doing:

conan install . --deployer...

and the . is the path of the app conanfile.py, then the deployer can not deploy the app binaries, because they have not been built at all yet. The conan install only install the dependencies (and deploy those dependencies, but app is not a dependency package in this case).

This results in not being able to write a deployer script that extract runtime not only for dependencies for for the top-level package itself unless the package is store in the local repo using "conan create", but in my case, I need to test the Dockerfile, before submitting it, so I need to be able to deploy the recipe after a conan build.

Yes, that is the thing, the package needs to be conan create first if you want to deploy it. The thing is that this is not a "top-level" package, it is not a package at all, the binaries haven't been built yet at all, so they cannot be deployed. A deployer might have access to the local consumer path (https://docs.conan.io/2/reference/extensions/deployers.html, the graph object, the graph.root object might have a path that is pointing to the local consumer recipe), but that is still irrelevant if there are no binaries built yet to deploy.

How are you building? Are you using conan build command? It is still not fully clear to me why the conan create + conan install --requires=app/version is not a good solution in this case, could you please clarify?

Maybe adding the current recipe as editable might work, if it correctly defined the layout() to be editable, something like:

conan editable add .
conan install --requires=app/1.0 --deployer=mydeployer --build=editable

[hpe-ykoehler]

I was doing something like

conan build . conan install . --deploy

Since I did build the package the binaries are available.

I tried to use editable but encountered many issues where conan-center-index package would not even build using that method.

I am now unclear as to the purpose of conan build, which now seems very limited (kind of useless) as anything more advanced request a call to conan create. And the conan editable is not a viable path either.

The issue I have with conan create is that the whole build artifact is in some ~/.conan2/p folder which makes things a lot harder to debug for developer and unfamiliar, while conan build with the build/ folder is simple, but apparently other than building you can't do much about it which is surprising....

[memsharded]

conan build . does not build a package. It does a local build, which is different. It mostly misses the last cmake.install() or package() which is what actually creates a package. Packages only live in the Conan cache, and the binaries that conan build builds are in the local user folder.

conan build . actually does internally a conan install as a first step (to install the necessary dependencies), which calls generate() to have the necessary files, and then finally calls the build() method to launch a local build, basically it is a short cut for conan install + cmake --preset .... It is not useless, it is very convenient in lots of situations, and many Conan users are happy about it. But it is important to understand that it is not creating a package, but doing a user "consumer" build, which is different.

I tried to use editable but encountered many issues where conan-center-index package would not even build using that method.

yes, most conan-center-index package recipes do not implement the necessary layout() to support editables. This is expected. I thought that you were trying to deploy as consumer your own application, not trying to deploy applications from conan-center-index.

The issue I have with conan create is that the whole build artifact is in some ~/.conan2/p folder which makes things a lot harder to debug for developer and unfamiliar, while conan build with the build/ folder is simple, but apparently other than building you can't do much about it which is surprising....

This is fine, of course the local flow with conan build is intended for developers to have an easier access to the build. This is common in the "local flow" intended for developers that are developing the C++ source code, or for testing part of the recipes.

But deployers aren't typically part of this flow, but the opposite, extracting binaries from existing packages in the cache. Note that for example deploying the right set of only public headers for a library if it is the "consumer" recipe is almost impossible, because the cmake.install() inside the package() method has not been called, then any attempt to copy headers will likely copy both public and private headers from the source tree.

Still, what I have said above is possible: the deployers will still have access to the current recipe folder, and even to its build_folder, something like graph.root.conanfile.build_folder, so you might "deploy" from there what you consider you need to deploy. But take into account that it will not be a clean "bin" folder with the binaries ready, but you can find different subfolders (like x64/Release in VS) in different system, because it is a "build" layout, not the final "package" layout, and your deploy logic will have to deal with that variability.