Open hlewin opened 3 months ago
Hi @hlewin
Thanks for your question.
This would be mostly intended behavior.
The VirtualBuildEnv
and tool-requires
are a "build-time" thing, while the deploy
is a pure runtime thing, independent of the build process.
I think that in Conan 2 you might have some new functionality, for example the deployers
that you can use externally to the recipes, and there is a new tools.graph:skip_binaries
conf that forces to not skip binaries of dependencies, even when they are not necessary, but I am not sure if this would work for your use case.
For deployment and runtime things, we have seen users doing a 2 step approach, first conan install --tool-requires=..
(or a conanfile with tool-requires) then activating the generated environment, then calling the deploy, that will use those tool-requires because the environment has been activated in the shell.
Hi @hlewin
Thanks for your question.
This would be mostly intended behavior. The
VirtualBuildEnv
andtool-requires
are a "build-time" thing, while thedeploy
is a pure runtime thing, independent of the build process.
Yes and yes - which in my understanding means explicitly using VirtualBuildEnv(self)
should reflect the build environment. It makes no sense that dependencies that are scoped with self.require(...., build=true)
do not propagate definitions of their buildenv_info
to VirtualBuildEnv
. Or to put it another way: The current state of affairs is that the contents of VirtualBuildEnv
are virtually unforseeable: They are defined by whatever accidentally happened to be being built in the invocation of conan.
I think that in Conan 2 you might have some new functionality, for example the
deployers
that you can use externally to the recipes, and there is a newtools.graph:skip_binaries
conf that forces to not skip binaries of dependencies, even when they are not necessary, but I am not sure if this would work for your use case.
This is something I'll have a look at. Deciding the course of action for a transition to conan 2 is one reason of this question.
For deployment and runtime things, we have seen users doing a 2 step approach, first
conan install --tool-requires=..
(or a conanfile with tool-requires) then activating the generated environment, then calling the deploy, that will use those tool-requires because the environment has been activated in the shell.
This would be no real option for the use case at hand as the question which packages and definitions are needed for a given deployment are nontrivial, ie one would have to mirror the logic of picking options and such that is done in conanfile.py
s in shell scripts. - this is not feasible. The possible workaround is a "forwarder" package with no content of it's own that self.require()
s the package to be actually deployed and make it being built on the command line, as this will instantiate the full build-environment. But this is not about workarounds. It is about semantics. The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info
to VirtualBuildEnv
anywhere.
The possible workaround is a "forwarder" package with no content
Sounds it could be possible.
It would be good to understand a bit more the full picture:
deploy()
method?tool_requires
are needed to enable the deploy functionality? What are they doing?tool_requires
defined in recipes conanfiles, or injected via [tool_requires]
in profiles?The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info to VirtualBuildEnv anywhere.
the semantics are that tool_requires
can propagate their environment to the packages that directly tool_require
it, exclusively at build time (that means build()
and package()
), but not outside of the build time. It might be possible to add some clarifications in the docs about this, but at least this has been the spirit and intention since Conan 1 and also still in Conan 2.
The possible workaround is a "forwarder" package with no content
Sounds it could be possible.
It would be good to understand a bit more the full picture:
- Are all packages deployed?
- Do all packages contain a
deploy()
method?- How many packages are actually deployed this way?
- How many different
tool_requires
are needed to enable the deploy functionality? What are they doing?- Are the
tool_requires
defined in recipes conanfiles, or injected via[tool_requires]
in profiles?- Is it enough if they inject themselves in the PATH for finding the executables, or anything else?
We deploy a package and it's (runtime) dependencies including transitive ones. Although most of those packages implement a (actually "the same") deploy method only the deploy method of the package directly deployed matters here.
In this deploy method we need (at least) the environment variables from the [tool_requires]
given in the host-profile as well as the variables given in the [buildenv]
section. (Note that this is not really sound: It might be that a tool-requirement in that profile would need to be build).
Typically the tool-requirements in the profile set up their PATHs; but also setting build-system style environment variables like CHRPATH=armv7-debian-linux-gnu-chrpath
or INSTALL_NAME_TOOL=llvm-install-name-tool
is common.
The deploy() method of the package directly deployed makes use of such definitions to process all deployed packages (given this is functionality that should be moved to a separate generator/deployer).
We (in particular) are not so much concerned about the tool-requirements defined in the packages themselves - for those use-cases the profiles are much more important.
The documentation makes no indication that a tool-requirement will not propagate it's buildenv_info to VirtualBuildEnv anywhere.
the semantics are that
tool_requires
can propagate their environment to the packages that directlytool_require
it, exclusively at build time (that meansbuild()
andpackage()
), but not outside of the build time. It might be possible to add some clarifications in the docs about this, but at least this has been the spirit and intention since Conan 1 and also still in Conan 2.
Yes, clarifications would indeed be helpful in this case. Currently there are no mentions of VirtualBuildEnv
being restricted to certain methods. Another (and in our use-case more helpful) possibility would be to expose a method that actually instantiates the build-environment if this hasn't already be done so.
Alternatively a way to instantiate a given environment defined in one or more given profiles would suffice for our (concrete) use-case but would still leave VirtualBuildEnv(self)
somewhat dubious without clarifications.
That makes sense thanks for the clarification.
I am going to check a little bit and experiment to see if there is some possible solution in Conan 2.
Just to make sure, I understand that this doesn't work either in Conan 1, and you are mostly interested in knowing if Conan 2 has some new functionality that could help with this, is this correct? Or is it something that you managed to get it working somehow in Conan 1, and now Conan 2 is more strict in that regard and not working?
Just to make sure, I understand that this doesn't work either in Conan 1, and you are mostly interested in knowing if Conan 2 has some new functionality that could help with this, is this correct? Or is it something that you managed to get it working somehow in Conan 1, and now Conan 2 is more strict in that regard and not working?
No, I did not really get this working in conan 1. We are - for now - forcing rebuilds of packages as a workaround. I am making our stuff conan 2 compatible piece by piece and was fiddling around with VirtualBuildEnv
and self.buildenv_info
as a colleague happened to run into this problem (of missing settings). The whole thing was unnoticed for quite some time as - by coincidence - the package to be deployed happened to be freshly built in virtually all cases for other reasons.
So this caught us by surprise and the question popped up if this was still the case with conan 2: we will have to finish up the porting process one way or the other and this could have been a good reason to hurry up a little.
If there is way to get rid of such workarounds in conan 2 this would be nice. Most of the porting to version 2 did not make any structural changes to established processes. I am just evaluating options...
I have made this test pass in Conan 2.
def test_deploy_method_tool_requires():
c = TestClient()
tool = textwrap.dedent(r"""
import os
from conan import ConanFile
from conan.tools.files import save, chdir
class Pkg(ConanFile):
name = "tool"
version = "0.1"
type = "application"
def package(self):
with chdir(self, self.package_folder):
echo = "@echo off\necho MYTOOL RUNNING!!"
save(self, "mytool.bat", echo)
save(self, "mytool.sh", echo)
os.chmod("mytool.sh", 0o777)
def package_info(self):
self.buildenv_info.prepend_path("PATH", self.package_folder)
""")
conanfile = textwrap.dedent("""
from conan import ConanFile
from conan.tools.env import VirtualBuildEnv
class Pkg(ConanFile):
name = "pkg"
version = "0.1"
tool_requires = "tool/0.1"
settings = "os"
def deploy(self):
venv = VirtualBuildEnv(self)
with venv.vars().apply():
ext = "bat" if self.settings.os == "Windows" else "sh"
self.run(f"mytool.{ext}")
""")
c.save({"tool/conanfile.py": tool,
"pkg/conanfile.py": conanfile})
c.run("create tool")
c.run("create pkg")
c.run("install --requires=pkg/0.1 -c:b tools.graph:skip_binaries=False --deployer-package=*")
assert "MYTOOL RUNNING!!" in c.out
the keys are:
VirtualBuildEnv
with with venv.vars().apply():
in the deploy()
method-c:b tools.graph:skip_binaries=False
. Note --conf:build
to not skip tool-requiresI think a similar approach would be feasible using the Conan 2 new external --deployer
, as they work mostly the same as the in-recipe deploy()
method.
Also the approach of using a dedicated recipe that uses standard generate()
+ build()
method to do what you would do in the deploy()
method of the direct recipe could work well too.
This does indeed look exactly like the thing I need - thank you for your efforts!
I did not think about generating separate package that does the "bundling" in this scenario - although we already have a similar package in place to add implicit dependencies like compiler-specific runtime libs ( libc++ and such ) from the Conan cache to a deployment. That could indeed be extended easily to do a "dumb" deployment without gathering any implicit dependencies. All in all I think we should do the grunt-work and go through the rest of packages that need porting to Conan v2 and finish up the transition.
That having said I would still welcome a remark in the documentation that VirtualBuildEnv
might not contain settings from tool-requirements that were skipped because they weren't needed for the (conan-)command to be executed. It is always better to have such things nailed down.
Again - thank you for this discussion and your efforts!
That having said I would still welcome a remark in the documentation that VirtualBuildEnv might not contain settings from tool-requirements that were skipped because they weren't needed for the (conan-)command to be executed. It is always better to have such things nailed down.
Sounds good, I'll move this ticket to the docs
repo.
Again - thank you for this discussion and your efforts!
Thanks to you for your feedback!
What is your question?
Hello! At least when using conan 1
VirtualBuildEnv(self)
does not seem to contain definitions from tool(or build) dependencies if the package was not built in the conan invocation. The use case here is that we are doing some stuff in thedeploy()
method like setting up RPATHs, signing binaries etc. that would need dependencies given in the build-profile as tool requirements. But when the package to be deployed was not built in the same conan invocation the tool requirements do not show up inself.dependencies
and hence they do not propagate any settings inVirtualBuildEnv
. The only things we get there are variables from the[buildenv]
section of the profile. Is this intended behaviour and is it still present in conan2? I get it is an optimization to not instantiate build-context dependencies if not necessary but the behviour right now seems strange... - like a bug to be honest.Have you read the CONTRIBUTING guide?