[feature] Support for virtual environments

[jsallay]

What is your suggestion?

I work on a open source project (https://github.com/gnuradio/gnuradio) that uses a plugin model and I am trying to convince them to use conan for their package management. People have written hundreds on these plugins which end up compiling to dynamically linked libraries.

There is a workflow that I would like to enable where the user finds a package, "installs" it, and immediately is able to use it along with any other "installed" packages. This is presently difficult in conan because each install is independent. If I do a conan install --requires package1 and later do conan install --requires package2 it produces two sets of conan(run/build).sh scripts. The scripts will have the same names, so I can accidentally overwrite the previous script if I call install from the same folder twice.

I could have the user do a single install command - conan install --requires package1 --requires package2, but as the number of plugins installed grows, this becomes more and more difficult.

There is also the difficulty of managing the environment. The presently defined deployers, install packages to unique folders per package. So even though the vast majority of these packages only need to set a few common variables like LD_LIBRARY_PATH, we need to source a new environment after every install.

It would be really nice to have something that works more like a python venv, where we essentially create an alternate root directory and install everything in there. I could set LD_LIBRARY_PATH=<alt_root>/usr/lib and any package that was installed would be immediately available. I wouldn't have to worry about sourcing a new set of environment variables (unless the package had some non-standard ones, but you could write a check for that).

Another bonus to this approach is that it would make final deployment of an application really easy. In a multistage docker build, we could install the conan packages to this virtual environment in the first stage (also installing the system_requirements to the alt root. A second stage could just copy --from builder <alt_root> /. The final docker image would not need to have conan or anything else installed, it would just contain the final application and its dependencies.

Is there an interest in this feature? I have written a v2.0 conan venv command and a custom deployer that might be useful as a start.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @jsallay

Thanks very much for your suggestion.

There is also the difficulty of managing the environment. The presently defined deployers, install packages to unique folders per package. So even though the vast majority of these packages only need to set a few common variables like LD_LIBRARY_PATH, we need to source a new environment after every install.

Yes, the current built-in deployers are intended for other use case, but I can see how other kind of deployer puts everything in the same target folder (and the user would be responsible for the potential collisions, maybe there could be checks for files being overwritten by more than one package).

Is there an interest in this feature? I have written a v2.0 conan venv command and a custom deployer that might be useful as a start.

Yes, it would be useful to see such command, while I see the deployer, I don't fully get the functionality of the conan venv command (maybe the name is not the best, as it would overlap a bit with the VirtualBuildEnv and other similar tools in Conan, and I think this is a different use cases).

[jsallay]

Let me try to explain the venv command a little better. I picked the name based on python virtual environments but am open to a different name. The goal is for it to allow conan to work more like pip or apt. When I do an apt install pkgx, as a user I don't have to know what packages are already installed on my system. apt keeps track of that for me. At install time, it checks the package that I want to install against what is already installed to ensure that there aren't any conflicts.

If I just use a custom deployer, then conan can't ensure that the newly installed package doesn't conflict with any of the previously installed packages. There is also difficulty with setting up the environment. If a previously installed package sets an environment variable X and the newly installed package sets Y, then I need a new environment that sets both X and Y.

My thought is to maintain a list of all of the installed packages. When a user wants to install a new one, I compute the graph of the new package and all of the previous ones. This way I can ensure that there are no conflicts.

I hope that clarifies the use case.

[memsharded]

Thanks for the feedback. So if I understand it correctly, the conan venv (or different name) will be an automation over the creation of a simple conanfile.txt that is stored inside the venv, these would be the implementation of the command:

• conan venv <path/name-to-venv> --requires=mydep/myversion
• That locates the conanfile_env.txt inside <path>, creates it empty if it doesn't exist
• Appends to the conanfile mydep/myversion if it doesn't exist there. Warn/Fail/Update if another version of the same mydep exist in the venv
• Run internally the equivalent to conan install conanfile_env.txt -G VirtualBuildEnv. This will manage to raise conflicts if something happens.
• If conflicts, revert the conanfile file changes
• If ok, update and overwrite the conanbuildenv scripts

User would still need to activate/deactivate the environment, as that cannot be done from a Conan execution, as it is a subprocess.

[jsallay]

This is largely what I had in mind. I have a few additional notes. The command ideally includes some management functions such as being able to remove an installed package and list the full set of packages installed in the environment - (not just the ones listed in the conanfile.txt).

As an added bonus, it would be really cool to be able to do a chroot install to the environment folder, so that all of the system packages were also self contained.

[memsharded]

This is largely what I had in mind. I have a few additional notes. The command ideally includes some management functions such as being able to remove an installed package and list the full set of packages installed in the environment - (not just the ones listed in the conanfile.txt).

Ok, sounds good and possible. The main problem is that this might have different user expectations, use cases to learn, etc., I think this would be better to be developed and tested not as built-in, but as an independent Conan 2.0 command. We might have an "incubator" project after the 2.0 release that contains different commands or groups of commands, or if you can share it, we could link it somewhere, and learn also from the community.

[jsallay]

I agree with what you are saying. I think there is some work to be done on my part. I was centally managing the virtual environments in the .conan2 folder. I like your idea better of managing the information in the installation folder instead. I am going to have to put some thought into how that changes some of what I am doing. After I change the code around I'll try and see if I can get it released publicly.

[AgentK9]

Any updates here? This looks like a great feature!

[josephbirkner]

Yeah, this would be awesome to have 🚀

[jsallay]

Sorry it took me so long to respond. I have something that kind of works, but it very specific to my use case. It wouldn't be something that someone can run with. I do have some lessons learned that I'll put here and these could be turned into individual issues that take us much of the way there.

1. The activate/deactivate scripts are difficult to use correctly when switching between environments. If I source the activate script from one environment and then activate another environment, my env now contains both. If I call deactivate multiple times or from the wrong env, bad things can happen.

I would propose that we do what python venvs do. They create a deactivate function. If it exists, it is automatically called whenever an activate script is called. We can see an example for bash here: https://github.com/python/cpython/blob/3.12/Lib/venv/scripts/common/activate#L4 . I don't know if we would want to replace the functionality of VirtualRun/BuildEnv with this or create a different generator. This would make it much easier to manage multiple virtual environments at the same time.

2. We need a deploy script that cuts off all of the package information. (No arch, debug/release, version, etc in the path.) For example all of the libraries would be co-mingled in the lib folder. This way we can just set the LD_LIBRARY_PATH (in linux) to point to this one folder.

I tried using the conan file structure with the full_deploy deployer, but users found it very confusing. "Why do I need to have an LD_LIBRARY_PATH that points to 30 different places." There are also use cases where we want to install into /usr/local/lib and just have it work without having to set any extra variables. When I wrote this deployer, complaints and questions went way down.

As part of this we need a custom generator that can "reduce" the paths in the output variables to just unique strings. If I just write the above deployer, the activate script will contain an LD_LIBRARY_PATH with the same path repeated 30 times. I had to add an extra step to keep only the first unique occurrence of each path.

3. We need to save off a lock file and the conanmanifest.txt inside the virtual environment. That way we can query the packages installed and see if files have been modified.

This leads to a big open question. What do we do if the user tries to install over an existing file. It is possible that two packages will contain the same file or we will want to install two versions of the same package (shared=True and False). I don't presently have a great solution for this.

What do we do if the user tries to install a version that requires downgrades of installed packages?