How to keep tool versions in subshells and/or different directories?

[francois]

Hi!

Just installed asdf and find it very useful.

I'm trying to add a PostGIS plugin. PostGIS has a dependency on PostgreSQL. You can see my current attempts on https://github.com/francois/asdf-postgis.

The install script creates a subshell, then cds into the directory where the tar was unpacked. Because the PostGIS tar does not have a .tool-versions file, when any PostgreSQL shim is called, asdf complains with No version set for postgres. This is a problem as PostGIS runs pg_config as part of the configure process.

How should I go about fixing this issue? I thought about copying .tool-versions from the CWD before running the installation. Are there any other recommended methods of working with dependencies?

[Stratus3D]

Hi @francois, this is very timely question. I ran into the same issue with https://github.com/Stratus3D/asdf-rebar earlier this week with it's dependency on Erlang.

To answer your question, no, there isn't a recommended way dealing with asdf-managed dependencies. The issue with no version being set in the temp directory can be fixed by setting the version in the users home directory (in $HOME/.tool-versions). Of course this doesn't work if the user hasn't chosen to set a global version for a plugin, so we probably shouldn't rely on this.

When I was working on asdf-rebar there were other issues I thought of:

• What would happen if the behavior of the resulting binaries changed based on the version of the dependency (in our cases Postgres and Erlang)? A user might need to make sure Rebar was compiled with a specific version of Erlang to get a certain feature.
• Should we allow a version of the tool (Rebar) compiled with one version of Erlang to be used with another version of Erlang? What if it was compiled with a newer version of Erlang but then used in a project that required an old version of Erlang. There is the potential it wouldn't work.
• What if a user wants to make sure a projects Rebar version was compiled with that projects Erlang version? Right now that would mean installing the right version of Erlang, uninstalling the Rebar version if it's installed, then reinstalling it from in the project directory.

I'm not really sure how to best to solve these problems. With asdf we usually try to keep things as simple as possible, but I do think these issues need to be addressed. Early on this wasn't an issue because there weren't that many plugins, and none of them depended on each other. Now that's changing.

@asdf-vm/maintainers any ideas on how to solve this?

[francois]

The more I thought about the problem, the more I thought asdf will eventually become a full-fledged package manager... If A depends on B, then B must be installed before A. It's a Directed Graph, and there may be transitive dependencies as well.

If asdf is to be responsible for managing dependencies, then it will have to look for .tool-versions files in dependencies and make the appropriate decision at compile time.

Alternatively, a simpler solution, but which requires more human intervention, is to take .tool-versions and parse it line by line and resolve dependencies at that time. If a human determines that a dependency must be installed before another one, then the line for the dependency must be before the other one. In other words:

postgres 9.3.15
postgis 2.3.2

would ensure a valid install of PostGIS, while:

postgis 2.3.2
postgres 9.3.15

would not.

Regarding the compilation stage itself, asdf would have to implement something like Bundler's BUNDLE_GEMFILE: when Bundler boots, it looks for a few environment variables. If they aren't found, then Bundler declares itself the master Bundler process and adds environment variables that describe the path to the project's Gemfile (the equivalent of .tool-versions), and future Bundler instances (in child processes) will look for and use those variables to make sure that the same dependencies are used. asdf could communicate that way as well.

But, if asdf is to manage all direct and indirect dependencies, I feel like the solution will reimplement most of Nix. Nix was designed from the ground up to trace a path through the dependency tree using checksums.

I was just reading About Nix and found this gem:

Nix is extremely useful for developers as it makes it easy to automatically set up the the build environment for a package. Given a Nix expression that describes the dependencies of your package, the command nix-shell will build or download those dependencies if they’re not already in your Nix store, and then start a Bash shell in which all necessary environment variables (such as compiler search paths) are set.

search for Managing build environments on the About Nix page.

I don't know if wrapping or depending of Nix is a good idea, but I'm just throwing this out here, to collect feedback.

[Stratus3D]

We definitely could attempt to solve this with proper ordering of dependencies in the .tool-versions file. Or perhaps explicitly listing required tools in an optional callback shell script in each plugin. For example, postgis would have a bin/dependencies script which would simply echo "postgresql", which would force asdf to tell the user they must install a plugin for postgres first. This would make asdf more of an all or nothing sort of thing, which probably would be a step in the wrong direction. Proper order in .tool-versions is probably better.

It definitely seems like Nix is the future as far as generic package management goes. Last time I looked at it I felt it was a little more heavy handed than I wanted, so I went with asdf. I'm not sure having asdf depend on Nix is a good idea though. Unless asdf could provide some advantages over plain nix it seems like it might be better just to refer people to nix if they need explicit compile/install time dependency management.

@asdf-vm/maintainers what do you guys think? This is definitely not a simple problem to solve.

[vic]

@francois responding to your original question, perhaps setting ASDF_POSTGRES_VERSION environment variable like env ASDF_POSTGRES_VERSION=xxxx asdf install postgis yyyy could help. The asdf README mentions how you can set versions as environment variables and not depend on having a tool-versions file.

@Stratus3D regarding how to differentiate between a rebar-3.4.1 installation compiled against say, erlang 19 and erlang 20, I'd propose to support version tags. For example, having the following .tool-versions file and doing asdf install

erlang 20.0
rebar 3.4.1@otp-20

asdf would install erlang 20.0 and then install rebar 3.4.1@otp-20 the string @otp-20 is just a user supplied tag to identify different-purpose installations of rebar 3.4.1, in this case compiled against different erlang versions. So asdf would install rebar into ~/.asdf/installs/rebar-3.4.1@otp-20.

The plugins would not need to know about these tags, as they would be handled only by the asdf install command, and the asdf-erlang installer would just get an ASDF_INSTALL_VERSION=3.4.1 and ASDF_INSTALL_PATH=/home/vic/.asdf/installs/rebar-3.4.1@otp-20.

I'd be happy to provide a PR if this makes sense to you.

The only thing I'd like to make sure, is that asdf install activates the corresponding tool versions in order, so that we can be sure that my rebar is actually compiled with erlang 20.

[francois]

@vic, who's responsible for the installation order? Is asdf reading all the lines and then doing a topological sort or is asdf reading the file line by line and the human is responsible for choosing the correct order?

[vic]

@francois AFAIK asdf is reading line by line and trying to install each tool at a time from top to bottom of the file. Thing is, if you have A and B on the file, even if A gets installed before B, i'm not sure A gets activated (meaning you can use it) at B installation time, would have to take a closer look at the code, but as I can recall it's just installed but not activated.

[vic]

@francois Just made a short test, and looks like yes, when you install B it's being installed with the A version as specified in the previous line :)

Here's the files I used, and how to test https://gist.github.com/vic/f4d8ef0c1d3232c7591e1ef80e31a082

[krainboltgreene]

I just ran into this issue because Rails has a command called bin/rake db:structure:dump which uses Kernal.system() to evoke pg_dump in a subshell (which uses /bin/sh), which has it's own $PATH that ignores whatever I've setup in .zshrc. The result is as above, asdf shims not being in my path.