feat: install dependencies in .tool-versions order

[chrisjpalmer]

UPDATE

Summary

This PR changes the behaviour of asdf install to install tools in the order they are listed in .tool-versions.

The purpose is to prevent installation failures due to dependencies between plugins. An example is the poetry python plugins. When running asdf install for these, the poetry plugin will typically fail because python is not available (see here). This requires the user to manually run asdf install python xxx first and then continue with the rest of the install.

Rather than maintain the correct plugin order as part of the tool, this responsibility can be forwarded to users of the tool via .tool-versions file. asdf install will respect the order listed in .tool-versions when installing the tools.

Other Information

1. Tests have been written to verify the new behaviour works. All the old tests are passing locally except for two, but they seem unrelated.
   • asdf update --head should checkout the master branch
   • plugin_list_all should sync repo when check_duration set to 0
2. The behaviour of the tool is to start in the deepest directory (which is the cwd) and recurse upwards to shallower directories.
   • install tools in the .tool-versions file one by one
   • install any tools from legacy file versions if a plugin is installed for one, and it didn't appear in .tool-versions file
   • continue to recurse to shallower directories until a tool for every plugin has been installed

Example

Given the following .tool-versions:

nodejs 18.16.1
python 3.10.13
poetry 1.4.2

before

asdf install

kubectl 1.24.16 is already installed
nodejs 18.16.1 is already installed
No preset version installed for command python3
Please install a version by running one of the following:

asdf install python 3.10.13

or add one of the following versions in your config file at /Users/xxx/workspace/service-developer-portal/.tool-versions
python 3.10.2

Cleanup: Something went wrong!

48 /Users/xxx/.asdf/plugins/poetry/bin/install: POETRY_HOME=$install_path python3 - --version "$version" $flags

after

asdf install

nodejs 18.16.1 is already installed
python-build 3.10.13 /Users/xxx/.asdf/installs/python/3.10.13
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.13.tar.xz...
-> https://www.python.org/ftp/python/3.10.13/Python-3.10.13.tar.xz
Installing Python-3.10.13...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.13 to /Users/xxx/.asdf/installs/python/3.10.13
asdf: Warn: You have configured asdf to preserve downloaded files (with always_keep_download=yes or --keep-download). But
asdf: Warn: the current plugin (python) does not support that. Downloaded files will not be preserved.
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/Users/xxx/.asdf/installs/poetry/1.4.2/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.4.2): Done

Poetry (1.4.2) is installed now. Great!

To get started you need Poetry's bin directory (/Users/xxx/.asdf/installs/poetry/1.4.2/bin) in your `PATH`
environment variable.

Add `export PATH="/Users/xxx/.asdf/installs/poetry/1.4.2/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/Users/xxx/.asdf/installs/poetry/1.4.2/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Configuring poetry to behave properly with asdf ...
Running: "poetry config virtualenvs.prefer-active-python true".

('Configuration file exists at /Users/xxx/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/xxx/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
asdf: Warn: You have configured asdf to preserve downloaded files (with always_keep_download=yes or --keep-download). But
asdf: Warn: the current plugin (poetry) does not support that. Downloaded files will not be preserved.
Either specify a tool & version in the command
OR add .tool-versions file in this directory
or in a parent directory

... old PR description

Summary

This PR allows asdf to install tools in the order they are listed in .tool-versions. It works by the command asdf install --ordered-install to be provided.

The purpose is to prevent installation failures due to dependencies between plugins. An example is the poetry python plugins. When running asdf install for these, the poetry plugin will typically fail because python is not available (see here). This requires the user to manually run asdf install python xxx first and then continue with the rest of the install.

The PR takes a simple approach to resolve this problem. Rather than maintain the correct plugin order as part of the tool, which is much too difficult, allow developers to maintain the order in.tool-versions file. Then make the asdf tool follow that order when --ordered-install is provided.

Other Information

I have not written any tests, nor have I done extensive testing to verify this doesn't break any old functionality. I also have not addressed this change for multiple levels of .tool-versions files in higher directories. I can look into these things, if people like this change and we think its worth pursuing. Do let me know.

Our company uses asdf extensively for streamlining developer workflows so we are interested in improving the tool if possible.

Thanks!

[chrisjpalmer]

Excited to see this pr! Why even add the option? I'd vote for just making this the unconditional behavior.

I'm down for that, just thought its better to leave the option in there in case it breaks anything for people using the tool now.

[HashNuke]

@chrisjpalmer Thanks for this very useful PR + the notes on the tests and the scenarios not handled.

Ordered install can be the default behaviour.

But I have not given any thought to how we could handle order across .tool-versions files (especially $HOME/.tool-versions and ${pwd}/.tool-versions). Open to ideas.

[HashNuke]

Changed this to a draft PR since this is work in progress.

[chrisjpalmer]

Thanks Akash. This sounds good.

I have some initial thoughts on how we can support multiple .tool-versions files in higher directories. First I wanted to confirm the current behvaviour.

Current Process

Based on some testing and reverse engineering the code, I can see that ASDF does the following steps:

1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early.
   • Note: Higher level directories with .tool-versions are not searched.
2. Iterate plugins in alpha order. For each plugin:
   • Recursively search .tool-versions files for a corresponding version - starting with $PWD/.tool-versions, going up a directory each time.
   • As soon as a match is found, install that version and move on to the next plugin

Let me know if you agree with the above analysis. Assuming I'm right, then the key takeaway is that the .tool-versions files are recursively searched and the default .tool-versions file is considered last.

New Process

To emulate the same behaviour whilst now considering the version order in .tool-versions I believe we need the following process:

1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed... (keep this the same)
2. Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file:
   • Install versions in the order they appear in the file.
   • Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.
3. Search ends when one of two conditions is true:
   • there are no more directories to search
   • we have satisfied every plugin that is installed on the system

Just for sanity checking its important to note the following about this new process:

• versions in earlier .tool-versions take precedence over ones in higher directories - this is the same as the previous process which resolves the version for each plugin in earlier .tool-versions files before searching higher
• versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

---

Let me know what you think :) Chris

[chrisjpalmer]

Hi there. I have actually gone ahead and implemented this. Current status is:

• basic code is in place
• replaced old install process with ordered install process
• no unit tests written
• hand tested on my machine with .tool-version files
• have not tested legacy install files

[chrisjpalmer]

Hey @HashNuke just checking in to see if you are happy with the progress on this PR ?

[chrisjpalmer]

Update

I ran the unit test suite... majority are passing!

I added a new unit test which verifies the ordered install behaviour:

• install_command installs plugins in order

Note: these are failing (on my local) but I believe they are unrelated:

• asdf update --head should checkout the master branch
• plugin_list_all should sync repo when check_duration set to 0

For consideration

I was chatting with a colleague of mine, and we were thinking that in addition to having tools be installed in .tool-version order, it might be good to consider the directory structure as well when installing tools. Suppose you have the following structure:

/
  - .tool-versions: python xxx
       /subdir
          - .tool-versions: poetry yyy

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Thoughts on this ?

Thanks

[chrisjpalmer]

Hi there guys... I think this is ready for review now :)

[chrisjpalmer]

@HashNuke or @jfly would you mind taking a look to confirm you are happy with progress ?

[Stratus3D]

1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early. Note: Higher level directories with .tool-versions are not searched.

If this is the case I think this might be a bug in the old process (probably out of scope for this PR).

---

Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file:

Install versions in the order they appear in the file. Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.

For the new process we might actually want to start at the uppermost .tool-versions file, installing versions in order, and work our way down to the current directory. I feel like the home/root dir is typically used for system wide stuff that the user just happens to be using asdf for (ie version doesn't vary between dirs). This is definitely a nitpick. No need to implement right now.

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Yep, exactly!

---

versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

Interesting, I wonder if that is best. I don't think we should be doing anything until all required plugins (ie all those referenced in all .tool-version files at and above the current dir) are installed. Again, definitely out of scope for this PR but I think it may have ended up this way by accident rather than by design. Any idea why we might want to silently ignore missing plugins at this step?

[chrisjpalmer]

Thanks @Stratus3D for your review. Appreciate that time you put into it. I have tried to address all your comments and respond to anything that I still wasn't sure about.

1. Check $PWD/.tool-versions to see if there are any versions who don't have a corresponding plugin installed on the system. If there are any versions without a corresponding plugin, a message is issued and the command exits early. Note: Higher level directories with .tool-versions are not searched.

If this is the case I think this might be a bug in the old process (probably out of scope for this PR).

Quite possibly. We can always follow this PR up with another one which hardens this behaviour.

Recursively scan .tool-versions files starting with $PWD/.tool-versions, going up a directory each time. For each .tool-versions file: Install versions in the order they appear in the file. Store the plugins we have installed in a variable to keep track of what we have already covered off - this allows us to skip version installs for same plugin versions as we traverse higher directories.

For the new process we might actually want to start at the uppermost .tool-versions file, installing versions in order, and work our way down to the current directory. I feel like the home/root dir is typically used for system wide stuff that the user just happens to be using asdf for (ie version doesn't vary between dirs). This is definitely a nitpick. No need to implement right now.

Currently in both the previous and new install process, poetry will be installed before python because asdf install installs tools in the deepest directory and the recurses upwards to shallower directories. However, there is an argument that it might be more useful to do this the opposite way. In a monorepo for example a user may specify a general tool in the root directory, and then sub directories could be free to declare additional tools. It may be possible that those additional tools have a dependency on tools declared in the root directory. Therefore it could make sense to actually modify asdf install's behaviour to recurse from the shallower directory downwards.

Yep, exactly!

I would be happy to chase this up in a follow up PR.

versions without a corresponding plugin will not contribute to the installation process. This is the same as the previous process because, the previous installation process is driven by the list of plugins that do exist and hence will also ignore versions that have no corresponding plugin.

Interesting, I wonder if that is best. I don't think we should be doing anything until all required plugins (ie all those referenced in all .tool-version files at and above the current dir) are installed. Again, definitely out of scope for this PR but I think it may have ended up this way by accident rather than by design. Any idea why we might want to silently ignore missing plugins at this step?

Ah I think when I said that "a version without a corresponding plugin will not contribute to the installation process" what I meant was, ignoring the sanity check that runs before installation kicks off, the main installation loop ignores unknown plugins. However there is a sanity check that runs before the main installation which lets the user know if it detects an unknown plugin. This is only in place for the .tool-versions file in the current directory. Again suggests to me a follow up PR to harden this behavior.

[Stratus3D]

Thanks @chrisjpalmer! just scanned through the code and left some comments.

I see our build is broken here, but it isn't due to your changes. I'll see about fixing it on master.

[stoutput]

Just checking in on this as it's been a while @Stratus3D. This would be an extremely useful feature to have implemented especially for python + poetry. Seems like the remaining asks are easy cleanups that could be knocked out in 2 min