search

jdx / mise

dev tools, env vars, task runner
https://mise.jdx.dev
MIT License
9.77k stars 278 forks source link

possible bug in legacy version file support with multiple plugins using the same underlying source (e.g. hashicorp) #621

Closed jhcolville closed 1 year ago

jhcolville commented 1 year ago

Describe the bug

Hi, first thanks very much to all the contributors to this project, I think it's a great tool. I'm not 100% sure this is a bug, but the behaviour I'm seeing is that if I have multiple plugins installed using the same source (e.g. terraform-ls, terraform, packer - all sharing the asdf-hashicorp.git source) and I try to use legacy version files in a project (e.g. .terraform-version, .packer-version) rtx does not attempt to manage the version specified in those files. The behaviour seems to be that the version to use is cached as an empty file in the cache legacy folder of the lexicographically last plugin name.

By removing the order-last plugin one at a time the prior plugin's .tool-version begins to work, e.g. removing the terraform-ls plugin enables the .terraform-version to be correctly parsed and managed.

To Reproduce

[xxx@hostname tmp]$ rtx plugin list | grep -E 'terraform|packer'
iam-policy-json-to-terraform
packer
terraform
terraform-docs
terraform-ls
[xxx@hostname tmp]$ rtx cache clear
rtx cache cleared
[xxx@hostname tmp]$ ls -la ~/.cache/rtx/*/legacy/*
ls: cannot access '/home/xxx/.cache/rtx/*/legacy/*': No such file or directory
[xxx@hostname tmp]$ cat project/.terraform-version
1.5.0
[xxx@hostname tmp]$ cat project/.packer-version
1.9.1
[xxx@hostname tmp]$ cd project
[xxx@hostname project]$ rtx bin-paths
[xxx@hostname project]$ which terraform
~/.local/share/rtx/shims/terraform
[xxx@hostname project]$ which packer
~/.local/share/rtx/shims/packer
[xxx@hostname project]$ terraform -version
rtx No version is set for shim: terraform
Set a global default version with one of the following:
rtx use -g terraform@0.11.15
rtx use -g terraform@0.12.31
rtx use -g terraform@0.13.7
rtx use -g terraform@0.14.11
rtx use -g terraform@0.15.5
rtx use -g terraform@1.5.0
rtx Run with RTX_DEBUG=1 for more information
[xxx@hostname project]$ packer -version
^C
[xxx@hostname project]$ ls -la ~/.cache/rtx/*/legacy/*
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/a81883f0c92a18c.txt
[xxx@hostname project]$ rtx plugin remove terraform-ls
rtx terraform-ls uninstalled                                                                              ✓ 0s 
[xxx@hostname project]$ touch .terraform-version
[xxx@hostname project]$ rtx bin-paths
/home/xxx/.local/share/rtx/installs/terraform/1.5.0/bin
[xxx@hostname project]$ terraform -version
Terraform v1.5.0
on linux_amd64
[xxx@hostname project]$ ls -la ~/.cache/rtx/*/legacy/*
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:44 /home/xxx/.cache/rtx/terraform/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 5 Jun 20 16:44 /home/xxx/.cache/rtx/terraform/legacy/a81883f0c92a18c.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/a81883f0c92a18c.txt
[xxx@hostname project]$ rtx plugin remove terraform
rtx terraform uninstalled                                                                                 ✓ 0s
[xxx@hostname project]$ touch .packer-version
[xxx@hostname project]$ rtx bin-paths
/home/xxx/.local/share/rtx/installs/packer/1.9.1/bin
[xxx@hostname project]$ packer -version
1.9.1
[xxx@hostname project]$ ls -la ~/.cache/rtx/*/legacy/*
-rw-r--r--. 1 xxx domain users 5 Jun 20 16:45 /home/xxx/.cache/rtx/packer/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:45 /home/xxx/.cache/rtx/packer/legacy/a81883f0c92a18c.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:44 /home/xxx/.cache/rtx/terraform/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 5 Jun 20 16:44 /home/xxx/.cache/rtx/terraform/legacy/a81883f0c92a18c.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/8a677c53c1dfab91.txt
-rw-r--r--. 1 xxx domain users 0 Jun 20 16:43 /home/xxx/.cache/rtx/terraform-ls/legacy/a81883f0c92a18c.txt
[xxx@hostname project]$

Expected behavior

rtx should make the tool version matching the specification in the legacy version file available

rtx doctor output

$ rtx doctor
rtx version:
  1.32.0 linux-x64 (da025fe 2023-06-03)

build:
  Target: x86_64-unknown-linux-gnu
  Features: SELF_UPDATE
  Built: Sat, 03 Jun 2023 14:21:01 +0000
  Rust Version: rustc 1.69.0 (84c898d65 2023-04-16)
  Profile: release

shell:
  /bin/bash
  GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)
  Copyright (C) 2020 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

  This is free software; you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

rtx environment variables:
  RTX_SHELL=bash

settings:
  {"experimental": "false", "missing_runtime_behavior": "warn", "always_keep_download": "false", "always_keep_install": "false", "legacy_version_file": "true", "plugin_autoupdate_last_check_duration": "10080", "trusted_config_paths": "[]", "verbose": "false", "asdf_compat": "false", "jobs": "4", "disable_default_shorthands": "false", "log_level": "INFO", "raw": "false"}

config files:
  /tmp/project/.packer-version
  /tmp/project/.terraform-version

plugins:
  age                            https://github.com/threkk/asdf-age#4df80a0
  argocd                         https://github.com/beardix/asdf-argocd.git#429bbf9
  aws-vault                      https://github.com/karancode/asdf-aws-vault.git#678cacf
  awscli                         https://github.com/MetricMike/asdf-awscli.git#2e31396
  chezmoi                        https://github.com/joke/asdf-chezmoi.git#eefb05b
  direnv                         https://github.com/asdf-community/asdf-direnv.git#7945cbb
  fx                             https://gitlab.com/wt0f/asdf-fx.git#8ba1b03
  fzf                            https://github.com/kompiro/asdf-fzf.git#d19eb67
  go-getter                      https://github.com/ryodocx/asdf-go-getter.git#7483a12
  helm                           https://github.com/Antiarchitect/asdf-helm.git#a39e17b
  helmfile                       https://github.com/feniix/asdf-helmfile.git#3f1681e
  iam-policy-json-to-terraform   https://github.com/carlduevel/asdf-iam-policy-json-to-terraform.git#ac2b087
  k9s                            https://github.com/looztra/asdf-k9s.git#2b3af87
  kubectl                        https://github.com/asdf-community/asdf-kubectl.git#cbe6df4
  kubectx                        https://gitlab.com/wt0f/asdf-kubectx.git#8c0b4aa
  kubent                         https://github.com/virtualstaticvoid/asdf-kubent.git#7a1d974
  kubeval                        https://github.com/stefansedich/asdf-kubeval.git#6ccc18f
  kustomize                      https://github.com/Banno/asdf-kustomize.git#2efaa38
  node                           (core)
  packer                         https://github.com/asdf-community/asdf-hashicorp.git#c44c314
  python                         (core)
  ripgrep                        https://gitlab.com/wt0f/asdf-ripgrep.git#e836665
  shellcheck                     https://github.com/luizm/asdf-shellcheck.git#b160a52
  shfmt                          https://github.com/luizm/asdf-shfmt.git#a42c5ff
  sops                           https://github.com/feniix/asdf-sops.git#954411c
  starship                       https://github.com/grimoh/asdf-starship.git#f546452
  stern                          https://github.com/looztra/asdf-stern.git#e17efba
  task                           https://github.com/particledecay/asdf-task.git#54ee5e6
  terraform-docs                 https://github.com/looztra/asdf-terraform-docs.git#3ec72e5
  terragrunt                     https://github.com/ohmer/asdf-terragrunt.git#4a6651a
  yq                             https://github.com/sudermanjr/asdf-yq.git#772992f

toolset:
  packer@1.9.1

No problems found

I hope I've provided enough detail, if there's anything more I can do to help illustrate the problem please let me know.

Using .tool-versions file, or .toml, or RTX_PLUGIN_VERSION environment variables all works fine and I'd just do that except that I'm trying to migrate a team of engineers and a lot of repositories over to rtx to replace a lot of piecemeal environment manager scripts and it would be ideal if rtx could be a drop-in replacement.

Thanks,

Johan

jhcolville commented 1 year ago

Hi,

thanks very much for taking a look at this one so quickly, I looked forward to trying the fix.

Unfortunately the behaviour I'm seeing persists, e.g. here's a simple project folder with .terraform-version and .terragrunt-version, only .terragrunt-version is respected until I remove the terraform-ls plugin. I'm using rtx 1.32.1 with cleared cache and can see the file paths have changed, though that said it seems to me the plugin name was already incorporated into the path.

❯ rtx cache clear
rtx cache cleared

❯ rtx bin-paths

❯ cd project

❯ cat .terraform-version
0.12.24

❯ cat .terragrunt-version
0.37.3

❯ rtx bin-paths
/home/xxx/.local/share/rtx/installs/terragrunt/0.37.3/bin

❯ rtx plugin remove terraform-ls
rtx terraform-ls uninstalled                                                                                            ✓ 0s

❯ rtx bin-paths
[WARN] Tool not installed: terraform@0.12.24
/home/xxx/.local/share/rtx/installs/terragrunt/0.37.3/bin

❯ rtx install
...
rtx terraform@0.12.24 Extracting terraform archive                                                                      ✓ 3s
❯ rtx bin-paths
/home/xxx/.local/share/rtx/installs/terragrunt/0.37.3/bin
/home/xxx/.local/share/rtx/installs/terraform/0.12.24/bin

❯ rtx doctor
rtx version:
  1.32.1 linux-x64 (8ce9896 2023-06-27)

build:
  Target: x86_64-unknown-linux-gnu
  Features: SELF_UPDATE
  Built: Tue, 27 Jun 2023 14:03:07 +0000
  Rust Version: rustc 1.70.0 (90c541806 2023-05-31)
  Profile: release

shell:
  /bin/bash
  GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)
  Copyright (C) 2020 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

  This is free software; you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

rtx environment variables:
  RTX_SHELL=bash

settings:
  {"experimental": "false", "missing_runtime_behavior": "warn", "always_keep_download": "false", "always_keep_install": "false", "legacy_version_file": "true", "plugin_autoupdate_last_check_duration": "10080", "trusted_config_paths": "[]", "verbose": "false", "asdf_compat": "false", "jobs": "4", "disable_default_shorthands": "false", "log_level": "INFO", "raw": "false"}

config files:
  /home/xxx/.config/rtx/config.toml
  /home/xxx/git/yyy/.terraform-version
  /home/xxx/git/yyy/.terragrunt-version

plugins:
  age                            https://github.com/threkk/asdf-age#4df80a0
  argocd                         https://github.com/beardix/asdf-argocd.git#429bbf9
  aws-vault                      https://github.com/karancode/asdf-aws-vault.git#678cacf
  awscli                         https://github.com/MetricMike/asdf-awscli.git#2e31396
  chezmoi                        https://github.com/joke/asdf-chezmoi.git#eefb05b
  direnv                         https://github.com/asdf-community/asdf-direnv.git#7945cbb
  fx                             https://gitlab.com/wt0f/asdf-fx.git#8ba1b03
  fzf                            https://github.com/kompiro/asdf-fzf.git#d19eb67
  go-getter                      https://github.com/ryodocx/asdf-go-getter.git#7483a12
  helm                           https://github.com/Antiarchitect/asdf-helm.git#a39e17b
  helmfile                       https://github.com/feniix/asdf-helmfile.git#3f1681e
  iam-policy-json-to-terraform   https://github.com/carlduevel/asdf-iam-policy-json-to-terraform.git#ac2b087
  k9s                            https://github.com/looztra/asdf-k9s.git#2b3af87
  kubectl                        https://github.com/asdf-community/asdf-kubectl.git#cbe6df4
  kubectx                        https://gitlab.com/wt0f/asdf-kubectx.git#8c0b4aa
  kubent                         https://github.com/virtualstaticvoid/asdf-kubent.git#7a1d974
  kubeval                        https://github.com/stefansedich/asdf-kubeval.git#6ccc18f
  kustomize                      https://github.com/Banno/asdf-kustomize.git#2efaa38
  node                           (core)
  packer                         https://github.com/asdf-community/asdf-hashicorp.git#c44c314
  python                         (core)
  ripgrep                        https://gitlab.com/wt0f/asdf-ripgrep.git#e836665
  shellcheck                     https://github.com/luizm/asdf-shellcheck.git#b160a52
  shfmt                          https://github.com/luizm/asdf-shfmt.git#a42c5ff
  sops                           https://github.com/feniix/asdf-sops.git#954411c
  starship                       https://github.com/grimoh/asdf-starship.git#f546452
  stern                          https://github.com/looztra/asdf-stern.git#e17efba
  task                           https://github.com/particledecay/asdf-task.git#54ee5e6
  terraform                      https://github.com/asdf-community/asdf-hashicorp.git#c44c314
  terraform-docs                 https://github.com/looztra/asdf-terraform-docs.git#3ec72e5
  terragrunt                     https://github.com/ohmer/asdf-terragrunt.git#4a6651a
  yq                             https://github.com/sudermanjr/asdf-yq.git#772992f

toolset:
  terragrunt@0.37.3, terraform@0.12.24

No problems found

❯ ls -la ~/.cache/rtx/*/legacy/*/*
-rw-r--r--. 1 xxx domain users 7 Jun 28 14:10 /home/xxx/.cache/rtx/terraform/legacy/terraform/c0b89868100570d9.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:09 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/c0b89868100570d9.txt
-rw-r--r--. 1 xxx domain users 6 Jun 28 14:10 /home/xxx/.cache/rtx/terragrunt/legacy/terragrunt/9a4d5188f8450664.txt

It seemed to me anyway that the cache files were already separated by plugin name, but when multiple plugins had the same source the cache files would always be zero size and they'd all be dropped into the directory indicated by the alphabetically last plugin name. Sorry if I'm not making this very clear, but here's another example where I incrementally remove plugins and you can see the cache directories being correctly populated as a result:

❯ rtx cache clear
rtx cache cleared

~/git
❯ ls -la ~/.cache/rtx/*/legacy/*/*
ls: cannot access '/home/xxx/.cache/rtx/*/legacy/*/*': No such file or directory

❯ cd project

❯ ls -la ~/.cache/rtx/*/legacy/*/*
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/c0b89868100570d9.txt

❯ rtx bin-paths

❯ rtx plugin remove terraform-ls
rtx terraform-ls uninstalled                                                                                            ✓ 0s

❯ ls -la ~/.cache/rtx/*/legacy/*/*
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:35 /home/xxx/.cache/rtx/terraform/legacy/terraform/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 7 Jun 28 14:35 /home/xxx/.cache/rtx/terraform/legacy/terraform/c0b89868100570d9.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/c0b89868100570d9.txt

❯ rtx bin-paths
[WARN] Tool not installed: terraform@0.12.24

❯ rtx plugin remove terraform
rtx terraform uninstalled                                                                                               ✓ 0s

❯ ls -la ~/.cache/rtx/*/legacy/*/*
-rw-r--r--. 1 xxx domain users 5 Jun 28 14:35 /home/xxx/.cache/rtx/packer/legacy/packer/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:35 /home/xxx/.cache/rtx/packer/legacy/packer/c0b89868100570d9.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:35 /home/xxx/.cache/rtx/terraform/legacy/terraform/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 7 Jun 28 14:35 /home/xxx/.cache/rtx/terraform/legacy/terraform/c0b89868100570d9.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/b1da58780313d359.txt
-rw-r--r--. 1 xxx domain users 0 Jun 28 14:34 /home/xxx/.cache/rtx/terraform-ls/legacy/terraform-ls/c0b89868100570d9.txt

❯ rtx bin-paths
/home/xxx/.local/share/rtx/installs/packer/1.8.6/bin

If there's anything more I can do to illustrate let me know, I'll have a dig through the source but rust is far from my strongest suit.

Thanks again,

Johan

jdx commented 1 year ago

ah yeah I see and I can repro. Once you add terraform-ls it breaks the config parsing for terraform. There is an assumption in the data model that each legacy file only has a single tool associated with it: https://github.com/jdxcode/rtx/blob/7d4ed3979dab1a6e681280304847764f4acfeb1a/src/config/mod.rs#L338

I think that might need to be modified to be a list, but that's easier said than done since it requires a fair bit of changes to other code.