Support multiple archives for a single `.chezmoiexternal` entry

[felipecrs]

Is your feature request related to a problem? Please describe.

I want to install both kubectx and kubens into ~/.local/bin. However, they are distributed as archives, and currently chezmoi doesn't support extracting multiple archives into a single dir.

Describe the solution you'd like

I guess something like below

# .chezmoiexternal.yaml

".local/bin":
  - type: archive
    url: "https://github.com/ahmetb/kubectx/releases/download/v0.9.4/kubectx_v0.9.4_linux_x86_64.tar.gz"
    include:
      - kubectx
  - type: archive
    url: "https://github.com/ahmetb/kubectx/releases/download/v0.9.4/kubens_v0.9.4_linux_x86_64.tar.gz"
    include:
      - kubens

EDIT: I had previously reported an issue with the above alternative, but it was my mistake. I was using archive instead of file.

And chezmoi would either:

1. Merge contents giving preference to the last one
2. Error out if there are colliding files

Describe alternatives you've considered

".local/bin/kubectx":
  type: file
  url: "https://github.com/ahmetb/kubectx/releases/download/v0.9.4/kubectx_v0.9.4_linux_x86_64.tar.gz"
  filter:
    command: tar
    args: ["--extract", "--gunzip", "--to-stdout", "kubectx"]
  executable: true
".local/bin/kubens":
  type: file
  url: "https://github.com/ahmetb/kubectx/releases/download/v0.9.4/kubens_v0.9.4_linux_x86_64.tar.gz"
  filter:
    command: tar
    args: ["--extract", "--gunzip", "--to-stdout", "kubens"]
  executable: true

Additional context

None.

[twpayne]

The alternative you've considered works, right? What are the reasons for not using it?

Note that .chezmoiexternal is a template, so you can use a range loop over a list of dicts to reduce the code duplication.

Extracting multiple archives to the same directory is likely to quickly lead to conflicts, for example many archives contain some combination of COPYING, LICENSE, and README files.

[felipecrs]

The alternative you've considered works, right?

Yeah, it works 100%.

What are the reasons for not using it?

I'm using it by the way.

But my gut feeling is that using filters is slower than not, I think I have came to believe in this in some of our old discussions.

Note that .chezmoiexternal is a template, so you can use a range loop over a list of dicts to reduce the code duplication.

That's cool, thanks for the tip. There's also the includeTemplate function which I'm yet to adopt.

Extracting multiple archives to the same directory is likely to quickly lead to conflicts, for example many archives contain some combination of COPYING, LICENSE, and README files.

The include option can help sorting this issue.

[bradenhilton]

You could try this

dot_local/bin/.chezmoiexternal.yaml:

{{ range list "kubectx" "kubens" -}}
"{{ . }}":
  type: file
  url: "https://github.com/ahmetb/kubectx/releases/download/v0.9.4/{{ . }}_v0.9.4_linux_x86_64.tar.gz"
  filter:
    command: tar
    args: ["--extract", "--gunzip", "--to-stdout", "{{ . }}"]
  executable: true
{{ end -}}

[twpayne]

But my gut feeling is that using filters is slower than not, I think I have came to believe in this in some of our old discussions.

It's certainly a bit slower as it involves spawning a process to do the unarchiving rather than doing the unarchiving in chezmoi's own process, but exactly much slower it is depends a bit on the archive format. In the case where you want only one file from each archive (as is the case here) this is probably not significant. If you want to extract multiple files from the same .tar archive then it will be slower because .tar files lack a central directory providing access to individual files, i.e with .tar files you have to process the full archive for each file you extract.

[felipecrs]

Alright. Thanks for the inputs. You can close this issue if you think it's not worth implementing.

[bradenhilton]

I just noticed the template I posted wasn't quite right, I've updated it.

[felipecrs]

For completeness, I realized that it's better to use short options in tar rather than long ones, like -xzO, due to portability. My tests pointed out that alpine doesn't support long options:

https://github.com/felipecrs/dotfiles/actions/runs/3387672896/jobs/5628720911#step:3:10571