arrrgi commented 11 months ago

What exactly are you trying to do?

On a Linux server that I do not have root/sudo access to, I am fetching the vlt binary with .chezmoiexternal.yaml as so:

{{- $vltVersion := "0.2.2" -}}
".local/bin/vlt":
  type: archive-file
  url: "https://releases.hashicorp.com/vlt/{{ $vltVersion }}/vlt_{{ $vltVersion }}_{{ .chezmoi.os }}_{{ .chezmoi.arch }}.zip"
  executable: true

When I run chezmoi init username --apply I get a chezmoi error

$ chezmoi init arrrgi --apply --branch=feature/setup-scripts --verbose
chezmoi: .local/bin/vlt: missing path

Before running the command above, chezmoi doctor reports that vlt is not found in $PATH.

This obviously comes down to the Application Order that has been raised in previous issues but I'm struggling to find another way to make this scenario work without manually installing vlt before chezmoi.

What have you tried so far?

Describe what you have tried so far.

Where else have you checked for solutions?

[x] I have read chezmoi's user guide, and not found the answer.
[x] I have searched chezmoi's reference guide, and not found the answer.
[ ] Other, please give details.

Output of `chezmoi doctor`

```console $ chezmoi doctor ```

Additional context

Add any other context about the problem here.

arrrgi commented 11 months ago

Dug into this further and found I was missing the "path" spec for the archive-file external type, a miss on my part. However, after I updated this, I can now see when I apply these changes, no destination file is created in the target for the unpacked file.

ie.

{{ $ageVersion := "1.1.1" -}}
{{ $vltVersion := "0.2.2" -}}

{{- if and .target.wsl (or .secrets.apikeys .secrets.sshkeys .secrets.storagekeys) }}
".local/bin/age":
  type: archive-file
  url: "https://github.com/FiloSottile/age/releases/download/v{{ $ageVersion }}/age-v{{ $ageVersion }}-{{ .chezmoi.os }}-{{ .chezmoi.arch }}.tar.gz"
  executable: true
  path: "age/age"

".local/bin/vlt":
  type: archive-file
  url: "https://releases.hashicorp.com/vlt/{{ $vltVersion }}/vlt_{{ $vltVersion }}_{{ .chezmoi.os }}_{{ .chezmoi.arch }}.zip"
  executable: true
  path: "vlt"
{{- end }}

should put the age binary into ~/.local/bin but the file never makes it there. Running apply with --debug I can see it fetches the archive but doesn't go beyond that.

Output from command:

$ chezmoi apply --refresh-externals=always --debug
<trimmed for brevity>
2023-10-05T10:42:16+10:00 INF HTTPRequest component=sourceState duration=1.073034003s method=GET size=4465769 status="200 OK" statusCode=200 url=https://github.com/FiloSottile/age/releases/download/v1.1.1/age-v1.1.1-linux-amd64.tar.gz
2023-10-05T10:42:17+10:00 ERR Mkdir error="mkdir /home/rgillson/.cache/chezmoi/external: file exists" component=system name=/home/rgillson/.cache/chezmoi/external perm=448
2023-10-05T10:42:17+10:00 INF Stat component=system name=/home/rgillson/.cache/chezmoi/external
2023-10-05T10:42:17+10:00 INF WriteFile component=system data="\x1f�\b\x00\x00\x00\x00\x00\x00\x03�Y\x7ftSU�\x7f�M�TZ^Tj[-4���uQ���F-�%)7�*�A\x01q��\x12ˎ�R\x12\n�CiZ�7..." name=/home/rgillson/.cache/chezmoi/external/c5cc7163b8e325246e5a6882e3d166dbcfdf7fe57b227428102e004abe98534d perm=384 size=4465769
2023-10-05T10:42:17+10:00 INF Chtimes atime=2023-10-05T00:42:15Z component=system mtime=2023-10-05T00:42:15Z name=/home/rgillson/.cache/chezmoi/external/c5cc7163b8e325246e5a6882e3d166dbcfdf7fe57b227428102e004abe98534d
<trimmed for brevity>
2023-10-05T10:42:17+10:00 ERR Output error="exec: \"vlt\": executable file not found in $PATH" args=["vlt","secrets","get","--plaintext","--app-name","chezmoi","--project","<REDACTED>","--organization","<REDACTED>","sshSigningKeyPersonal"] dir=/home/rgillson duration="18.3µs" output= path=vlt size=0
chezmoi: template: dot_config/git/standard.tmpl:4:18: executing "dot_config/git/standard.tmpl" at <hcpVaultSecret "sshSigningKeyPersonal">: error calling hcpVaultSecret: vlt secrets get --plaintext --app-name chezmoi --project <REDACTED> --organization <REDACTED> sshSigningKeyPersonal: exec: "vlt": executable file not found in $PATH

zydou commented 11 months ago

I think a temporary workaround is to create a run_once_before script to download the vlt into $PATH

arrrgi commented 11 months ago

One of the great things about Chezmoi externals is that it supports multiple archive formats.

The workaround scenario you mention would require both curl and a suitable unzip binary to also be available in the path, which is undoubtedly workable, but slightly more than trivial to implement on systems I don't have root access to install these via the built in package manager, and properly avoiding version overlaps on a system that I do have root access to install these via package manager as a dependency.

It's definitely more tempting to do things the Chezmoi way, but I could see this also being less than trivial to implement. Interested to here what @twpayne and @halostatue think

I'm not entirely discounting using a run_before script, I'd just prefer not to if I can help reduce the number or size of scripts to execute.

twpayne commented 11 months ago

I've also hit this problem. I'm not sure how to solve it. Feedback and experiences welcome.

As I understand it, we both need to install some sort of prerequisites before chezmoi will fully work for you. In my case, I need the 1Password CLI. @arrrgi needs the vlt binary and maybe to run vlt login and set some environment variables first.

I think we need some kind of "pre-init" script to fix things up, but I'm not sure when this should be run. For example:

Should "pre-init" only be run once when you run chezmoi init on new machine for the first time? If so, how can you iteratively test this?
What if "pre-init" needs template variables? Should it run before chezmoi init writes the config file or after?

All ears here.

arrrgi commented 11 months ago

With #3290 merged I should in theory be able to skip the vlt login step, I'll confirm in the next day or two with any luck.

The problem to be solved here is that the tooling/binaries to support some of the templated secret retrieval functions needs to be in place before Chezmoi has to substitute these variables out with the actual secrets.

Some ideas I've had on solutions to this (in no specific order of preference) are:

A user supplied pre-init script that is run first by Chezmoi to install a package or fetch binaries and put them into either system or user path. This option puts the support and issues burden predominantly on the user. User defined template variables should not be supported in this pre-init stage, only the builtins otherwise there is an endless cycle of init before pre-init and vice versa
A command-line switch which calls a Chezmoi supported helper function to fetch only binary versions of secrets/encryption tools and leave system package installation as an edge case. This option puts the burden of support and issues with the Chezmoi maintainers and community and would require code to at least interpret how to fetch the most recent version of each supported secrets tool (1P, Bitwarden, Vlt, etc)
Tweaking the Application Order so that externals are processed after init and before scripts and templates and have the Chezmoi internal config updated with the detected paths to the vlt, op, age, gpg, etc binaries which are used for secret expansion/decryption. This option would allow externals to continue working as-is, and defer updating the target with scripts and templates without moving the needle on the support required for either group.
Use builtin's like Age, but for Vault, 1P, Bitwarden, etc - this option would require Go libraries for each to exist and be imported as modules so the support may be limited to only a handful depending on what is available

Just a note on Option 1: in practical application that might actually look like changing the current init function to run these scripts, then a config function which replaces init and would be where the user can build out their own variables for templating.

A few questions I had to maybe guide some deeper conversation on this:

Do each of the secrets integrations today have supporting Go libraries that they could be included as modules in Chezmoi as has already been done with Age?
If the pre-init idea is a proposed solution to the binaries for these secret managers being available before Chezmoi updates the target, would changing the Application Order be (as I see it) the most impactful way to address this as it has the least impact to users?

zydou commented 11 months ago

I work on various servers located in different countries. However, due to internet censorship in China, the network connections to many western services such as GitHub, npm, pypi, and crates.io are unreliable. To address this issue, I need to set a variable in the .chezmoi.toml.tmpl to indicate whether the server is located in China (country code CN). And I have to adjust the scriptEnv to use mirror URLs like PIP_INDEX_URL and HOMEBREW_API_DOMAIN. This ensures the scripts that install anaconda or homebrew in the .chezmoiscripts function properly.

This is exactly the pre-init process before chezmoi init. My current solution is somewhat awkward:

A script .chezmoiscripts/run_once_before_00_init.sh
```
#!/bin/bash
```

This script is executed before the chezmoi init step.

[[ ! -d "${HOME}/.local/state" ]] && mkdir -p "${HOME}/.local/state"

function detect_GFW() {

detect whether we are behind the China's Great Fire Wall (GFW)

if COUNTRY="$(curl -sSLkq4 --max-time 2 --proxy '' https://ipinfo.io/country)"; then
    mark_GFW "${COUNTRY}"
elif COUNTRY="$(curl -sSLkq4 --max-time 2 --proxy '' https://ipapi.co/country)"; then
    mark_GFW "${COUNTRY}"
else
    mark_GFW "CN"
fi

}

function mark_GFW() {

save the GFW indicator file

COUNTRY="${1}"
if [[ "${COUNTRY}" = "CN" ]]; then
    touch "${HOME}/.local/state/in-GFW" || true # behind the GFW
else
    touch "${HOME}/.local/state/out-GFW" || true  # outside the GFW
fi

}

if [[ ! -f "${HOME}/.local/state/in-GFW" && ! -f "${HOME}/.local/state/out-GFW" ]]; then echo "detecting whether we are behind the GFW..." detect_GFW fi


2. Put the following line on **TOP** of the `.chezmoi.toml.tmpl`

{{- output (joinPath .chezmoi.sourceDir ".chezmoiscripts/run_once_before_00_init.sh") }} {{- $GFW := "in_gfw" }} {{- if stat (joinPath .chezmoi.homeDir ".local/state/out-GFW") }} {{- $GFW = "out_gfw" }} {{- end }}

twpayne commented 11 months ago

Thank you @zydou. The use of output in .chezmoi.toml.tmpl to run a script is very clever.

Can you achieve the same result with the following (warning: untested)?

.chezmoi.toml.tmpl:

[data]
    country = {{ output "curl" "-sSLkq4" "--max-time" "2" "--proxy" "" "https://ipinfo.io/country" | quote }}

.local/share/chezmoi/dot_local/state/in-GFW.tmpl:

{{ if eq .country "CN" }}
true
{{ end }}

.local/share/chezmoi/dot_local/state/out-GFW.tmpl:

{{ if ne .country "CN" }}
true
{{ end }}

There is a slight difference to your solution in that the in-GFW and out-GFW files will contain either true or not exist, as opposed to being either empty or not existing, but this difference should not affect other scripts that rely on the presence or absence of these files.

With this approach, you can use the template test {{ if eq .country "CN" }} in your chezmoi templates instead of using the $GFW variable.

twpayne commented 11 months ago

Thank you @zydou. The use of output in .chezmoi.toml.tmpl to run a script is very clever.

Thinking about this further, this is a likely solution to the problem posed in https://github.com/twpayne/chezmoi/issues/3269#issuecomment-1756175751, i.e. how to do some kind of pre-initialization.

As .chezmoi.toml.tmpl is executed on chezmoi init but before chezmoi reads its config file, you can put something like:

{{ $_ := output "my-pre-init-script" }}

in your .chezmoi.init.tmpl to run my-pre-init-script whenever you run chezmoi init or chezmoi --init apply. This might be enough to solve the problem (warning: untested).

zydou commented 11 months ago

Can you achieve the same result with the following (warning: untested)?

.chezmoi.toml.tmpl:
[data]
    country = {{ output "curl" "-sSLkq4" "--max-time" "2" "--proxy" "" "https://ipinfo.io/country" | quote }}
With this approach, you can use the template test {{ if eq .country "CN" }} in your chezmoi templates instead of using the $GFW variable.

Thank you for your suggestion. This approach is indeed more convenient. However, there is a minor mistake that I need to trim the response from the curl query.

country = {{ output "curl" "-sSLkq4" "--max-time" "2" "--proxy" "" "https://ipinfo.io/country" | trim | quote }}

arrrgi commented 11 months ago

Nice write up on how you solved that @zydou !

Does this script get executed twice then because you have it located in .chezmoiscripts? Or did you create an ignore rule so it is only executed as a function of output ?

@twpayne - this is very close to Option 1 that I mentioned above. Is this the method you think you will end up adopting for getting pre-requisites ready for your templated 1Password secrets (closing this issue with it)? Or would you also consider exploring Option 3 to tweak Application Order?

zydou commented 11 months ago

Does this script get executed twice then because you have it located in .chezmoiscripts? Or did you create an ignore rule so it is only executed as a function of output ?

In my case, this doesn't matter because the pre-init script I'm using is very simple, so I don't have any ignore rules. However, based on my tests, the number of times this script is executed depends on how you invoke the chezmoi command.

[Twice]: run chezmoi init and then run chezmoi apply
[Once]: run chezmoi init --apply
[Once]: run chezmoi apply --init

arrrgi commented 11 months ago

Something close to this example script would be barely sufficient to get the job done in comparison to using externals, it's far from ideal though as it also has to manage vendoring versions of unzip and jq, and to make it more robust will add more code complexity.

I also assumed these templated variables are available for chezmoi init and not after because they are not builtin text/template functions.

I honestly think this is pretty dirty way to achieve the outcome and doing this with .chezmoiexternal is a more self-contained and robust method to achieve this given the extra dependencies that were required (I didn't necessarily need jq, but added it to support my case for anyone who prefers to always use latest instead of version pinning)

Keen for your thoughts on this now @twpayne (when you return from a well earned break!)

NOTE: Code example is a POC only, not working code - unzip can only be installed via system package manager or compiled from source, which predominantly requires sudo!

#!/bin/bash

# Define the base URL for HashiCorp releases
BASE_URL="https://releases.hashicorp.com/vlt"

# Define the target directory for `vlt`, `jq`, and `unzip`
TARGET_DIR="$HOME/.local/bin"

# Check if the target directory exists, create it if not
if [ ! -d "$TARGET_DIR" ]; then
    mkdir -p "$TARGET_DIR"
fi

# Check if `jq` is available or download it
if ! command -v jq &> /dev/null; then
    # needs better OS/arch handling
    JQ_BIN_URL="https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64"
    JQ_BIN="$TARGET_DIR/jq"

    # Use `curl` to download the `jq` binary
    curl -Lo "$JQ_BIN" "$JQ_BIN_URL"

    # Make the `jq` binary executable
    chmod +x "$JQ_BIN"
    echo "jq has been downloaded and installed to $TARGET_DIR"
fi

# Check if `unzip` is available or download it
if ! command -v unzip &> /dev/null; then
    # phony - this is an example only as no binary is available
    UNZIP_BIN_URL="https://github.com/madler/unzip/releases/download/v6.0/unzip_6.0_{{ .chezmoi.os }}_{{ .chezmoi.arch }}"
    UNZIP_BIN="$TARGET_DIR/unzip"

    # Use `curl` to download the `unzip` binary
    curl -Lo "$UNZIP_BIN" "$UNZIP_BIN_URL"

    # Make the `unzip` binary executable
    chmod +x "$UNZIP_BIN"
    echo "unzip utility has been downloaded and installed to $TARGET_DIR"
fi

# Use `curl` to fetch the page listing all versions
VERSIONS_JSON=$(curl -s "$BASE_URL/index.json")

# Add .local/bin to the PATH temporarily
export PATH="$TARGET_DIR:$PATH"

# Use `jq` to parse the JSON and extract the latest version
LATEST_VERSION=$(echo "$VERSIONS_JSON" | jq -r '.versions[0].version')

# Construct the URL for the latest release file
LATEST_FILE="vlt_${LATEST_VERSION}_{{ .chezmoi.os }}_{{ .chezmoi.arch }}.zip"
LATEST_RELEASE_URL="$BASE_URL/$LATEST_VERSION/$LATEST_FILE"

# Use `curl` to download the latest release file for `vlt`
curl -LO "$LATEST_RELEASE_URL"

# Use `unzip` to extract the 'vlt' file
unzip -o "$LATEST_FILE" -d "$TARGET_DIR" vlt

# Remove the downloaded ZIP file
rm -f "$LATEST_FILE"

echo "vlt $LATEST_VERSION has been downloaded and extracted to $TARGET_DIR"

zydou commented 11 months ago

Hello @arrrgi, since the lack of unzip binary, how about using a statically compiled 'busybox' as an alternative? The unzip tool is built-in on macOS, so we only need to focus on Linux.

arrrgi commented 10 months ago

Thanks @zydou for the suggestion, and sorry for the delay responding. Your ideas are always outside the box, much appreciated. This does solve one problem, but misses the point of Chezmoi being the one 'self-contained' tool to bootstrap your environment.

I think the init script use case you have provided is awesome and will solve a similar scenario for some people. Looking at how this incident has mutated through conversation, it probably now deserves an enhancement label to help get the Application Order tweaked to support fetching externals and then confirming they are in the users PATH.

arrrgi commented 10 months ago

@twpayne is #3343 the proposed approach to this issue? It shouldn't take me long to write up a working code example to grab the latest version of vlt once you cut a new release.

@zydou I managed to get around unzip being missing as most of the major Debian variants ship with Python, thus the following POC code removes the dependency on unzip:

python3 -c "import zipfile, os; zipfile.ZipFile('vlt_1.0.0_linux_amd64.zip').extractall(); os.chmod('vlt', 0o755)"

twpayne commented 9 months ago

@twpayne is #3343 the proposed approach to this issue? It shouldn't take me long to write up a working code example to grab the latest version of vlt once you cut a new release.

Yes, exactly. You can read how to install your password manager on chezmoi init and I just cut the chezmoi 2.42.0 release. Depending on how you install chezmoi, it may take a day or two for the new release to be available on your OS/distribution.

@arrrgi please do report back if this approach works for you.

arrrgi commented 9 months ago

Works, albeit with a couple of new issues which I'll open. ie.

vlt secrets are no longer fetched and populated in templates in first apply
unexpected call to read-source-state when running chezmoi data

Otherwise, this is fantastic and really closed that gap for installation of password managers, etc required for templates. Feel free to refer to the shell script I created https://github.com/arrrgi/dotfiles/blob/719c26d4ba0c10636d7d091e2e27338f4785f746/home/.hooks/.install-hcp-vlt.sh

Closing.

twpayne / chezmoi

Possible race condition when using .chezmoiexternal to source vlt binary needed for template functions #3269

What exactly are you trying to do?

What have you tried so far?

Where else have you checked for solutions?

Output of `chezmoi doctor`

Additional context

This script is executed before the chezmoi init step.

detect whether we are behind the China's Great Fire Wall (GFW)

save the GFW indicator file

twpayne / chezmoi

Possible race condition when using .chezmoiexternal to source vlt binary needed for template functions #3269

What exactly are you trying to do?

What have you tried so far?

Where else have you checked for solutions?

Output of chezmoi doctor

Additional context

This script is executed before the chezmoi init step.

detect whether we are behind the China's Great Fire Wall (GFW)

save the GFW indicator file

Output of `chezmoi doctor`