Deploy automatically .chezmoidata.yaml then use variables in other templates files in when calling chezmoi init

max13fr commented 6 months ago

What exactly are you trying to do?

I would like to deploy automatically the dot_chezmoidata.yaml file and use the variables inside in other templates files.

dot_local/share/chezmoi/dot_chezmoidata.yaml :

bash_aliases:
  - name: backend
    server: myhost
    path: /srv/backend
  - name: intranet
    server: myhost
    path: /srv/www/intranet
  - name: extranet
    server: myhost
    path: /srv/www/extranet

dot_bash_aliases.tmpl :

{{ range .bash_aliases }}
alias {{ .name }}="ssh -t {{ .server }} 'tmux new-session -A -s {{ .name }} -c {{ .path }}'"
{{- end }}

When I run chezmoi init, I got the following error :

info found version 2.48.0 for latest/linux/amd64
info installed ./bin/chezmoi
Cloning into '/root/.local/share/chezmoi'...
remote: Enumerating objects: 331, done.
remote: Counting objects: 100% (156/156), done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 331 (delta 70), reused 0 (delta 0), pack-reused 175
Receiving objects: 100% (331/331), 62.81 KiB | 2.09 MiB/s, done.
Resolving deltas: 100% (161/161), done.
chezmoi: .bash_aliases: template: dot_bash_aliases.tmpl:4:9: executing "dot_bash_aliases.tmpl" at <.bash_aliases>: map has no entry for key "bash_aliases"

The template file dot_bash_aliases.tmpl doesn't find the variable .bash_aliases.

What have you tried so far?

As dot_bash_aliases.tmpl is deployed before dot_local (chezmoi deploys files sorted by name), I tried to add a before script to force deploy of dot_chezmoidata.yaml before deploying other files (and so dot_bash_aliases.tmpl).

run_before_10_copy_chezmoidata.sh :

#!/bin/bash

# copy chezmoidata (vars) before all other files copy to allow templates files to load vars (like dot_bash_aliases)

SOURCE_DIR="$HOME/.local/share/chezmoi/dot_local/share/chezmoi"
DESTINATION_DIR="$HOME/.local/share/chezmoi"

mkdir -p $DESTINATION_DIR
cp "$SOURCE_DIR/dot_chezmoidata.yaml" "$DESTINATION_DIR/.chezmoidata.yaml"

echo "Copy .chezmoidata to get all vars before running templates"

It fixes the issue of the deployement order but chezmoi init doesn't load the variables from .chezmoidata.yaml (I assume that chezmoi already tried to load .chezmoidata before running that script and didn't find any file, so the variable is not available) :

info found version 2.48.0 for latest/linux/amd64
info installed ./bin/chezmoi
Cloning into '/root/.local/share/chezmoi'...
remote: Enumerating objects: 331, done.
remote: Counting objects: 100% (156/156), done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 331 (delta 70), reused 0 (delta 0), pack-reused 175
Receiving objects: 100% (331/331), 62.81 KiB | 2.09 MiB/s, done.
Resolving deltas: 100% (161/161), done.
Copy .chezmoidata to get all vars before running templates
chezmoi: .bash_aliases: template: dot_bash_aliases.tmpl:4:9: executing "dot_bash_aliases.tmpl" at <.bash_aliases>: map has no entry for key "bash_aliases"

Now, if I run chezmoi update after the chezmoi init, it's now working as .chezmoidata.yaml already existing before running that command :

Already up to date.
Copy .chezmoidata to get all vars before running templates
Cloning into '/root/.vim/bundle/Vundle.vim'...
remote: Enumerating objects: 3160, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 3160 (delta 3), reused 8 (delta 0), pack-reused 3145
Receiving objects: 100% (3160/3160), 944.82 KiB | 7.62 MiB/s, done.
Resolving deltas: 100% (1111/1111), done.

And now, .bash_aliases is available & correct.

Where else have you checked for solutions?

[x] I have read chezmoi's user guide, and not found the answer.
[x] I have searched chezmoi's reference guide, and not found the answer.
[x] Other, please give details : lot of search in issues

Output of `chezmoi doctor`

```console $ chezmoi doctor RESULT CHECK MESSAGE ok version v2.48.0, commit c758a1c57fb6084631736a7dda5ac0aaeeffa946, built at 2024-04-26T20:38:57Z, built by goreleaser ok latest-version v2.48.0 ok os-arch linux/amd64 (Debian GNU/Linux 12 (bookworm)) ok uname Linux myhost 6.1.0-20-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux ok go-version go1.22.2 (gc) ok executable ~/bin/chezmoi ok upgrade-method replace-executable ok config-file no config file found warning source-dir ~/.local/share/chezmoi is a git working tree (dirty) ok suspicious-entries no suspicious entries warning working-tree ~/.local/share/chezmoi is a git working tree (dirty) ok dest-dir ~ is a directory ok umask 022 ok cd-command found /bin/bash ok cd-args /bin/bash info diff-command not set ok edit-command found /usr/bin/vim ok edit-args /usr/bin/vim ok git-command found /usr/bin/git, version 2.39.2 ok merge-command found /usr/bin/vimdiff ok shell-command found /bin/bash ok shell-args /bin/bash info age-command age not found in $PATH ok gpg-command found /usr/bin/gpg, version 2.2.40 info pinentry-command not set info 1password-command op not found in $PATH info bitwarden-command bw not found in $PATH info bitwarden-secrets-command bws not found in $PATH info dashlane-command dcli not found in $PATH info doppler-command doppler not found in $PATH info gopass-command gopass not found in $PATH info keepassxc-command keepassxc-cli not found in $PATH info keepassxc-db not set info keeper-command keeper not found in $PATH info lastpass-command lpass not found in $PATH info pass-command pass not found in $PATH info passhole-command ph not found in $PATH info rbw-command rbw not found in $PATH info vault-command vault not found in $PATH info vlt-command vlt not found in $PATH info secret-command not set ```

Additional context

Is there any solution to predeploy some chezmoi config files in a cleaner way, or perhaps a specific flag on chezmoi init to force to reload the configs files before deploying templates files ?

Thanks in advance, Best regards, Max

PS : a great thanks for your tools, it's really good & useful !

twpayne commented 6 months ago

Thank you for the very complete problem report and your kind words!

The underlying problem here is that you're managing one of chezmoi's config files (~/.local/share/chezmoi/.chezmoidata.yaml) with chezmoi itself. This leads to the problem that chezmoi reads is config before making changes and so the changes to the config are only visible after chezmoi has run.

The fix is to use chezmoi's config file template mechanism.

Specifically, I'd recommend adding your bash aliases to the data section in ~/.local/share/chezmoi/.chezmoi.yaml.tmpl:

data:
  bash_aliases:
    - name: backend
      server: myhost
      path: /srv/backend
    - name: intranet
      server: myhost
      path: /srv/www/intranet
    - name: extranet
      server: myhost
      path: /srv/www/extranet

This will make the .bash_aliases template variable available in all your templates. You can remove the dot_chezmoidata.yaml file.

max13fr commented 6 months ago

Thanks for your reply.

The issue when using ~/.local/share/chezmoi/.chezmoi.yaml.tmpl is that it's work only when calling first chezmoi init but not with later refresh with chezmoi update.

But you give me another idea : as git init is cloning all files in ~/.local/share/chezmoi/ why not just create .chezmoidata.yaml at the root of my repository (instead of dot_local/share/chezmoi/dot_chezmoidata.yaml).

Here everything working fine, ~/.local/share/chezmoi/chezmoidata.yaml already exists when chezmoi init is deploy files (and so variables are correctly loaded).

Thanks for you help and your awesome work !

halostatue commented 6 months ago

This might be worth adding as a post to Show & Tell.

twpayne / chezmoi