Frontmatter support in template source files

[zacwest]

Is your feature request related to a problem? Please describe.

Some of my data files are separated from the templates which use them; I'd like to colocate them.

Describe the solution you'd like

Allow me to specify frontmatter (using --- for YAML, +++ for TOML, there may be other standards?), for example a match config for Espanso:

# collapsed for brevity in example
greek_alphabet: [{alpha: α}, {beta: β}, {chi: χ}, {delta: δ}, {epsilon: ε}, {eta: η}, {gamma: γ}, {iota: ι}, {kappa: κ}, {lambda: λ}, {mu: μ}, {nu: ν}, {omega: ω}, {omicron: ο}, {phi: φ}, {pi: π}, {psi: ψ}, {rho: ρ}, {sigma: σ}, {tau: τ}, {theta: θ}, {upsilon: υ}, {xi: ξ}, {zeta: ζ}]
---
matches:
{{- range $trigger, $replace := .greek_alphabet }}
  - trigger: ":{{ $trigger }}"
    replace: "{{ $replace }}"
    propagate_case: true
{{- end }}

I can't really think of why one might want to render files containing frontmatter in Chezmoi, but this may necessitate a source attribute.

Describe alternatives you've considered

I can of course define most of these data types in the go template system directly but its syntax is tortured for many data types.

I've got it in a greek_alphabet.yaml file in the .chezmoidata directory, which keeps it mostly clean. I did run into an issue with trying to colocate a .chezmoidata.yaml in the same directory as the template usage -- chezmoi execute-template could find it, but chezmoi apply couldn't see the information from that file.

[halostatue]

I don't think that this would work well as currently suggested, as templates can be anything, which leads to a potential parsing problem. I can think of three possible approaches to this if it’s something we want to consider:

1. Specific directives like the chezmoi:template:… directives.

   # chezmoi:template:data-begin format=yaml
   this:
    is:
      a:
        data: path
   # chezmoi:template:data-end

2. A marked data segment like __DATA__ (Perl) or __END__ (Ruby). This would be a chezmoi:template: directive, but the data would always be at the end of the file.

3. ~Support for multiple colocated .chezmoidata.FORMAT files.~ Looks like may already be supported.

[twpayne]

Frontmatter doesn't work for the reasons that @halostatue gives. The specific directives and data segment are interesting, but I don't think that they are necessary because .chezmoidata should work for your use case.

All .chezmoidata files are merged into the same template data structure at the top level, so your .greek_alphabet variable will always be available under that name, irrespective of where the .chezmoidata file is located.

I did run into an issue with trying to colocate a .chezmoidata.yaml in the same directory as the template usage -- chezmoi execute-template could find it, but chezmoi apply couldn't see the information from that file.

What is the exact configuration where you observed this issue? I tried to replicate the issue with the following testscript, but everything passed for me, i.e. chezmoi execute-template and chezmoi apply behaved the same.

# test that chezmoi apply sees .chezmoidata files in a subdirectory
exec chezmoi apply
cmp $HOME/.dir/file golden/file

# test that chezmoi execute-template sees .chezmoidata files in a subdirectory
stdin $CHEZMOISOURCEDIR/dot_dir/file.tmpl
exec chezmoi execute-template
cmp stdout golden/file

-- golden/file --
value
-- home/user/.local/share/chezmoi/dot_dir/.chezmoidata.yaml --
key: value
-- home/user/.local/share/chezmoi/dot_dir/file.tmpl --
{{ .key }}

[halostatue]

Would this work, @twpayne?

# test that chezmoi apply sees .chezmoidata files in a subdirectory
exec chezmoi apply
cmp $HOME/.dir/file golden/file

# test that chezmoi execute-template sees .chezmoidata files in a subdirectory
stdin $CHEZMOISOURCEDIR/dot_dir/file.tmpl
exec chezmoi execute-template
cmp stdout golden/file

-- golden/file --
key value
-- home/user/.local/share/chezmoi/.chezmoidata.yaml --
old: key
-- home/user/.local/share/chezmoi/dot_dir/.chezmoidata.yaml --
key: value
-- home/user/.local/share/chezmoi/dot_dir/file.tmpl --
{{ .old }} {{ .key }}

[twpayne]

This should work. If it doesn't then it's a bug in chezmoi. Trying your test now...

[twpayne]

...and this test passes for me.

[zacwest]

I can think of another option: frontmatter_file.tmpl This would allow you to know for sure how to parse the frontmatter part without having to inspect the file.

But I totally understand your reasoning too.

Here's the error I get:

> chezmoi diff
chezmoi: template: private_dot_config/private_espanso/match/greek.yml.tmpl:6:32: executing "private_dot_config/private_espanso/match/greek.yml.tmpl" at <.greek_alphabet>: map has no entry for key "greek_alphabet"

but:

> chezmoi execute-template <private_dot_config/private_espanso/match/greek.yml.tmpl
# `propagate_case` allows e.g.:
# alpha^ => α
# ALPHA^ => Α [or Alpha^ since it's just 1 char]

matches:
  - trigger: "alpha^"
    replace: "α"
    propagate_case: true
# <snip>

Could it be the depth that I'm creating at?

Full files:

private_dot_config/private_espanso/match/greek.yml.tmpl

# `propagate_case` allows e.g.:
# alpha^ => α
# ALPHA^ => Α [or Alpha^ since it's just 1 char]

matches:
{{- range $trigger, $replace := .greek_alphabet }}
  - trigger: "{{ $trigger }}^"
    replace: "{{ $replace }}"
    propagate_case: true
{{- end }}

private_dot_config/private_espanso/match/.chezmoidata.yaml

greek_alphabet:
  alpha: α 
  beta: β
  chi: χ
  delta: δ
  epsilon: ε
  eta: η
  gamma: γ
  iota: ι
  kappa: κ
  lambda: λ
  mu: μ
  nu: ν
  omega: ω
  omicron: ο
  phi: φ
  pi: π
  psi: ψ
  rho: ρ
  sigma: σ
  tau: τ
  theta: θ
  upsilon: υ
  xi: ξ
  zeta: ζ

[twpayne]

private_dot_config/private_espanso/match/greek.yml.tmpl

I think this file needs to be renamed to .chezmoidata.yaml so chezmoi can find it as template data.

[zacwest]

Oh I got those names backwards when I pasted. Sorry. Fixed in the comment.

[bradenhilton]

It only works for me if I move the .chezmoidata.yaml file to the root of the source directory, is this correct behavior? To me, the docs imply that it can be placed anywhere in the source state.

❯ gci

    Directory: C:\Users\User\.local\share\chezmoi\home\private_dot_config\private_espanso\match

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2023-09-01     01:13            309 .chezmoidata.yaml
-a----        2023-09-01     01:17            265 greek.yml.tmpl

❯ chezmoi data
{
  ...
  "greek_alphabet": {
    "alpha": "α",
    "beta": "β",
    "chi": "χ",
    "delta": "δ",
    "epsilon": "ε",
    "eta": "η",
    "gamma": "γ",
    "iota": "ι",
    "kappa": "κ",
    "lambda": "λ",
    "mu": "μ",
    "nu": "ν",
    "omega": "ω",
    "omicron": "ο",
    "phi": "φ",
    "pi": "π",
    "psi": "ψ",
    "rho": "ρ",
    "sigma": "σ",
    "tau": "τ",
    "theta": "θ",
    "upsilon": "υ",
    "xi": "ξ",
    "zeta": "ζ"
  }
}

❯ chezmoi diff
chezmoi: template: private_dot_config/private_espanso/match/greek.yml.tmpl:6:32: executing "private_dot_config/private_espanso/match/greek.yml.tmpl" at <.greek_alphabet>: map has no entry for key "greek_alphabet"

❯ mv .\.chezmoidata.yaml ..\..\..

❯ chezmoi diff
diff --git a/.config/espanso/match/greek.yml b/.config/espanso/match/greek.yml
new file mode 100666
index 0000000000000000000000000000000000000000..cfcae37fb58ae1d82f48944bf31cee2642a52c69
--- /dev/null
+++ b/.config/espanso/match/greek.yml
@@ -0,0 +1,77 @@
+# `propagate_case` allows e.g.:
+# alpha^ => α
+# ALPHA^ => Α [or Alpha^ since it's just 1 char]
+
+matches:
+  - trigger: "alpha^"
+    replace: "α"
+    propagate_case: true
+  - trigger: "beta^"
+    replace: "β"
+    propagate_case: true
+  - trigger: "chi^"
+    replace: "χ"
+    propagate_case: true
+  - trigger: "delta^"
+    replace: "δ"
+    propagate_case: true
+  - trigger: "epsilon^"
+    replace: "ε"
+    propagate_case: true
+  - trigger: "eta^"
+    replace: "η"
+    propagate_case: true
+  - trigger: "gamma^"
+    replace: "γ"
+    propagate_case: true
+  - trigger: "iota^"
+    replace: "ι"
+    propagate_case: true
+  - trigger: "kappa^"
+    replace: "κ"
+    propagate_case: true
+  - trigger: "lambda^"
+    replace: "λ"
+    propagate_case: true
+  - trigger: "mu^"
+    replace: "μ"
+    propagate_case: true
+  - trigger: "nu^"
+    replace: "ν"
+    propagate_case: true
+  - trigger: "omega^"
+    replace: "ω"
+    propagate_case: true
+  - trigger: "omicron^"
+    replace: "ο"
+    propagate_case: true
+  - trigger: "phi^"
+    replace: "φ"
+    propagate_case: true
+  - trigger: "pi^"
+    replace: "π"
+    propagate_case: true
+  - trigger: "psi^"
+    replace: "ψ"
+    propagate_case: true
+  - trigger: "rho^"
+    replace: "ρ"
+    propagate_case: true
+  - trigger: "sigma^"
+    replace: "σ"
+    propagate_case: true
+  - trigger: "tau^"
+    replace: "τ"
+    propagate_case: true
+  - trigger: "theta^"
+    replace: "θ"
+    propagate_case: true
+  - trigger: "upsilon^"
+    replace: "υ"
+    propagate_case: true
+  - trigger: "xi^"
+    replace: "ξ"
+    propagate_case: true
+  - trigger: "zeta^"
+    replace: "ζ"
+    propagate_case: true

[twpayne]

I'm still unable to reproduce this. I tried in #3213 which reproduces the problem as closely as I could, and passes on all operating systems. What's missing from the test?

# collapsed for brevity in example
greek_alphabet: [{alpha: α}, {beta: β}, {chi: χ}, {delta: δ}, {epsilon: ε}, {eta: η}, {gamma: γ}, {iota: ι}, {kappa: κ}, {lambda: λ}, {mu: μ}, {nu: ν}, {omega: ω}, {omicron: ο}, {phi: φ}, {pi: π}, {psi: ψ}, {rho: ρ}, {sigma: σ}, {tau: τ}, {theta: θ}, {upsilon: υ}, {xi: ξ}, {zeta: ζ}]
---
matches:
{{- range $trigger, $replace := .greek_alphabet }}
  - trigger: ":{{ $trigger }}"
    replace: "{{ $replace }}"
    propagate_case: true
{{- end }}

Note that this template data isn't quite right. The greek_alphabet variable is an array of objects in the frontmatter, but treated as a single object in the template. This is fixed in the reproduction case in #3213.

Edit: fix links to PR

[twpayne]

@zacwest What is the output of chezmoi doctor on your machine?

[zacwest]

This happens on 2 of my Macs, here's the result from one of them:

> chezmoi doctor
RESULT    CHECK                MESSAGE
ok        version              v2.38.0, commit 0ce82b3a958191ee441034ee78da4b9440b51dc0, built at 2023-08-21T12:12:17Z, built by Homebrew
ok        latest-version       v2.38.0
ok        os-arch              darwin/amd64
ok        uname                Darwin C02YW094M0XV 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64 x86_64
ok        go-version           go1.21.0 (gc)
ok        executable           /usr/local/bin/chezmoi
ok        upgrade-method       replace-executable
ok        config-file          ~/.config/chezmoi/chezmoi.yaml, last modified 2023-08-23T13:07:17-07:00
warning   source-dir           ~/.local/share/chezmoi is a git working tree (dirty)
ok        suspicious-entries   no suspicious entries
warning   working-tree         ~/.local/share/chezmoi is a git working tree (dirty)
ok        dest-dir             ~ is a directory
ok        umask                022
ok        cd-command           found /usr/local/bin/fish
ok        cd-args              /usr/local/bin/fish
info      diff-command         not set
ok        edit-command         found /usr/local/bin/hx
ok        edit-args            /usr/local/bin/hx
ok        git-command          found /usr/local/bin/git, version 2.42.0
ok        merge-command        found /usr/bin/vimdiff
ok        shell-command        found /usr/local/bin/fish
ok        shell-args           /usr/local/bin/fish
ok        age-command          found /usr/local/bin/age, version 1.1.1
info      gpg-command          gpg not found in $PATH
info      pinentry-command     not set
ok        1password-command    found /usr/local/bin/op, version 2.20.0
ok        bitwarden-command    found /usr/local/bin/bw, version 2023.8.2
info      dashlane-command     dcli not found in $PATH
info      doppler-command      doppler not found in $PATH
info      gopass-command       gopass not found in $PATH
info      keepassxc-command    keepassxc-cli not found in $PATH
info      keepassxc-db         not set
info      keeper-command       keeper not found in $PATH
info      lastpass-command     lpass not found in $PATH
info      pass-command         pass not found in $PATH
info      passhole-command     ph not found in $PATH
info      rbw-command          rbw not found in $PATH
info      vault-command        vault not found in $PATH
info      vlt-command          vlt not found in $PATH
info      secret-command       not set

[bradenhilton]

This is so bizarre.

Bare source/destination directories:

Details

```console ❯ tree /f /a . Folder PATH listing Volume serial number is 000000A3 F88C:B623 C:\USERS\USER\DESKTOP\3206 +---dest \---source \---private_dot_config \---private_espanso \---match .chezmoidata.yaml greek.yml.tmpl ❯ chezmoi --source "$env:USERPROFILE\Desktop\3206\source" --destination "$env:USERPROFILE\Desktop\3206\dest" diff diff --git a/.config b/.config new file mode 40700 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 --- /dev/null +++ b/.config diff --git a/.config/espanso b/.config/espanso new file mode 40700 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 --- /dev/null +++ b/.config/espanso diff --git a/.config/espanso/match b/.config/espanso/match new file mode 40777 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 --- /dev/null +++ b/.config/espanso/match diff --git a/.config/espanso/match/greek.yml b/.config/espanso/match/greek.yml new file mode 100666 index 0000000000000000000000000000000000000000..cfcae37fb58ae1d82f48944bf31cee2642a52c69 --- /dev/null +++ b/.config/espanso/match/greek.yml @@ -0,0 +1,77 @@ +# `propagate_case` allows e.g.: +# alpha^ => α +# ALPHA^ => Α [or Alpha^ since it's just 1 char] + +matches: + - trigger: "alpha^" + replace: "α" + propagate_case: true + - trigger: "beta^" + replace: "β" + propagate_case: true + - trigger: "chi^" + replace: "χ" + propagate_case: true + - trigger: "delta^" + replace: "δ" + propagate_case: true + - trigger: "epsilon^" + replace: "ε" + propagate_case: true + - trigger: "eta^" + replace: "η" + propagate_case: true + - trigger: "gamma^" + replace: "γ" + propagate_case: true + - trigger: "iota^" + replace: "ι" + propagate_case: true + - trigger: "kappa^" + replace: "κ" + propagate_case: true + - trigger: "lambda^" + replace: "λ" + propagate_case: true + - trigger: "mu^" + replace: "μ" + propagate_case: true + - trigger: "nu^" + replace: "ν" + propagate_case: true + - trigger: "omega^" + replace: "ω" + propagate_case: true + - trigger: "omicron^" + replace: "ο" + propagate_case: true + - trigger: "phi^" + replace: "φ" + propagate_case: true + - trigger: "pi^" + replace: "π" + propagate_case: true + - trigger: "psi^" + replace: "ψ" + propagate_case: true + - trigger: "rho^" + replace: "ρ" + propagate_case: true + - trigger: "sigma^" + replace: "σ" + propagate_case: true + - trigger: "tau^" + replace: "τ" + propagate_case: true + - trigger: "theta^" + replace: "θ" + propagate_case: true + - trigger: "upsilon^" + replace: "υ" + propagate_case: true + - trigger: "xi^" + replace: "ξ" + propagate_case: true + - trigger: "zeta^" + replace: "ζ" + propagate_case: true ```

The same files but in my existing source directory:

Details

```console ❯ tree /f /a . Folder PATH listing Volume serial number is 000000B3 F88C:B623 C:\USERS\USER\.LOCAL\SHARE\CHEZMOI\HOME | ..chezmoi.toml.tmpl | .chezmoiignore | +---.chezmoiscripts | | run_onchange_99-starship-init.ps1.tmpl | | | \---windows | .run_once_before_10-enable-symlink-privilege.ps1 | .run_once_before_11-install-powershellget.ps1 | .run_once_before_12-update-psreadline.ps1 | .run_once_before_90-disable-lock-screen.ps1 | +---.chezmoitemplates | +---mpv | | | external.toml | | | input.conf | | | mpv.conf | | | | | \---script-opts | | osc.conf | | sub-select.json | | sub_select.conf | | uosc.conf | | | +---powershell | | profile.ps1 | | winget.ps1 | | | +---vscode | | settings.json | | | \---yt-dlp | external.toml | +---AppData | \---Roaming | +---Code | | \---User | | settings.json.tmpl | | | +---mpv | | | .chezmoiexternal.toml | | | input.conf.tmpl | | | mpv.conf.tmpl | | | | | \---script-opts | | osc.conf.tmpl | | sub-select.json.tmpl | | sub_select.conf.tmpl | | uosc.conf.tmpl | | | \---yt-dlp | .chezmoiexternal.toml | config | config-jpn | config-kor | config-noaudio | +---Documents | +---PowerShell | | Microsoft.PowerShell_profile.ps1.tmpl | | | \---WindowsPowerShell | Microsoft.PowerShell_profile.ps1.tmpl | +---private_dot_config | +---mpv | | | .chezmoiexternal.toml | | | input.conf.tmpl | | | mpv.conf.tmpl | | | | | \---script-opts | | osc.conf.tmpl | | sub-select.json.tmpl | | sub_select.conf.tmpl | | uosc.conf.tmpl | | | +---powershell | | Microsoft.PowerShell_profile.ps1.tmpl | | | +---private_Code | | \---User | | settings.json.tmpl | | | +---private_espanso | | \---match | | .chezmoidata.yaml | | greek.yml.tmpl | | | \---scoop | modify_config.json | +---private_Library | \---private_Application Support | \---private_Code | \---User | settings.json.tmpl | \---scoop \---persist +---mpv | \---portable_config | | .chezmoiexternal.toml | | input.conf.tmpl | | mpv.conf.tmpl | | | \---script-opts | osc.conf.tmpl | sub-select.json.tmpl | sub_select.conf.tmpl | uosc.conf.tmpl | \---mpv-git \---portable_config | .chezmoiexternal.toml | input.conf.tmpl | mpv.conf.tmpl | \---script-opts osc.conf.tmpl sub-select.json.tmpl sub_select.conf.tmpl uosc.conf.tmpl ❯ chezmoi diff chezmoi: template: private_dot_config/private_espanso/match/greek.yml.tmpl:6:32: executing "private_dot_config/private_espanso/match/greek.yml.tmpl" at <.greek_alphabet>: map has no entry for key "greek_alphabet" ```

I deleted my config prior (note also the deliberate .. prefix for my config template).

I started manually copying entries from my existing source directory to the 3206 one. When I copy my .chezmoiignore file over, it breaks:

Details

```console ❯ gc $env:USERPROFILE\.local\share\chezmoi\home\.chezmoiignore {{ if ne .chezmoi.os "darwin" -}} Library/ Library/** {{- end }} {{ if ne .chezmoi.os "linux" -}} .config/Code/ .config/Code/** {{- end }} {{ if ne .chezmoi.os "windows" -}} .chezmoiscripts/windows/ .chezmoiscripts/windows/** AppData/ AppData/** Documents/*PowerShell/ Documents/*PowerShell/** {{- else -}} .config/mpv/ .config/mpv/** .config/powershell/ .config/powershell/** {{- end }} ❯ cp $env:USERPROFILE\.local\share\chezmoi\home\.chezmoiignore $env:USERPROFILE\Desktop\3206\source\ ❯ chezmoi --source "$env:USERPROFILE\Desktop\3206\source" --destination "$env:USERPROFILE\Desktop\3206\dest" diff chezmoi: template: private_dot_config/private_espanso/match/greek.yml.tmpl:6:32: executing "private_dot_config/private_espanso/match/greek.yml.tmpl" at <.greek_alphabet>: map has no entry for key "greek_alphabet" ```

It also breaks even if .chezmoiignore is empty, as long as it's present.

[zacwest]

Yep, I just reduced it down to only erroring when .chezmoiignore is present as well.

[twpayne]

Thank you both! Indeed, adding an empty .chezmoiignore is enough to reveal the bug. I just updated #3213.

[twpayne]

Initial thoughts are that maybe the empty .chezmoiignore is changing chezmoi's default behavior for .chezmoidata.<FORMAT> files in subdirectories.

[bradenhilton]

It also breaks if .chezmoiignore is not empty though.

❯ gc .\.chezmoiignore | chezmoi execute-template
Library/
Library/**

.config/Code/
.config/Code/**

.config/mpv/
.config/mpv/**
.config/powershell/
.config/powershell/**

[twpayne]

OK, so this turned out to be a cached value not being invalidated at the right time.

What was happening was:

• chezmoi reads files in order.
• When it finds the first template, it caches the final template data.
• chezmoi reads .chezmoiignore, which is a template, so it caches the final template data.
• Later, chezmoi reads .chezmoidata.yaml in the subdirectory, but doesn't invalidate the cached final template data.
• When chezmoi executes the greek.yml template, it uses the stale final template data, which doesn't include the .chezmoidata.yaml file.

I've updated #3213 to fix this, which should resolve this issue, as you can now use a separate .chezmoidata file instead of handling frontmatter.

[twpayne]

Fixed with #3213.

[zacwest]

Thanks for the fix! I appreciate it!