modify_ script to ignore matching lines

[ankurkotwal]

What exactly are you trying to do?

I use fish shell with the hydro prompt. The Hyrdo prompt adds a number of variables to the ~/.config/fish/fish_variables file. These variables don't need to be committed to the chezmoi repo.

What have you tried so far?

The lines added look like:

SETUVAR _hydro_git_2021984:master\u20223\x20
SETUVAR _hydro_git_2361545:master\u20223\x20

I wrote a modify_fish_variables script as follows:

#!/bin/sh
sed "/^SETUVAR\s*_hydro_git/d"

When I run chezmoi diff, the diff still shows that the working file has the hydro_git lines whilst my repo version doesn't.

My expectation was that due to the modify script, chezmoi diff shouldn't show any differences. It should be comparing source and destination after being filtered through the modify script, which should result in no changed lines. Is my understanding correct?

Where else have you checked for solutions?

• [✅] I have read chezmoi's user guide, and not found the answer.
• [✅] I have searched chezmoi's reference guide, and not found the answer.
• [✅] Other, please give details. Looked through the chezmoi GitHub issues.

Output of any commands you've tried with --verbose flag

$ chezmoi --verbose diff
## Shows diff using git-delta but nothing else

Output of chezmoi doctor

$ chezmoi doctor
RESULT    CHECK                MESSAGE
warning   version              v2.32.0, built at 2023-03-12T11:50:15Z
ok        latest-version       v2.32.0
ok        os-arch              linux/amd64 (Arch Linux)
ok        uname                Linux sneaky3 6.2.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 13 Mar 2023 17:02:08 +0000 x86_64 GNU/Linux
ok        go-version           go1.20.2 (gc)
ok        executable           /usr/bin/chezmoi
ok        config-file          ~/.config/chezmoi/chezmoi.toml, last modified 2022-05-20T13:32:17+10:00
warning   source-dir           ~/.local/share/chezmoi is a git working tree (dirty)
warning   suspicious-entries   ~/.local/share/chezmoi/private_dot_config/chezmoi/chezmoi.toml
warning   working-tree         ~/.local/share/chezmoi is a git working tree (dirty)
ok        dest-dir             ~ is a directory
ok        umask                022
ok        cd-command           found /usr/bin/fish
ok        cd-args              /usr/bin/fish
ok        diff-command         found /usr/bin/delta
ok        edit-command         found /usr/bin/vim
ok        edit-args            /usr/bin/vim
ok        git-command          found /usr/bin/git, version 2.40.0
ok        merge-command        found /usr/bin/vimdiff
ok        shell-command        found /usr/bin/fish
ok        shell-args           /usr/bin/fish
info      age-command          age not found in $PATH
ok        gpg-command          found /usr/bin/gpg, version 2.2.41
info      pinentry-command     not set
info      1password-command    op not found in $PATH
info      bitwarden-command    bw not found in $PATH
info      dashlane-command     dcli not found in $PATH
info      gopass-command       gopass not found in $PATH
info      keepassxc-command    keepassxc-cli not found in $PATH
info      keepassxc-db         not set
info      keeper-command       keeper not found in $PATH
info      lastpass-command     lpass not found in $PATH
info      pass-command         pass not found in $PATH
info      passhole-command     ph not found in $PATH
info      rbw-command          rbw not found in $PATH
info      vault-command        vault not found in $PATH
info      secret-command       not set

Additional context

Add any other context about the problem here.

[bradenhilton]

If Hydro adds those lines to ~/.config/fish/fish_variables, and your modify_ script removes them, you're going to get a diff result if/when Hydro re-adds the lines.

If you still need the variables on the machine, it would be better to keep them in ~/.config/fish/fish_variables, and instead use the modify_ script to add the changes that are relevant to you to the file.

As an aside, this line:

warning   suspicious-entries   ~/.local/share/chezmoi/private_dot_config/chezmoi/chezmoi.toml

means that chezmoi is managing its own config. This is strongly discouraged. We recently added checks to prevent this from happening.

If you want to manage chezmoi's config, you should instead use a config template. Remember to run chezmoi init after creating or making changes to the template to re-create the config file.

[ankurkotwal]

Thanks @bradenhilton for the quick response.

If Hydro adds those lines to ~/.config/fish/fish_variables, and your modify_ script removes them, you're going to get a diff result if/when Hydro re-adds the lines.

If you still need the variables on the machine, it would be better to keep them in ~/.config/fish/fish_variables, and instead use the modify_ script to add the changes that are relevant to you to the file.

The variables don't seem necessary and are certainly not needed across machines. I thought the modify script was a way to ignore those lines when detecting if the file was modified. What's the right way to achieve that?

As an aside, this line:

warning   suspicious-entries   ~/.local/share/chezmoi/private_dot_config/chezmoi/chezmoi.toml

means that chezmoi is managing its own config. This is strongly discouraged. We recently added checks to prevent this from happening.

If you want to manage chezmoi's config, you should instead use a config template. Remember to run chezmoi init after creating or making changes to the template to re-create the config file.

I was unaware of this and will follow these instructions. Thank you for spotting it!

[bradenhilton]

Simplifying here, but the modify_ script receives the destination state (~/.config/fish/fish_variables) via stdin, and its stdout becomes the target state (<temp paths>/fish_variables). The target state and destination state are diffed. If you are removing lines in the target state, you'll always receive diff output because it differs from the destination state.

If you simply read the destination state, add changes you care about, and output the target state instead, those variables will still be there, so there should be no diff output (unless you have added line(s)), but the variables themselves are effectively ignored.

Another potential solution could be to make fish_variables a regular file with a readonly attribute, but I'm not too familiar with this workflow.

[ankurkotwal]

Simplifying here, but the modify_ script receives the destination state (~/.config/fish/fish_variables) via stdin, and its stdout becomes the target state (<temp paths>/fish_variables). The target state and destination state are diffed. If you are removing lines in the target state, you'll always receive diff output because it differs from the destination state.

Is there a way to remove lines from the destination state before diffing against the target state? I thought that's what the modify_ scripts would do since they're designed to ignore transient state in destination state. Am I misunderstanding the feature?

If you simply read the destination state, add changes you care about, and output the target state instead, those variables will still be there, so there should be no diff output (unless you have added line(s)), but the variables themselves are effectively ignored.

How would I do this?

Another potential solution could be to make fish_variables a regular file with a readonly attribute, but I'm not too familiar with this workflow.

This isn't quite what I want. I want to ignore certain changes to the file but not all of them.

[bradenhilton]

Is there a way to remove lines from the destination state before diffing against the target state? I thought that's what the modify_ scripts would do since they're designed to ignore transient state in destination state. Am I misunderstanding the feature?

@twpayne might have the best answer for this. I'm quite active here but my expertise is limited.

How would I do this?

You might be able to adapt this example to suit your needs.

Fundamentally, the modify_ script is about adding what you need, regardless of what's already in the file.

You can use a modify_ script to remove lines, but I don't think there is a way to avoid the diffs.

@halostatue could have additional insight as a fish user, if memory serves.

[twpayne]

This isn't quite what I want. I want to ignore certain changes to the file but not all of them.

So this isn't a modify_ question, it's a "how can I ignore diffs" question.

Right now, chezmoi doesn't have a mechanism to allow you to ignore differences within a single file. chezmoi does allow you to ignore diffs in classes of file (e.g. "don't show me any diffs that come from externals" or "don't show me changes to directories") but this is not what you want.

Is there a way to remove lines from the destination state before diffing against the target state? I thought that's what the modify_ scripts would do since they're designed to ignore transient state in destination state. Am I misunderstanding the feature?

modify_ scripts allow you to define the target state as a function of the destination state.

chezmoi currently always shows you the exact changes that it will make, compared the destination state.

I suspect that the fundamental problem here is that you're using a program that modifies its own config file, i.e. it's a program that mixes state and configuration into a single file. The likely correct long-term solution is to contact the program authors and get them to separate the two. As a tl;dr, configuration is something that is specified by the user and never changed by the program and state is something that is updated by the program. These two things should never be stored in the same file.

[ankurkotwal]

Firstly, @bradenhilton and @twpayne thank you so much for your patience as I come up to speed here. I appreciate the time you've taken and the thoughtful responses.

You might be able to adapt this example to suit your needs.

That's how I started and effectively, that's what I've done with my current script. But to your point, the diffs will be visible.

So this isn't a modify_ question, it's a "how can I ignore diffs" question.

Yeah, or rather "how do I ignore certain types of changes to the file".

I suspect that the fundamental problem here is that you're using a program that modifies its own config file, i.e. it's a program that mixes state and configuration into a single file. The likely correct long-term solution is to contact the program authors and get them to separate the two. As a tl;dr, configuration is something that is specified by the user and never changed by the program and state is something that is updated by the program. These two things should never be stored in the same file.

You've nailed the issue here. I'll go back to the authors that add this code and ask them about it.

Finally, thanks for chezmoi. It makes spinning up new machines/VMs with my environment so much easier and I really appreciate your work.

[halostatue]

Fundamentally, you shouldn’t mess with the content of fish_variables, and I would argue that one shouldn’t sync it, either, because there are values which, as of right now (fish version 3.6.0), will absolutely be user-specific.

Some of these are going to be because plugins store local paths in universal variables, and if they are $HOME/… or ~/…, they are likely to be expanded in fish_variables (some of my plugins may be guilty of this, and will probably be trying to store ~). In terms of fish builtins, such as fish_user_paths, those are always stored local to the current machine. On Linux, I would find bun in /home/austin/.bun/bin, but on macOS, that would be in /Users/austin/.bun/bin.

If you’re trying to use modify_ to remove configuration _hydro_git and possibly other values, you may be messing up the operation of the Hydro prompt (I use the Tide prompt, and as it uses asynchronous operation, it uses PID-based values to store current state).

I think that a better feature request for this would be a sort of "diff filter" (similar to what can be done with git diff filters), but even that would be of limited value and have fairly complex configuration that, if it isn’t supported by the diff library used in chezmoi or the git library used in chezmoi, I wouldn’t want to implement it.

[ankurkotwal]

Fundamentally, you shouldn’t mess with the content of fish_variables, and I would argue that one shouldn’t sync it, either, because there are values which, as of right now (fish version 3.6.0), will absolutely be user-specific.

That's an interesting take. I agree its user-specific but I am the user across all my chezmoi deployments and hence I do want the environment to match across all instances. fisher stores its list of plugins in fish_variables. So I would very much like that to be replicated across all my installs. Do you have any suggestions on the "right" way to sync this state?

Some of these are going to be because plugins store local paths in universal variables, and if they are $HOME/… or ~/…, they are likely to be expanded in fish_variables (some of my plugins may be guilty of this, and will probably be trying to store ~). In terms of fish builtins, such as fish_user_paths, those are always stored local to the current machine. On Linux, I would find bun in /home/austin/.bun/bin, but on macOS, that would be in /Users/austin/.bun/bin.

Yeah this is annoying. I wish there was a way to separate machine specific variables vs general ones. I end up manually choosing what to commit. fish_variables rarely changes though, other than the hydro variables.

If you’re trying to use modify_ to remove configuration _hydro_git and possibly other values, you may be messing up the operation of the Hydro prompt (I use the Tide prompt, and as it uses asynchronous operation, it uses PID-based values to store current state).

Yeah, hence why I wanted to sync fish_variables but filter out the machine-specific state.

I think that a better feature request for this would be a sort of "diff filter" (similar to what can be done with git diff filters), but even that would be of limited value and have fairly complex configuration that, if it isn’t supported by the diff library used in chezmoi or the git library used in chezmoi, I wouldn’t want to implement it.

Yeah agreed that its what I'm looking for and also agree that it seems like more hassle that its worth (to code and then to utilise).

[halostatue]

Fundamentally, you shouldn’t mess with the content of fish_variables, and I would argue that one shouldn’t sync it, either, because there are values which, as of right now (fish version 3.6.0), will absolutely be user-specific.

That's an interesting take. I agree its user-specific but I am the user across all my chezmoi deployments and hence I do want the environment to match across all instances. fisher stores its list of plugins in fish_variables. So I would very much like that to be replicated across all my installs. Do you have any suggestions on the "right" way to sync this state?

Things are better than they used to be. There was a machine-specific identifier at the top of the Fish 2.x version of fish_variables, meaning it could not be synced at all.

I misstated above; instead of “absolutely be user-specific”, that should have been “absolutely be user-, os-, and machine-specific”.

When it comes to syncing fisher-managed plugins, that’s pretty easy to do, since it’s built into fisher as the ~/.config/fish/fish_plugins file. I have a run_after script for this, run_after_fisher-update.sh. Just chezmoi add ~/.config/fish/fisher_plugins and you’re all good (this is true up to Fisher 4.4.3, the current version).

Most of the fish plugins that I manage are well-behaved or will be better behaved soon, but I still recommend against syncing fish_variables.

Hope that helps.