chezmoi always tell me file changed

[cathaysia]

Describe the bug

when I run chezmoi apply, it always tell me files had changed.

To reproduce

1. set a externl dir:

["Library/Rime"]
type = "archive"
url = "https://github.com/iDvel/rime-ice/archive/refs/heads/main.zip"
exact = true
stripComponents = 1
refreshPeriod = "720h"

2. manage a file in Library/Rime:

├── private_Library
│   └── rime
│       ├── default.custom.yaml
│       └── squirrel.custom.yaml

3. run chezmoi apply twice.

output of chezmoi applay -v

```console diff --git a/Library/Rime/default.custom.yaml b/Library/Rime/default.custom.yaml deleted file mode 100644 index 43a9c5d4d4b81d6585519a22bc14775c39144814..0000000000000000000000000000000000000000 --- a/Library/Rime/default.custom.yaml +++ /dev/null @@ -1,3 +0,0 @@ -patch: - menu: - page_size: 7 diff --git a/Library/Rime/default.custom.yaml b/Library/Rime/default.custom.yaml deleted file mode 100644 index 43a9c5d4d4b81d6585519a22bc14775c39144814..0000000000000000000000000000000000000000 --- a/Library/Rime/default.custom.yaml +++ /dev/null @@ -1,3 +0,0 @@ -patch: - menu: - page_size: 7 diff --git a/Library/Rime/default.custom.yaml b/Library/Rime/default.custom.yaml deleted file mode 100644 index 43a9c5d4d4b81d6585519a22bc14775c39144814..0000000000000000000000000000000000000000 --- a/Library/Rime/default.custom.yaml +++ /dev/null @@ -1,3 +0,0 @@ -patch: - menu: - page_size: 7 ```

Output of chezmoi doctor

```console $ chezmoi doctor RESULT CHECK MESSAGE warning version v2.33.6, built by nixpkgs warning latest-version v2.34.2 ok os-arch darwin/arm64 ok uname Darwin U-7WCVP4DN-0001.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:01:02 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T8112 arm64 ok go-version go1.20.4 (gc) ok executable ~/.nix-profile/bin/chezmoi ok upgrade-method replace-executable ok config-file no config file found ok source-dir ~/.local/share/chezmoi is a git working tree (clean) ok suspicious-entries no suspicious entries ok working-tree ~/.local/share/chezmoi is a git working tree (clean) ok dest-dir ~ is a directory ok umask 022 ok cd-command found /bin/zsh ok cd-args /bin/zsh info diff-command not set ok edit-command found ~/.nix-profile/bin/nvim ok edit-args ~/.nix-profile/bin/nvim ok git-command found ~/.nix-profile/bin/git, version 2.40.1 ok merge-command found /usr/bin/vimdiff ok shell-command found /bin/zsh ok shell-args /bin/zsh info age-command age not found in $PATH ok gpg-command found /opt/homebrew/bin/gpg, version 2.4.2 info pinentry-command not set info 1password-command op not found in $PATH info bitwarden-command bw not found in $PATH info dashlane-command dcli not found in $PATH info gopass-command gopass not found in $PATH info keepassxc-command keepassxc-cli not found in $PATH info keepassxc-db not set info keeper-command keeper not found in $PATH info lastpass-command lpass not found in $PATH info pass-command pass not found in $PATH info passhole-command ph not found in $PATH info rbw-command rbw not found in $PATH info vault-command vault not found in $PATH info secret-command not set ```

[twpayne]

I tried to reproduce this in 043ce0ae3aae11d57faa13260a9ed9059526ae17 but did not get any error.

Is it possible that rime-ice is modifying its own config files?

[halostatue]

This is expected and unavoidable with this configuration as chezmoi is expecting the state of an entity to be either one thing (an external) or another (managed by chezmoi), not both. In this case, Library/Rime is both an external and contains managed files, so chezmoi cannot ensure the state of either one.

The solution is to use after_ scripts to create the managed files after everything else is handled. See my answer on #3062.

[cathaysia]

How difficult is it for chezmoi to support this feature? This feature should be a reasonable request. I can contribute to this if I possible, but timing is not guaranteed. because I've been busy with other things recently.

[halostatue]

@twpayne can answer that question much better than I can, but the sense that I get is that it’s "non-trivial". I know that he’s been experimenting with using sqlite as the state management engine instead of boltdb, but that is an ongoing exploration and may not enable such "overlay" capabilities.

If I were to hazard a difficulty estimate, it would be Fairly Hard. I believe that chezmoi’s execution order is basically:

1. before scripts
2. regular files and templates
3. externals
4. after scripts

This would require an additional step:

1. before scripts
2. regular files and templates
3. externals
4. external overlays
5. after scripts

But the "external overlays" complicates things like chezmoi diff, chezmoi merge, etc., because their status can’t be determined until the external is applied. And externals are, well, external. It means that chezmoi marks the destination (whether it’s a file or folder) as something whose contents it cannot predict—with reason.

If Rime decided to start shipping default.custom.yaml and squirrel.custom.yaml files in what gets unpacked into Library/Rime…then chezmoi is left with always seeing things as changed and you may end up with data loss because of the way that the configuration is applied.

There might be a clever way to solve this that I haven’t seen, but it feels like a lot of risk for a fairly niche feature (as I said; I don’t currently use externals at all).

[twpayne]

chezmoi already supports this: https://www.chezmoi.io/user-guide/manage-different-types-of-file/#manage-part-but-not-all-of-a-file

[twpayne]

Also: https://www.chezmoi.io/user-guide/manage-different-types-of-file/#handle-configuration-files-which-are-externally-modified

[cathaysia]

I tried to reproduce this in 043ce0a but did not get any error.

Is it possible that rime-ice is modifying its own config files?

The root problem is that some files(default.custom.yaml and squirrel.custom.yaml) managed by chezmoi are located in the external unpacked path.

rime-ice will not change it. Just as you see, this repo is a configuration dir. Software never change these files.

[bradenhilton]

@twpayne Small nit, and apologies if this wasn't the case at time of writing, but I believe your test in https://github.com/twpayne/chezmoi/commit/043ce0ae3aae11d57faa13260a9ed9059526ae17 passes because it does not include the managed file(s) within Library/Rime:

Output

```text # test that chezmoi apply does not say that a file in an external has changed when it has not exec chezmoi apply exec chezmoi apply -v ! stdout . -- home/user/.local/share/chezmoi/.chezmoiexternal.toml -- ["Library/Rime"] type = "archive" url = "https://github.com/iDvel/rime-ice/archive/refs/heads/main.zip" exact = true stripComponents = 1 refreshPeriod = "720h" -- home/user/.local/share/chezmoi/private_Library/Rime/default.custom.yaml -- patch: menu: page_size: 7 ``` ```console ❯ go test ./pkg/cmd/... -run=TestScript/^issue3065$ --- FAIL: TestScript (0.00s) --- FAIL: TestScript/issue3065 (4.48s) testscript.go:534: # test that chezmoi apply does not say that a file in an external has changed when it has not (4.460s) > exec chezmoi apply [stderr] chezmoi: Library/Rime: inconsistent state (~/AppData/Local/Temp/go-test-script4016706903/script-issue3065/home/user/.local/share/chezmoi/private_Library/Rime, https://github.com/iDvel/rime-ice/archive/refs/heads/main.zip defined in ~/AppData/Local/Temp/go-test-script4016706903/script-issue3065/home/user/.local/share/chezmoi/.chezmoiexternal.toml) [exit status 1] FAIL: testdata\scripts\issue3065.txtar:2: unexpected command failure FAIL FAIL github.com/twpayne/chezmoi/v2/pkg/cmd 5.064s FAIL ```

I think this just confirms what @halostatue said in https://github.com/twpayne/chezmoi/issues/3065#issuecomment-1606200600.

[cathaysia]

@twpayne can answer that question much better than I can, but the sense that I get is that it’s "non-trivial". I know that he’s been experimenting with using sqlite as the state management engine instead of boltdb, but that is an ongoing exploration and may not enable such "overlay" capabilities.

If I were to hazard a difficulty estimate, it would be Fairly Hard. I believe that chezmoi’s execution order is basically:

1. before scripts
2. regular files and templates
3. externals
4. after scripts

This would require an additional step:

1. before scripts
2. regular files and templates
3. externals
4. external overlays
5. after scripts

But the "external overlays" complicates things like chezmoi diff, chezmoi merge, etc., because their status can’t be determined until the external is applied. And externals are, well, external. It means that chezmoi marks the destination (whether it’s a file or folder) as something whose contents it cannot predict—with reason.

If Rime decided to start shipping default.custom.yaml and squirrel.custom.yaml files in what gets unpacked into Library/Rime…then chezmoi is left with always seeing things as changed and you may end up with data loss because of the way that the configuration is applied.

There might be a clever way to solve this that I haven’t seen, but it feels like a lot of risk for a fairly niche feature (as I said; I don’t currently use externals at all).

thanks for your replay above (https://github.com/twpayne/chezmoi/issues/3065#issuecomment-1606200600), I will consider it.

chezmoi only needs to compare whether the external has changed with the remote time, but not the local one. The root cause of this issue is that the external decompressed path Rime and the path rime where the *custom.yaml file is located are the same path, and chezmoi treats it as two folders on a case-insensitive file system and mistakenly Doing a diff on some files caused the problem. If it is on Linux, this situation (the external decompressed file and the chezmoi manage file are in the same path) will cause an error:

chezmoi: Library/Rime: inconsistent state (~/.local/share/chezmoi/private_Library/Rime, https://github.com/iDvel/rime-ice/archive/refs/heads/main.zip defined in ~/.local/share/chezmoi/.chezmoiexternal.toml)

[cathaysia]

chezmoi already supports this: https://www.chezmoi.io/user-guide/manage-different-types-of-file/#manage-part-but-not-all-of-a-file

Also: https://www.chezmoi.io/user-guide/manage-different-types-of-file/#handle-configuration-files-which-are-externally-modified

thank you for your reply. Sorry for my unclear description (maybe because of Google Translate?), but this issue should not be related to the two links you listed.

[cathaysia]

@halostatue hi, I find this just now!!!

it seems chezmoi already support this feature. just use external and managed files with:

-- home/user/.local/share/chezmoi/.chezmoiexternal.toml --
["Library/Rime"]
    type = "archive"
    url = "https://github.com/iDvel/rime-ice/archive/refs/heads/main.zip"
    exact = true
    stripComponents = 1
    refreshPeriod = "720h"
-- home/user/.local/share/chezmoi/private_Library/extract_Rime/default.custom.yaml --
patch:
  menu:
    page_size: 7

now it all be right. :)

But as I said above, chezmoi seems to have some bugs on case-insensitive operating systems. If you don't need to pay attention to this problem, please close this issue.

[halostatue]

I’m glad to hear that seems to work.

[twpayne]

But as I said above, chezmoi seems to have some bugs on case-insensitive operating systems.

chezmoi realistically cannot solve this. File systems can be case-sensitive or case-insensitive, and also case-preserving or case-non-preserving, and there is no way to know without using the filesystem itself. For example, APFS is case-insensitive and case-preserving, but can be optionally case-sensitive and case-preserving. Furthermore, with mount points, different subdirectories can behave in different ways. Then there's the madness that is paths on Windows, where multiple significantly-different strings can resolve to the same file, and there are weird magic filenames that you can't use for backwards-compatibility with DOS in 1980.

[twpayne]

Thank you all for your contributions here. Much appreciated. Hopefully this is now resolved. Please re-open if needed.