Creating hook scripts in Windows

lukaszkadlubowski commented 11 months ago

What exactly are you trying to do?

First of all, many thanks for this very useful extension!

I am trying to create some "hello world" hook script, but in Windows (powershell). It is not clear to me if it is currently possible (however, duct::cmd seems to be multi-platform).

In Linux, making file executable and putting the shebang at the beginning of a file does the trick, but, as far as I understand, in Windows the file extension indicates both whether the file is executable and what kind of program should be used to run it. Here, I cannot use the file extension since the expected filename (.chezmoi_modify_manager.add_hook) is hardcoded.

Am I right that current filename approach won't work in Windows, or is there another way to achieve the intended result?

What have you tried so far?

I created minimum working example in Linux, put it in .chezmoi_modify_manager.add_hook file and made it executable:
```
#!/bin/bash

echo "This text is put in the .src.ini file"
```
and it is working as expected. It does not use three arguments passed by duct::cmd nor stdin, but I assume it is not important here yet.

I created some Windows "equivalents" (batch and Powershell syntax, respectively):

echo "This text is put in the .src.ini file"

Write-Output "This text is put in the .src.ini file"

But I had no means of setting the file extension to indicate what kind of executable it is. This is the result:

PS C:\Users\lk\.local\share\chezmoi\home> chezmoi_modify_manager --add C:\Users\lk\AppData\Roaming\GHISLER\wincmd.ini  
Adding "C:\\Users\\lk\\AppData\\Roaming\\GHISLER\\wincmd.ini"
  Existing (to chezmoi) file: "C:/Users/lk/.local/share/chezmoi/home/AppData/Roaming/GHISLER/modify_wincmd.ini.tmpl"
    Updating existing .src.ini file for "C:/Users/lk/.local/share/chezmoi/home/AppData/Roaming/GHISLER/modify_wincmd.ini.tmpl"...
    Executing hook script...
Error: %1 is not a valid Win32 application. (os error 193)

I tried making .chezmoi_modify_manager.add_hook script "executable", by invoking $env:Pathext += ";.add_hook" before using chezmoi_modify_manager in the current session. I didn't expect it to work (still no means of indicating the executable type) and indeed it doesn't. The result is the same as above.

Where else have you checked for solutions?

[x] I have read the README
[x] I have read the output of chezmoi_modify_manager --help-syntax and chezmoi_modify_manager --help-transforms (if relevant to my question).
[x] I have checked chezmoi_modify_manager --doctor and it lists some issues that need to be resolved, but it seems to be a bug related to #38 (more details below).
[x] I have looked through relevant parts of the source code.

Environment (please complete the following information):

chezmoi version: v2.40.0
chezmoi_modify_manager version: 2.1.4.

Output of `chezmoi_modify_manager --doctor`

$ chezmoi_modify_manager --doctor
RESULT    CHECK                MESSAGE
Info      version              2.1.4
Info      rustc-version        1.72.1
Info      host                 os=windows, arch=x86_64, info=Windows 10.0.19045 (Windows 10 Pro) [64-bit]
Error     has-chezmoi          chezmoi not found in PATH
Error     in-path              chezmoi_modify_manager is NOT in PATH
Error     has-ignore           chezmoi not found, can't check source directory

chezmoi doctor output not included since binary wasn't found

Error: Issues found, please rectify these for proper operation

Actually, the output indicates some errors. However, I have C:\Users\lk\AppData\Local\chezmoi_modify_manager\chezmoi_modify_manager.exe file and C:\Users\lk\AppData\Local\chezmoi_modify_manager is in PATH. chezmoi is installed using WinGet and is in PATH as well (and is working). So I assume it is a bug (#38).

VorpalBlade commented 11 months ago

Am I right that current filename approach won't work in Windows, or is there another way to achieve the intended result?

Maybe! Seems plausible at least.

I haven't used windows for years at this point. Wine does what I need. For windows my testing is limited to "does it pass cargo test in CI".

I would absolutely accept a patch for Windows to enable support if you are up for it. But what file extension should it use? Maybe someone wants a Python hook instead? What about someone making their hook in Rust and wanting a compiled .exe?

I think the patch itself will be rather simple once we figure out a sensible behaviour.

Actually, the output indicates some errors. However, I have C:\Users\lk\AppData\Local\chezmoi_modify_manager\chezmoi_modify_manager.exe file and C:\Users\lk\AppData\Local\chezmoi_modify_manager is in PATH. chezmoi is installed using WinGet and is in PATH as well (and is working). So I assume it is a bug (https://github.com/VorpalBlade/chezmoi_modify_manager/issues/38).

Yes most likely, I only thought about the windows case afterwards (as I said, I don't use it). But commit f7844ccfb429fbad21df99d0c8d4af5ca8e01ee6 (not in any release yet) should probably fix that. So if you are up for building the program yourself using cargo and reporting back in #38 if it works with that change I would appreciate it.

VorpalBlade commented 11 months ago

We could perhaps (on windows only) check for .chezmoi_modify_manager.add_hook.*, and use that one (if there is exactly one match). (Side note: There really shouldn't be a security concern here, as if someone can write to your chezmoi source state directory, you are already in deep trouble.)

lukaszkadlubowski commented 11 months ago

Thank you for very quick reply :)

But commit https://github.com/VorpalBlade/chezmoi_modify_manager/commit/f7844ccfb429fbad21df99d0c8d4af5ca8e01ee6 (not in any release yet) should probably fix that. So if you are up for building the program yourself using cargo and reporting back in https://github.com/VorpalBlade/chezmoi_modify_manager/issues/38 if it works with that change I would appreciate it.

I'll try to do it once I figure out how to do it ;) I have no prior experience with Rust, but I hope it is relatively easy.

We could perhaps (on windows only) check for .chezmoi_modify_manager.add_hook.*, and use that one (if there is exactly one match). (Side note: There really shouldn't be a security concern here, as if someone can write to your chezmoi source state directory, you are already in deep trouble.)

I think it would be the best approach (similar to what chezmoi actually does, as far as I know). So the user selects the most convenient extension for the task.

lukaszkadlubowski commented 11 months ago

I also have another loose thought (because I have no idea how difficult would be to implement it sometime in the future).

If the idea of hook script is to do some search&replace to mask some confidential data in the source state, can't it actually be approached in a way similar to modify_<config file>.tmpl?

For example, we have a file ~/example.ini

[sectionX]
password = someConfidentialPassword

we add the file and in modify_example.ini.tmpl we put the following (currently nonexistent) directive:

hide "sectionX" "password"

then we re-add the file, which results in the following example.ini.src.ini:

[sectionX]
password = XXX

This would be less verbose and platform-independent (once implemented, of course ;) )

Additionally, I'm wondering if even the existing ignore directive's behavior couldn't be extended to do the same as above:

we add the file with confidential fields replaced with XXX (new behavior) AND
we do not overwrite their real content in the target state at chezmoi apply (existing behavior).

But maybe I am missing some edge cases and oversimplifying?

VorpalBlade commented 11 months ago

That is a good idea. It is much more work though and is unlikely to happen on any sort of predictable schedule (this is purely a hobby project to satisfy my personal needs that I thought might be useful for others too). It might be a bit more work than you would expect (but is not insurmountable):

The adding code doesn't currently parse the modify_ script.
The modify script is templated by chezmoi (to resolve the path to the .src.ini file). If the file follows the standard pattern that is generated by default, this is not a big deal, just ignore the source directive and assume it is where it should be. But you are currently free to do something else there (not recommended, but not enforced either). At least I would have to detect that something weird is going on.

You may wonder why (and rightly so) I even need the source directive in the first place. Chezmoi executes the modify script from a temporary file, even if it isn't templated. So I can't use a path relative to the modify script to find the source file. And thus I need to trick chezmoi into expanding it by templates.

ignore alone isn't quite sufficient, but should in that case probably have the behaviour you suggested. We still need a hide directive to include the key (but not the value) when re-adding. (You could argue that I could just special case on keyring transform instead. But: The user could be using templating in the modify_ script to pull the secret value using chezmoi instead for example.)

Regardless of approach taken: I won't have time to look more at this until earliest mid next week due to upcoming travel.

lukaszkadlubowski commented 11 months ago

@VorpalBlade thank you for these explanations. Indeed it seems that supporting .chezmoi_modify_manager.add_hook.* filename pattern is more realistic plan. I hope I will have some time in the future to learn Rust basics and contribute to the project.

Have a good travel!

VorpalBlade commented 11 months ago

I believe this fix might work for Windows. I would appreciate it if you could test it. I don't really want to make a full release for it, and CI-wise I'm not set up to build a binary artefact except for releases currently. I have however cross compiled from Linux and attached it. It is not built with the MSVC toolchain like the releases are, but with the GNU one (since that is the only option when cross compiling).

(I did this since I believe setting up a working compiler on Windows might be a bit of a hassle.)

chezmoi_modify_manager.zip

It will look for a single file .chezmoi_modify_manager.add_hook.* on Windows. Only zero or exactly 1 match is valid.

lukaszkadlubowski commented 11 months ago

NOTE: edited to provide more useful example.

@VorpalBlade thank you very much for your work on this. Apologies for the late reply. I did some tests and the result is as follows:

If I create a single .chezmoi_modify_manager.add_hook.ps1 file or a single .chezmoi_modify_manager.add_hook.py file, the chezmoi_modify_manager --add 'path/to/file' still throws the same error:

Error: %1 is not a valid Win32 application. (os error 193)

However, once I change the file extension to a batch script (.chezmoi_modify_manager.add_hook.bat), the script is executed without errors. Below is an example in which I use a batch script that passes three of its input arguments to the python script (which is a rewrite of your example for zsh):

.chezmoi_modify_manager.add_hook.bat content:

@echo off
python "%userprofile%\.local\share\chezmoi\home\.chezmoi_modify_manager\python_hook_example.py" "%1" "%2" "%3"

python_hook_example.py file content:

# The file from the target directory will be available on STDIN.
# The data to add to the source state should be printed to STDOUT.

import sys
import re

# Currently only "ini"
type = sys.argv[1]
# Path of file as provided by the user to the command, may be a relative path
target_path = sys.argv[2]
# Path in the source state we are writing to. Will end in .src.ini for ini files.
source_data_path = sys.argv[3]

# Read data from STDIN
input_data = sys.stdin.read()

if "konversationrc" in source_data_path:
    # Filter out any set password.
    input_data = re.sub(r'Password=.*$', 'Password=PLACEHOLDER', input_data)

print(input_data)

Exemplary dotfile:

[Configuration]
Something=1
Password=pwd

Now, when I execute the command:

chezmoi_modify_manager --add "~/konversationrc"

The hook script is executed and the konversationrc.src.ini file is added with the content:

[Configuration]
Something=1
Password=PLACEHOLDER

and the proper modify_konversationrc.tmpl is created as well.

Thank you once more for making this work :)

lukaszkadlubowski commented 10 months ago

I edited previous response (added better example).

VorpalBlade commented 10 months ago

However, once I change the file extension to a batch script (.chezmoi_modify_manager.add_hook.bat), the script is executed without errors.

... That is baffling. Based on a vague memory and some googling: What is your PATHEXT (also this answer) environment variable set to? Though on modern Windows I'm not sure if that is still relevant. I haven't used Windows full time since the XP era, and hardly anything after 7...

There may or may not be something in the registry related to file associations as well, but I can not for the life of me remember any details.

EDIT: Found a second answer that was perhaps more informative.

VorpalBlade commented 10 months ago

Note for the future: The whole .chezmoi_modify_manager.add_hook mechanism is a left over from the 1.x era that had a zsh script for adding (and a python script for actual merging). It does not make much sense these days, and an approach using directives makes more sense.

At some point (not at the top of my TODO currently) I will add some suitable directives, and eventually deprecate the add_hook.

If anyone uses add_hook for anything except filtering out passwords, please speak up so I can take that into consideration. Leave a comment at issue #46 describing your use case if it doesn't work with what is described there.

lukaszkadlubowski commented 10 months ago

... That is baffling. Based on a vague memory and some googling: What is your PATHEXT (also this answer) environment variable set to? Though on modern Windows I'm not sure if that is still relevant. I haven't used Windows full time since the XP era, and hardly anything after 7...

There may or may not be something in the registry related to file associations as well, but I can not for the life of me remember any details.

I played with PATHEXT variable, by default it does not include python and powershell scripts:

.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL

I tried modified version

.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL;.PY;.PS1

but the result is the same.

I also changed default app for *.py files from text editor to python executable, but it didn't change anything.

Things like

Write-Host "Hi" | .\home\.chezmoi_modify_manager.add_hook.py "a" "b" "c"

do execute, so I suppose that the powershell treats python files as executables.

I think that editing the registry won't be a good idea in this context (installing my configuration on new Windows machine in "one click"), so I would rather stick to the "bat file calling another script from the same repo" solution as a good compromise. Additionally, in Windows scripts one does not have any shebang equivalent, so when having multiple PowerShell, python, etc. versions, bat file seems to be a good place to set the interpreter explicitly.

If you think that will suffice, I might prepare a pull request with my example (batch script calling python) added to the documentation.

Note for the future: The whole .chezmoi_modify_manager.add_hook mechanism is a left over from the 1.x era that had a zsh script for adding (and a python script for actual merging). It does not make much sense these days, and an approach using directives makes more sense.

At some point (not at the top of my TODO currently) I will add some suitable directives, and eventually deprecate the add_hook.

Sounds great!

VorpalBlade commented 10 months ago

I have no idea why it wouldn't be able to execute certain scripts then. Maybe it is something in either the Rust standard library Command or in the duct crate? But that seems rather farfetched to me. Either way, I lack the deep knowledge of Windows that I would need to dig into it further.

I of course welcome a PR describing how do this on Windows. I certainly prefer bat + python to powershell (as I would not be able to apply proper code review in that case).

A cross platform solution might make sense for people who use multiple platforms (e.g. a wrapper shell script and bat file that both exec a python script?)

lukaszkadlubowski commented 10 months ago

Hope #47 will be fine. I tested it under Windows and WSL (Ubuntu).

VorpalBlade commented 10 months ago

I believe this case is now fully solved. (With the future add:remove/add:hide covered by #46). As such I'm closing this issue.

I have made a new release including the changes. Please be aware that the built in self updater will probably not work for the binary that I uploaded to this case, so either use an older binary or download it anew from the releases page.

lukaszkadlubowski commented 10 months ago

Downloaded a new version as a chezmoi external. Everything is working fine. Thank you once more for your time!

VorpalBlade / chezmoi_modify_manager