Open JasonGantner opened 2 weeks ago
This is an interesting idea but there are a number of issues:
chezmoi apply
, chezmoi diff
, chezmoi status
. It's not clear how to achieve caching with an external curl
binary.curl
somehow, and only when needed.User-Agent
header as a courtesy to webmasters.Basically, the HTTP API that chezmoi needs is quite complex, and does not easily map onto replacing calls to Go's net/http
library with invocations of curl
.
Are there other workarounds you can use? For example:
run_
script to download the archive with curl
and unpack that?
- chezmoi caches HTTP responses to avoid re-downloading the same URL when it has not changed. This is critically important for performance as the contents of files retrieved by HTTP need to be inspected every time the user runs
chezmoi apply
,chezmoi diff
,chezmoi status
. It's not clear how to achieve caching with an externalcurl
binary.
Without having looked at the caching internals of chezmoi, I would guess $XDG_CACHE_HOME/chezmoi/
is a good place to store the downloaded files/responses.
To avoid re-downloading a file there's currently two solutions :
curl -z "$file" "$url"
or curl -z "$datetime" "$url"
curl --etag-compare "$etag_file" --etag-save "$etag_file" "$url"
- chezmoi adds its own OAuth2 tokens when making requests to the GitHub API. These would also need to be passed to
curl
somehow, and only when needed.
I admit I did not think outside of downloading external files.
Though passing the token is just a matter of correctly setting the header with curl -H "Authentication: Bearer $token" "$url"
; the logic of when to pass it needs to be implemented outside of the call to curl
- chezmoi adds its own
User-Agent
header as a courtesy to webmasters.
This one is the easiest to solve with curl --user-agent "$UA"
Basically, the HTTP API that chezmoi needs is quite complex, and does not easily map onto replacing calls to Go's
net/http
library with invocations ofcurl
.
The maintenance overhead of having to build each call to curl
in addition to the go implementation does seem hard to justify given the limited use case where it would apply
Are there other workarounds you can use? For example:
- Use a
run_
script to download the archive withcurl
and unpack that?
That was my first thought but at this point there's almost no benefit to using chezmoi to run the script. In my previous example, chezmoi is used to download, verify the checksum, extract and install the files with just a few lines of config.
We end up with the tofu executable in .local/bin/tofu
while the associated documentation (license, readme, ...) is put in .local/share/tofu
.
My idea was to delegate the download to curl
but let chezmoi
handle the rest of the operations
- Run a local proxy that transparently adds the authentication layer?
While this does seem like the "simple" workaround, it raises some additional security concerns.
Even if we ignore the security aspect, it can be hard to find a fitting proxy:
As a side note, spnego-proxy is written in go and has an MIT license which could be interesting to assess the cost/complexity of implementing SPNEGO proxy authentication in chezmoi.
chezmoi
calling curl
as a fallback brings too much overhead to the development to be a valid solution.
Relying exclusively on a run_
script brings almost no benefit as compared to running the script without chezmoi
Using a local proxy isn't a viable alternative when security is a concern
Would it be a viable alternative to have a "local archive(-file)" external type that checks against a path on the host instead of retrieving a remote file ?
The local file could be downloaded with a run_before_
script which also enables downloading through unsupported protocols/tools such as sftp
or svn
Thank you for the detailed further investigation!
I totally agree with your analysis and think there are several good ways to build on this, which can be pursued independently:
Would it be a viable alternative to have a "local archive(-file)" external type that checks against a path on the host instead of retrieving a remote file ?
Yes, absolutely. It should be fairly straightforward to add support for file://
URLs to chezmoi's externals, which can link to a local file. I would propose extending chezmoi's externals to allow a list of external URLs to fetch the archive from.
For example, as well as:
[".vim/autoload/plug.vim"]
type = "file"
url = "https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim"
you could write:
[".vim/autoload/plug.vim"]
type = "file"
urls = [
"file:///home/user/Downloads/plug.vim",
"https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim",
]
where chezmoi would try urls
in order and use the first one that succeeds. If both url
and urls
are specified, then, for backwards compatibility, chezmoi would try url
first, then urls
.
chezmoi
callingcurl
as a fallback brings too much overhead to the development to be a valid solution.
Let's not abandon this yet. You're right that there are ways for chezmoi to use curl
, but adding pluggable HTTP clients is more work than Way 1.
Even if we ignore the security aspect, it can be hard to find a fitting proxy:
Yup, these two projects have not been updated in while. It might be because they are abandoned. It could also because they are stable and just work as they are. I agree that integrating them into chezmoi adds a lot of complexity and I would prefer to avoid it.
Let's do Way 1 first, hold Way 2 in mind, and try to avoid Way 3.
What do you think?
Your idea of how to implement way 1 works even better than I imagined, and it can even allow using mirrors for http resources!
IMHO way 3 should not be integrating local proxies into chezmoi but adding support for more http authentication schemes.
The benefit I see from way 3 over way 2 is that it keeps chezmoi self-contained and not relying on external tools (unless desired)
On the other hand, way 2 is kinda future proof since curl is well maintained and we can expect it's CLI and API to be stable. It also adds support for some extra protocols that may be of interest (SFTP, FTP/S, SMB/S, ..., even GOPHER/S).
Regarding my specific issue, way 1 is more than enough. I'll let you be the judge for the need to implement way 2 and/or 3.
Thank you for the fast implementation, i'll test that out and come back to you.
Do you wish for me to close the issue once the tests are successful or do you want to keep it for later reference ?
I still want to implement supporting multiple URLs for externals. This is non-trivial to implement and I need to find time to work on it. So, keep this issue open for now.
Is your feature request related to a problem? Please describe.
Golang's HTTP proxy support only implements Basic authentication. Proxies that limit authentication to only digest, NTLM, or SPNEGO (kerberos) are unusable. In many cases, downgrading security is not an option.
Describe the solution you'd like
Ideally, adding support for other schemes would be the simplest from a user standpoint.
However, it means relying on multiple other go libraries for each scheme (eg. jcmturner/gokrb5 for SPNEGO, Azure/go-ntlmssp for NTLM) and leads to an increased maintainer workload and might be made impossible because of licensing issues.
Describe alternatives you've considered
The simpler workaround I thought of would be to use cURL as a fallback. This would be made possible through a configuration variable similarly to git/age fallbacks.
Additional context
Behaviour difference between
chezmoi
andcurl
with correct proxy definition in env vars :