Option to not make github API requests when it's not needed

Is your feature request related to a problem? Please describe.

When I work with my configuration files, I often like to make a quick tweak, do a chezmoi apply, see the results, tweak again, apply again, and so on.

Lately, I've started using externals, with gitHubLatestRelease to automatically upgrade a tool I'm using. But with my workflow of often calling chezmoi apply, I've been running into github's rate limiting.

I know the usual answer to issues with rate limiting seems to be "connect an API token to lift the limit", but I think it's a wrong solution in my particular case. I just find chezmoi's current behaviour slightly wrong.

Let's assume I had the following external configured:

['example/bin/chezmoi']
type = 'file'
url = 'https://github.com/twpayne/chezmoi/releases/download/{{ (gitHubLatestRelease "twpayne/chezmoi").TagName }}/chezmoi-linux-amd64'
refreshPeriod = '24h'

(I don't update chezmoi this way, this is just a hypothetical.)

Because of refreshPeriod, the download has no chance of running more often than once a day. However, every time I do chezmoi apply, the .chezmoiexternal.toml file gets evaluated, and gitHubLatestRelease is called, causing unnecessary web requests, and eventually rate limiting.

Of course, the problem is that chezmoi needs to evaluate the template functions in .chezmoiexternal.toml before it can read its contents and interpret the refreshPeriod value. But from the user perspective, this is a bit frustrating, as I feel there really should be an option to let chezmoi know that it shouldn't phone github more often than it's logically necessary.

Describe the solution you'd like

I'm going to only consider gitHub* functions in .chezmoiexternal files. I know they could be used in other templates, too, but I feel it's probably really rare, and superfluous to my problem.

I think the least problematic solution would be to add a command line flag that would make chezmoi not touch .chezmoiexternal files during a run. Like --refresh-externals never but stronger (--no-externals?).

It's probably the easiest way to flag that I, as a user, am sure that I need no externals update during this run under any circumstances.

Describe alternatives you've considered

Passing `refreshPeriod` to `gitHub*` functions as an argument

Since the problem is that template functions need to be evaluated before chezmoi can read the refreshPeriod value, perhaps it would help to make these functions aware of the value. Something like {{ (gitHubLatestRelease "twpayne/chezmoi" "24h").TagName }}.

Of course, then the problem becomes what should the function evaluate to when it doesn't make a call to github? Should they cache return values between calls? I feel like it's not ideal to ask to make these functions more complicated.

Lazy alternative to template functions for `.chezmoiexternal`

Some special syntax that would make chezmoi substitute values in the URL only after reading refreshPeriod if needed. For example:

['example/bin/chezmoi']
type = 'file'
url = 'https://github.com/twpayne/chezmoi/releases/download/$GITHUB_LATEST_RELEASE_TAG_NAME/chezmoi-linux-amd64'
refreshPeriod = '24h'

It's unfortunately inelegant compared to the template syntax.

Thank you for opening the issue and for a very well considered report! For reference, a similar issue is encountered with the gitHubKeys template function (#2253) and a "no network mode" has also been discussed previously (#2354).

tl;dr summary: caching is hard and there might already be a work-around.

A couple of questions/possible solutions:

Is there a particular reason why a GitHub token is not an option for you?
chezmoi apply takes an optional list of targets to update. The default is all targets, but you can run chezmoi apply ~/path/to/tweak to just update a single file or directory. This is both significantly faster than running chezmoi apply. chezmoi goes to some effort to do the minimum work required to get the answer you want (it's not perfect but not too bad). That said, with the current architecture chezmoi always needs to evaluate externals, so it won't actually help with the rate limiting.
Another solution is to move the call to gitHubLatestRelease to your config file template. Roughly you speaking you put data.program.version = {{ gitHubLatestRelease "repo/program" | quote }} in your config file template and then use {{ .program.version }} in your .chezmoiexternal file. This means that gitHubLatestRelease only gets called when you run chezmoi init (solving the rate limiting problem) and the change to a new version only occurs when you want (which can be a bit more secure, as you don't blindly import the latest version).

Would any of these be an option for you?

It's not that it's impossible to use in my case. But it just felt like the wrong solution to the problem, as it clearly makes no sense to evaluate gitHubLatestRelease in this particular situation in .chezmoiexternal. So I created this issue to see what may come of it.
That would be viable for me, as the offending .chezmoiexternal is off in its own directory. But from your comment, I understand that it's not actually viable in the current version?
Here I'm kind of confused. From the docs it sounds like init is used to create a new source root and config file. I already have a source root and a config file, and would not want to overwrite them. If init can be made to act in a way like "evaluate a config template and merge it with existing config", then it's not clear from the documentation.

And on the topic of using sneaky template evaluation methods to solve this problem: I already had an idea that I could use scripts and cache directory to keep my own timestamp of last run, and then gate the offending .chezmoiexternal entry with an if that compares it with output "date" "+%s". I may still end up doing it this way (for now, I just keep the version pinned).

But again, it feels like an ugly hack for something that should be relatively easy to provide by chezmoi itself. Especially since I expect my case to not really be rare. People probably use gitHub* functions in .chezmoiexternal, and those externals probably have reasonable refresh periods, that make unnecessary github requests.

Here I'm kind of confused. From the docs it sounds like init is used to create a new source root and config file. I already have a source root and a config file, and would not want to overwrite them. If init can be made to act in a way like "evaluate a config template and merge it with existing config", then it's not clear from the documentation.

@mkalinski Running chezmoi init for this purpose will not overwrite your source state. It will execute your config template and overwrite your existing config. This is expected (and encouraged) behavior. If you have local config file changes, you should put them in your config template.

twpayne / chezmoi