Do not (observably) reexecute any code in repository rules, ever

[dslomov]

Current behavior of repository rules, documented here: https://docs.bazel.build/versions/master/skylark/repository_rules.html#when-is-the-implementation-function-executed:

The implementation function can be restarted if a dependency it request is missing. The beginning of the implementation function will be re-executed after the dependency has been resolved.

causes pretty bad unexpected behaviors, evidenced, for example, in this patch: https://github.com/bazelbuild/bazel/commit/ada0c370c2a3ea484925bc4e7d927b015bc4ad92

We must make sure that repository rule functions never observably re-execute. Otherwise the semantics of repository rules is magic, extremely hard to debug and reason about. It pains me greatly that we are shipping something this broken.

One potential way to fix it is to implement restarts from arbitrary execution point in Skylark interpreter (continuation save/restore).

[dslomov]

2700 is one bug caused by this.

[aehlig]

Another related feature request is partial redoing of a repository rule if only parts of the dependencies changes. The use case (#3395) is changing the patches/programmatic transformations to an http_archive; in this case, we need to obtain the differently patched external repository without downloading the (unchanged) archive again. (Currently, this can be mitigated with --experimental_repository_cache, but an out-of-the-box solution would be preferable.)

One possible way to obtain this would be to split the repository rule in to

• a pure fetch rule (cachable by predicted hash), and
• a purely computational rule (depending on the fetch result) that takes a directory, some files and some strings and returns a modified directory.

[jmillikin-stripe]

Splitting repository rules into (fetch, modify_dir) steps would make it difficult to fetch different artifacts based on system state, as used when (for example) fetching different binary packages based on local os/arch/libc.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[fmeum]

@Wyverald is working on this