Introduce side-effects

[yannham]

The core language of Nickel is pure, but its usage in practice requires side-effects, e.g. for retrieving information about the local environment. We propose to add effects to Nickel.

The effect system must be:

• Extensible: users may provide new effects as external handlers
• Commutative: effects may be executed in any order without changing the semantics of the program, for parallelization

There are several possible directions to incorporate effects.

Internal

The first one is to provide effects inside the language through primitives, which are handled in a specific way by the interpreter. Primitives are akin to regular functions, as it is done in most general purpose languages, for example as in OCaml let availableMACs = map getInterfaceMAC (getNetworkInterfaces ()) in foo. Effects themselves would be implemented outside of the boundaries of the Nickel program, either directly by the interpreter or by an external handler. They may or may not be sequenced or tracked by the type system as in Haskell.

Tracking and composition

The point of requiring commutativity is to avoid undue sequentialization, which prevents parallel execution. Some effects may still need to be executed in a specific order, as in the previous example, but because they have a data dependency, not a temporal one. Requiring all effects to be linearly sequentialized from the toplevel, as the IO monad of Haskell, defeats the purpose of commutativity. One can still consider a monadic interface, but it must be tailored for commutativity, meaning it should be able to easily express independent computations. Since the core language is untyped, it makes less sense to track effects through types if these are not enforced nor visible in the signature of functions. Some form of effect tracking may be required later for incremental evaluation, though.

To sum up, with internal effects:

• Effects are performed at arbitrary places, and thus can be composed
• Effects may or may not be hidden and/or sequentialized, depending on the precise design

External

In the internal case effects may be hidden deep inside a program, stripping away the benefits of purity and hermeticity. A common approach to mitigate the side effects of side effects is to downright push the responsibility of executing them at the boundary of the program, and replace primitive calls with pure, top-level arguments. They can be then accessed like any other binding.

That is, our previous example becomes a top-level function fun availableMACs => foo. If there are other effects inside foo, they must also be hoisted as top-level arguments. Then, values can be directly passed on the command line as external variables in Jsonnet. In CUE, a dedicated external level, the scripting layer, is allowed to perform commands and pass the values to pure CUE code. Similarly, repository rules in Bazel are responsible for fetching environment specific data at the loading phase, while following operations are required to be hermetic.

To sum up, with external effects:

• All effects happen before the evaluation of the program, thus cannot be composed
• The program itself stays pure

Proposal: implicit and internal effects

External effects entail a satisfying separation of concerns, as well as keeping the program itself pure. But the inability to compose effects is limiting. One of the motivating use cases for Nickel is a dynamic configuration problem (Terraform) where some information, say an IP address, is not available until another part of the configuration is executed, resulting in the deployment a machine. This cannot be expressed in the external setting.

The simplest choice is to make performing and interleaving effects implicit, à la ML. Indeed, the value of a monadic interface in presence of commutative effects and untyped code is not clear. In this setting, an effectful primitive is not much different from a regular function from the user's point of view. Performing an effect is similar to a system call in C: the program is paused, the handler for the primitive is executed, and the program is resumed with the result of the call. Extensibility is simple, as executing an external handler boils down to a remote procedure call.

Example:

/* ... */
let availableMACs = map getInterfaceMAC (getNetworkInterfaces ()) in
let home = getEnv "HOME" in
foo

Remarks

• Syntax: one may want to make performing effects syntactically visible, either using name conventions or namespaces for effectful functions, or by using an explicit keyword, such as perform someEffect someArg (see Eff).

[edolstra]

I agree that implicit effects are probably better for our use cases. One downside is that caching of evaluation results between runs becomes harder since you'd have to keep track of what values are affected by what effects. But keeping track of such dependencies is probably needed for incremental re-evaluation anyway.

[hanshoglund]

I do not think this should be added to Nickel.

Side effects prevent meaningful reasoning about code and prevents advanced optimization strategies (including caching/reuse). These are well-known drawbacks.

Explicit effects systems (free monads, handlers) are popular, but in my view over-hyped. While writing custom handlers/semantics is cool, code using such effects share the same fundamental deficiencies as other non-pure code.

Maybe I'm misunderstanding what is being proposed. Maybe effects should be a strictly opt-in feature for embedders, meaning Nix can do what makes sense there. I can see an extremely limited set of implicit effects making sense for Nix, namely 1) debug logging and 2) non-recoverable error/panic. Anything else I think would just exacerbate the current problem of "lots of code that is hard to reason about".

As for environment reading we can already express this with lambdas/functions.

[thufschmitt]

I can see an extremely limited set of implicit effects making sense for Nix

That's a bit "hidden" currently in Nix as the language is tied to the rest of the system, but I think builtins.derivation and all the builtins.fetch* functions should be considered as side-effects.

[hanshoglund]

With the Flakes concept it is feasible to separate import resolution/fetching completely from evaluation, as in Dhall.

As for builtins.derivation this is pure, except for magic around paths. It could be made completely pure by restricting paths point to to content-addressed/immutable values.

In other words, we may need some implicit effects to model how Nix currently behaves, but I believe the long-term direction should be to make evaluation completely pure/hermetic (and the current state is pretty close to that already).

Again, for Nickel this probably just means "primitive functions can have (commutative) effects, at the discretion of the embedder". @yannham is this the intention?

@edolstra What are your thoughts on the above?

[hanshoglund]

Update: I can see this have already been discussed here.

I would argue for commutative and idempotent implicit effects solely for the purpose of compatibility with current Nixpkgs. Any other use-case is speculative at best and should not be a reason to throw away purity.

[MagicRB]

So the current plan is a totally pure language with no internal effects? I'm asking because I want to take a shot at a readFile getEnv interface and without internal effects this can only be represented as a Monad afaik. (External effects would disallow composition and make the API extremely cumbersome.)

[yannham]

@MagicRB I wouldn't say so. Effects have been on hold for now, as there have been other fundamental aspects of the language to discuss, design and implement. We'll probably come back to effects at some point, but internal effects are not out of question at this point. It's true that, for now, external effects are the only practical way to inject environment-dependent data in a Nickel configuration.

[MagicRB]

Thanks for the info, I'll try to throw something together with monads or smth, we'll see how badly it ends up working.

[a12l]

Has there been any additionell discussions about this since the release of 1.0?

[yannham]

@a12l we haven't brought this up recently. One of the original motivation was in part to potentially emulate string context in Nix, but we've introduced the much lighter symbolic strings for that. We'll still eventually need effects if we want Nickel to be able to drive any kind of Nix application, but I think nobody has an immediate urging needs for an effect system, which is why this isn't set as priority. Do you have a use-case in mind which would require such a thing?

[a12l]

Do you have a use-case in mind which would require such a thing?

First, sorry for the very late reply! I didn't see that I'd received an answer.

If I remember correctly my initial though while writing my question was to use Nickel to go out to github and take the latest checksum of a repo, and then include that into a JSON object. The thought would have been to create my own custom flake.lock file for my non-flake based Nix config.

But then I decided to just go with a YSH [1] shell script.

[1] https://www.oilshell.org/