Implicit parameters

[x] Proposed
[ ] Prototype: Not Started
[ ] Implementation: Not Started
[ ] Specification: Not Started

Summary

In this proposal, I would like to draw your attention to a feature called implicit parameters. My main inspiration comes from Scala, where it is known under this exact name, but some variations are present in other languages as well. The motivation is to increase clarity and comfort of writing and refactoring code that extensively passes environmental values through the stack.

Implicit parameters are syntactic sugar for function applications. The idea is to pass selected arguments to methods without necessarily mentioning them across the code, especially where it would be repetitive and unavoidable. Thus, instead of writing

bool f(Session session);
void g(int x, Session session);

void test(Session session) {
    if(f(session)) {
        g(21, session);
    } else {
        g(37, session);
    }
}

One could simplify it to something like:

bool f()(implicit Session session);
void g(int x)(implicit Session session);

void test(implicit Session session) {
    if(f()) {
        g(21);
    } else {
        g(37);
    }
}

Note that session is provided implicitly to every call that declares it as its implicit argument. While it still needs to be declared in the function signature, the application is handled automatically as long as there is a matching implicit value in the context.

Motivation

Since it is just a "smart" syntactic sugar, this feature does not provide any new control flows or semantics that were not achievable before. What it offers is that it lets one write code in a certain style more conveniently, and with less of boilerplate.

The promoted paradigm is to handle environment and state by passing it through stack, instead of keeping them in global variables. There are numerous benefits of designing applications this way; most notably the ease of parallelization, test isolation, environment mocking, and broader control over method dependencies and side effects. This can play crucial role in big systems handling numerous tasks in parallel, where context separation is an important security and sanity factor.

The previous snippet served as a brief visualization, so it does not necessarily capture the whole point. To get a more elaborate view, let us consider a more realistic example of a gRPC server converting image files:

void CheckCancellation()(implicit ServerCallContext ctx) => ctx.CancellationToken.ThrowIfCancelled();
bool FileCached(string fileName)(implicit ServerCallContext ctx) =>
    ctx.RequestHeaders.Get("use_cache") && Cache.Exists(fileName);

async Task ConvertJpgToPng(int fileSize, string fileName)
                          (implicit IAsyncStreamReader<Req> inStream,
                                    IServerStreamWriter<Res> outStream,
                                    ServerCallContext ctx) {
    bool cached = FileCached(fileName);

    Jpg jpg = null;
    if(cached) {
        await RequestNoFile();
        jpg = Cache.Get(fileName);
    } else {
        jpg = await RequestFile();
    }
    CheckCancellation();

    Png png = jpg.ToPng();

    await outStream.WriteAsync(new Res(){Png = png});
}

async Task RequestNoFile()(implicit IServerStreamWriter<Res> outStream) =>
    await outStream.WriteAsync(new Res(){SendFile = false});

async Task<Jpg> RequestFile()(implicit IAsyncStreamReader<Req> inStream,
                                       IServerStreamWriter<Res> outStream,
                                       ServerCallContext ctx) {
    await outStream.WriteAsync(new Res(){SendFile = true });
    CheckCancellation();
    Req msg = await inStream.ReadAsync();
    CheckCancellation();
    return msg.Png;
}

It is surely a matter of subjective opinion, but to me the snippet seems much cleaner than what I imagine it would look if it passed around ctx, inStream and outStream explicitly every time. The code focuses on the main logic without bringing up the contextual dependencies, which are mentioned only in method headers.

A nice addition is that implicits can make refactoring easier in some cases. Let us imagine that it turns out that RequestNoFile needs to check for cancellation, and therefore requires ServerCallContext to get access to the token:

async Task RequestNoFile()(implicit IServerStreamWriter<Res> outStream, ServerCallContext) {
    await outStream.WriteAsync(new Res(){SendFile = false});
    CheckCancellation();
}

Because in the presented snippet RequestNoFile is called only from scopes with ServerCallContext provided, no other changes in the code are required. In contrast, without implicit parameters, every single call to RequestNoFile would have to be updated. Of course, if the calling context does not have that variable, it needs to get it anyway -- but if it does so implicitly as well, this benefit propagates further. This nicely reduces the complexity of adding new dependencies to certain actions.

Inspirations

This feature is inspired by mechanisms found in other languages. My experience is mostly developed in Scala and Haskell, so I hereby present their approach to this.

Scala

In Scala every method can be equipped with a separate set of arguments marked as implicit:

def sendRequest(Request req)(implicit ExecutionContext ctx, Session sess) {
    // ...
}

def testSendRequest()(implicit ExecutionContext) {
    implicit val s = Session.FreshSession
    val req = new TestRequest

    sendRequest(req)
}

First important note: Scala allows multiple implicit parameters. They are resolved by types, so it is invalid to have many of them sharing a common type. Therefore, the advised style is to always have a dedicated class for each one:

class KittenNumber(n : Int)
class DoggoNumber(n : Int)

def cuteness()(implicit KittenNumber k, DoggoNumber d) {}

Next, note how implicit parameters are provided from an implicit-free context. In the first example, they are not given to the sendRequest method directly, but are declared as implicit in prior. That instructs the compiler to use them as arguments to methods that need them.

A thing that disturbs me with how it is implemented here, is that implicit variables do not have to be introduced anywhere near the method call. In fact, they can come from a separate module via an import. While the whole point is to make it, well, implicit, to me this goes way too far. This can leave the programmer perplexed about the source of the arguments, which can lead to convoluted bugs.

UPDATE they apparently revisited that heavily. My knowledge might be then outdated, please check out this post for a more recent view.

Haskell

While Haskell does not support implicit parameters directly, it offers more general mechanisms that can be used to achieve pretty equivalent behavior -- the Reader monad. I am not going to start a discussion about monads, so please allow me to be slightly imprecise in this section.

In Haskell we write functions like this:

plus :: Int -> Int -> Int  -- function that takes two ints and returns an int
plus x y = x + y           -- arguments are separated by space, no parentheses nor commas

Reader lets to have one implicitly passed argument (which can be a tuple or a record, if more are needed). In order to do so, one needs to declare the return type of a function as Reader env ret where env is the implicit argument, and ret is the actual return type. In action, it looks as follows:

sendRequest :: Request -> Reader (ExecutionContext, Session) Int
sendRequest req = do 
  (ctx, sess) <- ask  -- `ask` returns the implicit parameter
  return 0

sendTestRequest :: Reader (ExecutionContext, Session) Int
sendTestRequest = sendRequest (makeRequest "test")

testSendRequest :: Int
testSendRequest =
  -- Calling a Reader from a non-Reader
  runReader sendTestRequest (makeExecutionContext, makeSession)

Note some key differences compared to Scala:

There is no syntax dedicated specifically for this feature. There is some syntax that supports monads in general (like the <- operator), but it is possible to go without it as well.
There is just a single implicit parameter. In our case, we needed two objects (Session and ExecutionContext), so a tuple is used. While this solves the problem with ambiguity, it enforces manual conversion when calling a function that needs only a Session for example.
Outside Reader, implicit parameters are provided explicitly using the dedicated runReader function. There is no way they come from other module or cosmic space.
Implicit parameters are a part of the return type, not the domain (yeah, I know it's not entirely true).

What I like about it, is the clarity of the implicit behavior. In fact, Reader is not a language construct, but a casual type defined in a dedicated library. Therefore, it is easy to inspect its exact logic just by looking into its code. However, Haskell and C# differ massively, so this model is not entirely transportable. But it also does not mean we cannot get any inspirations from it.

Detailed design

General syntax

I have already shown one idea that resembles the way Scala does it. Syntactically, I think it is a quite pretty and compatible with C# option. Therefore, my first idea it to extend the method declaration syntax to optionally include a second parameter list prepended with a single implicit keyword:

void f(int x)(implicit int y, int z) {}

Alternatively, implicit parameters could stay together with optional parameters to prevent double parameter lists. In fact, their behavior is quite similar to the one provided by CompilerServices, such as CallerMemberName. Therefore, a less intrusive designs emerge:

// One idea, syntax-driven
void f(int x, implicit int y = 10, implicit int z = 3) {}

// Another idea, attribute-driven
void f(int x, [System.Runtime.CompilerServices.Implicit] int y = 10, [System.Runtime.CompilerServices.Implicit] int z = 3) {}

// Yet another idea, value-driven
void f(int x, int y = implicit, int z = implicit) {}

I personally like the "syntax-driven", single-parameter version the most. The main argument for it is compatibility with existing language constructions, while keeping neat looking syntax.

Supplying implicit arguments from non-implicit methods

As I said, I do not enjoy the way the arguments are picked in Scala. In my opinion, there should always be a clear way of finding the sources of implicit parameters. Therefore, I propose letting them be taken only:

From the implicit parameters of the current method
Through explicit application

Hence this:

int f(int x)(implicit int y);

int g() {
    return f(3, y: 42);
}

instead of this:

int g() {
    implicit int 42;
    return f(3);
}

...and definitely not this

using Fizz.Buzz.Enterprise.SomeMagicImplicitFortyTwoProvider;

int g() {
    return f(3); // So what does 'y' equal to???
}

On the other hand, declaring variables for implicit use may be handy in some cases. It does not stay in conflict with the ability to perform explicit applications to implicit arguments as well. But then there comes a question: what restrictions would we pose on it? Should the parameters be allowed to be taken only from the current block, method, class, namespace or any imported module? Maybe they deserve some special place, such as beginning of the current block? The more flexibility here, the less trivial the resolution is -- not only for the compiler, but more importantly for the programmer.

I think the safest option would be to start with how I proposed it (so to pass arguments implicitly only when they are implicit parameters of the current method). Then we could see how much a limitation it is, and pay attention to possible demands on extending it further.

If supplying them manually starts getting annoying, then a possible workaround would be to lift the context with another method. So this:

void f1()(implicit int x);
void f2()(implicit int x);
void f3()(implicit int x);

void g() {
    int arg = 123;
    f1(x: arg);
    f2(x: arg);
    f3(x: arg);
}

turns into this:

void f1()(implicit int x);
void f2()(implicit int x);
void f3()(implicit int x);

void g() {
    int arg = 123;
    gf(x: arg)
}

void gf()(implicit int arg) {
    f1();
    f2();
    f3();
}

Resolution of multiple implicit parameters

The design must consider ambiguities that emerge from use of multiple implicit parameters. Since they are not explicitly identified by the programmer, there must be a clear and deterministic way of telling what variables are supplied and in what order.

By type

Scala for instance allows only one implicit variable of a given type in every scope. That on one hand, makes the resolution easy, but on the other limits the flexibility to some extent. I am also unsure how it deals with subtyping (I assume it requires the type to match exactly). The preferred way of dealing with it is to create a separate class for every kind of implicit parameter. C# has more verbose syntax for class declaration, so I am slightly afraid that it would come out annoying.

By name

Yet another way of tackling this, is to make the resolution dependent on the name of the parameter instead. While it allows multiple parameters sharing the same type, as well as using subtypes to be supplied, the downside of it is that it fixes argument names across the whole flow. It is a certain limitation, but on the other hand is it necessarily bad, especially considering the preference that these arguments should generally serve the same purpose all the way down? If one would want to change the name in the middle, an explicit application could still be used.

Not at all

In the end, only a single implicit parameter could be allowed. If someone wants more, then record types could be used. A limitation would be noticeable in cases where a subset of the implicit arguments would be needed in a local call (recall the gRPC example, where RequestNoFile did not need the inStream, but required outStream). While this could be possibly solved with inheritance on that record, I do not see big benefits over the by-name resolution of multiple parameters.

Backwards compatibility

Since I propose reusing an existing keyword, all valid identifiers should remain valid. The only added syntax is an optional set of parameters, which currently does not serve any purpose, so no conflicts would arise from that either. Therefore, the feature should not break nor change the meaning of any existing code.

Performance

These parameters turn into normal ones in an early phase of the compilation, so no runtime overhead at all. Compilation time would be affected obviously, but it depends on the resolution algorithm. If kept simple (what I believe is achievable), the impact should not be very noticeable. More than that, there is no overhead if the feature is not used.

Editor support

Since the feature would be desugared quite early, it should be easy to retrieve what arguments are applied implicitly. Thus, if some users find it confusing, I believe it would not be very hard to have a VS (Code) extension that would inform about the details of the implicit application. A similar thing to adding parameter names to method calls.

Drawbacks

Well, "implicit". This word is sometimes enough to bring doubts and protests. As much as I personally like moving stuff behind the scenes, I definitely see reasons to be careful. All that implicit magic is a double-edged sword -- on one hand it helps keeping the code tidy, but on the other can lead to nasty surprices and overall degraded readability.

One of the most common accusations against Scala is the so-called "implicit hell", which is caused by sometimes overused combination of extension classes (known there as, of course, "implicit" classes), implicit parameters and implicit conversions. I am not a very experienced Scala programmer, but I do remember finding Akka (a Scala library that uses implicits extensively) quite hard to learn because of that.

As mentioned before, there is an article by Scala itself, that points out flaws in the Scala 2 design. I encourage the curious reader for a lecture on how not to do it.

Also, there is a discussion under a non-successful proposal for adding this to Rust. The languages and priorities are fairly different, but the critics there clearly have a point.

Alternatives

Here are a bunch of alternative ways of implementing implicit parameters. I do not think they are really comparable to the ones I proposed, but decided to leave them for inspirational purposes.

Haskell style

One alternative approach is to imitate the mentioned Reader from Haskell. Thus, instead of adding a new parameter list, a wrapper for the return type is created, that informs what is the implicit parameter:

WithEnv<Context, int> f(int t);

WithEnv<Context, int> g() {
    int x = f(3); // int, not WithEnv<Context, int>!
    Context ctx = WithEnv.GetEnv();
    return x + ctx.Length;
}

int h() {
    return g().run(new Context());
}

What I find unfavorable here, are the unobvious "casts" between wrapped and unwrapped types. I am talking especially about the line with int x = f(3), where the type of x is int, despite the fact that f returns WithEnv. At first glance it is similar to what happens with Task in async methods, but there the conversion is indicated explicitly with the await keyword. Here coming with a similar solution would weaken the implicitness, which in my opinion contradicts the whole point.

Haskell solved it by using a different <- assignment operator, and in OCaml they write let! x = ... instead of let x = .... While it could work here as well, I do not think that adding a new operator just for this particular feature would be worth the hassle. Both languages have it widely generalized, so it made sense there.

Python style

We can always use method annotations (a.k.a. "decorators")

[WithEnv(Context)]
int f(int t);

[WithEnv(Context ctx)]
int g() {
    int x = f(3);
    return x + ctx.Length;
}

int h() {
    return WithEnv.run(g(), new Context());
}

What I do not like about this, is that it separates implicit parameters from the type signature. To me, implicit parameters are an unseparable part of the method's type, so delegating them outside stays against the intuition. More than that, I believe it would make it harder for the language and tooling to support the feature properly. Last, it is very verbose and looks awful.

Unresolved questions

Should we allow implicit parameters on properties?
Should we allow implicit parameters on constructors?
What resolution strategy would be the best for multiple implicit parameters (by type/by name/single only)?
Do we want two separate parameter lists (like in Scala) or keep them all in a single one (like we do with optional parameters)?
- If we keep two lists, which one should go first?
- If we integrate with optionals, should be use a keyword, an attribute, or replace the value?
Do we want to enforce/forbid default values for implicit parameters?
Do we want to allow declaring implicit variables, or should we stick to explicit application?
- If we add implicit variables, where should they be allowed? Local scopes, class scopes, cross-module scopes?

dotnet / csharplang

[Proposal]: Implicit parameters #5988