New implicit parameter & argument syntax

[odersky]

Motivation

The current syntax for implicit parameters has several shortcomings.

1. There can be only one implicit parameter section and it has to come at the end. Therefore normal and implicit parameter types cannot depend on other implicit parameters except by nesting in an inner object with an apply method.
2. The syntax (implicit x: T, y: U) is a bit strange in that implicit conceptually scopes over x and y but looks like a modifier for just x.

3. Passing explicit arguments to implicit parameters is written like normal application. This clashes with elision of apply methods. For instance, if you have

    def f(implicit x: C): A => B

   then f(a) would pass the argument a to the implicit parameter and one has to write f.apply(a) to apply f to a regular argument.

   Proposal

   • Introduce a new symbolic delimiter, ?(. This is one token, no spaces allowed between the ? and the (.
   • Write implicit parameter definitions with ?( instead of (implicit. E.g.

   def f(x: Int)?(ctx: Context) = ...

   instead of

   def f(x: Int)(implicit ctx: Context) = ...

   • Explicit arguments to implicit parameters have to be enclosed in ?(...). E.g.

   f(3)?(ctx)

   • There can be several implicit parameter sections and they can be mixed with normal parameter sections. E.g.

   def f ?(ctx: Context)(tree: ctx.Expr) = ...

   Problems

   • ? can be already used as an infix operator. So the meaning of (a)?(b) would change but only if no space is written after the ?, which should be rare.
   • ? can be part of a symbolic operator. In this case the longest match rule applies. So def #?(x: T) defines an operator #? with a regular parameter list. To define # with an implicit parameter list an additional space is needed: def # ?(x: T).

[odersky]

Very thought provoking proposal! It's true that implicits and defaults have overlapping use cases. As @acjay notes, it is already possible to combine them, like this:

def mapAsync[B](f: A => B)(implicit parallelism: Int = 2)

I am dubious about using named parameters as a disambiguation tool though. First, implicits coming from context bounds [A: Ord] or implicit function types don't have a named parameter. So if we want to pass explicit arguments to them we'd need a different scheme. Second, even for normal implicits you might get into a situation like this:

    class C { def apply(x: T): T }
    def foo(implicit x: T): C

Then foo(x = t) is again ambiguous. So it seems named arguments by themselves are too weak and fragile as a disambiguation mechanism.

There's also an important difference between default parameters and implicits that seems to have been glossed over so far: they behave differently under partial application. Given:

implicit val i: Int = 2
def f(implicit x: Int): Int
def g(x: Int = 1): Int

we have:

f   maps to   f(2)             : Int
g   maps to   (x: Int) => g(x) : Int => Int

[acjay]

@propensive My understanding is that today, parameter lists are typechecked one at a time, and so dependent types or values have to be fully resolved in the previous parameter list:

@ case class X(x: Int = 5)(implicit val y: Int = x) 
defined class X
@ X().x 
res3: Int = 5
@ X().y 
res4: Int = 5

This is certainly pretty easy to reason about, but it's also quite limiting. Just this week, I wanted to do something where a parameter's default value depended on implicitly available context. I'll try to simplify the essence of it to be bite-sized:

case class ReifiedOperation(
  arg: String, 
  blockingStrategy: BlockingStrategy = Block(timeout)
)(
  implicit timeout: Int
)

sealed trait BlockingStrategy
case object DontBlock extends BlockingStrategy
case class Block(timeout: Int) extends BlockingStrategy

val defaultOperation = ReifiedOperation("this is normal")
val specialOperation = ReifiedOperation("i'm special", blockingStrategy = DontBlock)

This doesn't work, for reasons I'm sure we all understand.

• The point Jon made brings up the question of whether it's possible to make parameter list type checking more flexible? Could types and values be resolved in a more dynamic order, as long as it's possible to infer a DAG of dependencies?

It seems like it would solve my modeling issue. Although, with Li Haoyi's idea, I'd probably do:

case class ReifiedOperation(
  arg: String, 
  implicit timeout: Int,
  blockingStrategy: BlockingStrategy = Block(timeout)
)

• So, that raises another follow-up question. What are the rules for exposure as val in case class and for use in the synthesized unapply method?

Let me end by saying I've long pined for keyword args to have the same level of syntactic support as positional args, which could be a side-effect of exploring this proposal.

[gbersac]

@lihaoyi when I began scala, this his how I expected implicits to work. The power of this proposal is that it feels so natural. Great proposal !

On the other hand, the proposition of implicits with the ? keyword feel alien to me. Why a question mark to represent implicit ? Why not a hashtag or anything else ? It would required to google it to understand this synthax.

[LPTK]

@odersky

First, implicits coming from context bounds [A: Ord] or implicit function types don't have a named parameter. So if we want to pass explicit arguments to them we'd need a different scheme. Second, even for normal implicits you might get into a situation like this:

class C { def apply(x: T): T }
def foo(implicit x: T): C

Then foo(x = t) is again ambiguous.

I see a very simple solution to that: an implicit argument list can only be omitted if it's the last specified argument list.

So foo(x = t) has type C (sets foo's implicit x to t), and if you wanted to actually call C's apply method you'd write either foo.apply(x = t) as is currently required, or foo()(x = t) – which is enabled by @lihaoyi's proposal, and solves the issue of apply elision failure. It's also very intuitive IMHO.

What's more, I think the rule above is backward compatible: in current Scala, foo(x = t) also means what it does under @lihaoyi's proposal plus this rule.

[lihaoyi]

@odersky some responses:

First, implicits coming from context bounds [A: Ord] or implicit function types don't have a named parameter. So if we want to pass explicit arguments to them we'd need a different scheme.

Here's one scheme that fits well with my proposal: you can pass implicit parameters positionally, only if you use the implicit keyword.

def foo(a: Int, implicit b: String, c: Boolean) = ???
foo(1, true) // b is implicit
foo(1, b = "lol", true) // b is explicitly provided
foo(1, implicit "lol", true) // b is explicitly provided

That will allow both passing arguments positionally while also allowing inference of interleaved implicit and non-implicit parameters.

Another alternative, is to give up interleaved-implicit-and-non-implicit parameters-within-single parameter-list: then we could just require that implicit parameters always occur after non-implicit parameters in a parameter lists:

def foo(a: Int, b: String, implicit c: Boolean) = ??? // implicits must come at the end
foo(1, "lol") // b is implicit
foo(1, "lol", c = true) // b is explicitly provided
foo(1, "lol", true) // b is explicitly provided

There are probably other schemes we could come up with to allow passing in implicit parameters positionally, if we really wanted to explore that design space. I think both of the above syntaxes look a lot more natural than foo(1, true).explicitly("lol")

Second, even for normal implicits you might get into a situation like this. Then foo(x = t) is again ambiguous.

I think this is a problem we can solve by decree. For example, we may say that given:

    class C { def apply(x: T): T }
    def foo(implicit x: T): C

Then foo(x = t) returns C, and if you want to apply it twice you use foo(implicitly)(x = t) or foo.apply(x = t). This is not unlike what we already have, and I think wouldn't be too surprising.

This point is a bit like PEG parsers v.s. context-free-grammars: when there's ambiguity it's a bit arbitrary to pick one parse over the other, but as long as it's simple & predictable, it may be good enough or even better than an unambigious parse based around clever constraint-solving.

here's also an important difference between default parameters and implicits that seems to have been glossed over so far: they behave differently under partial application.

To be more precise, currently implicits behave differently under currying partial application, and there's an arbitrary restriction on using implicits for non-curried partial application:

// non-curried partial application with default arguments
def foo(i: Int, b: Boolean = true) = ???
f(1)

// non-curried partial application with implicits (doesn't work at all in status quo)
def foo(i: Int, implicit b: Boolean) = ???
f(1)

My proposal makes implicits and defaults behave the same under non-curried partial application (the second example above). Making them behave the same under non-curried partial application means someone learning about implicits doesn't need to learn about currying at the same time. Maybe for people with a Haskell/Scheme/OCaml background are already familiar, but multiple-argument-list currying is very foreign to people from Java/C#/Python/Javascript backgrounds (see e.g. the perpetual confusion around parametrized-decorators in Python).

And ultimately, "i want to pick a default parameter value from the local scope" and "I want to use multiple sets of parentheses when i call this function" are entirely orthogonal concerns

We could preserve the existing syntax & semantics (eliding entire argument lists, etc.) for curried implicit parameter lists, whether for compatibility or for other reasons, without impacting the main body of the proposal.

[LPTK]

@lihaoyi

if you want to apply it twice you use foo(implicitly)(x = t) or foo.apply(x = t).

I don't understand why not foo()(x = t), as I suggested above. If you allow def f(a: Int, implicit b: Int) to be called as f(1), then for sure you would also allow def f(implicit b: Int) to be called as f(), no?

Also, I missed what your rationale was for disallowing passing implicit arguments positionally. Why not simply use exactly the same rules as are currently in place for default arguments?

// current Scala:
def f(x: Int=0, y: Int) = x + y
f(0) // not enough arguments
f(y = 0) // ok
f(0, 1) // ok

// under new proposal:
def g(x: Int = implicit, y: Int) = x + y  // or in your proposed syntax (implicit x: Int, y: Int)
g(0) // not enough arguments
g(y = 0) // ok
g(0, 1) // ok

[odersky]

@lihaoy OK. As far as I see it then, the proposal does not affect the curried implicits of current Scala since by-name parameters are not a good disambiguation mechanism. We still need something else, and foo(x).explicitly(y) is a possible candidate which has the advantage that it fits the current syntax well.

The question is then, should we add another way to define implicits which makes them similar to default parameters? If we do, we have to solve the problem that implicits and defaults behave differently with respect to partial application. Here's a problematic scenario. Say you have

def f(x: Int = 1, implicit y: C)

Then f() is f(1, implicitly[C]). Fine.

Now say you want to turn the first parameter into an implicit as well, since you need more flexibility:

def f(implicit x: Int, implicit y: C)

Now f() is illegal, or, depending on its result type, means something completely different! You have to write f instead. I don't think we should open the door to surprises like this. The other possibility would be to reconcile implicits and defaults by changing the behavior of default parameters. I.e. given a function like

def g(x: Int = 1)

g() would be illegal and instead g would expand to g(1). But that would be a backwards incompatible change. And it would open another discrepancy where I cannot eta expand g to x => g(x) anymore just because g has a default argument. Altogther this looks at least as bad as auto-unit insertion, which we just dropped.

I also wanted to raise another red flag. All the examples we gave in this thread are extremely bad code since they pretend it's OK to define an implicit for a common type like Int. Most default arguments do have common types, but implicits should never have them. So it looks like moving from default parameters to implicits should specifically not be seamless and require some effort from the programmer.

[odersky]

While we are discussing implicit definition syntax, here's another thought. Currently when we write

    def f(x: Int)(implicit c: C) = ... implicitly[C] ...

we get "two-way" implicitness - the second argument of f is synthesized, and c is in turn available as an implicit in the body of f. It would be nice if there was a way to disentangle these two functions. One possible syntax to express this would be to also allow implicit in front of a parameter list:

   def f(x: Int) implicit (c: C) = ... c ...

This would still synthesize the second argument but c would have to be referred to explicitly in the body of f. As far as I can see, this change could subsume the functionality of MacWire as a dependency injection mechanism. Just set up your program like this:

    class C_1 implicit (C_1_1, ..., C_1_m1)
    ...
    class C_n implicit (C_n_1, ..., C_n_mn)

Here C_1, ..., C_n are the component classes, that each have a subset of the C_i as dependencies. Dependencies are expressed in a implicit parameter clause. Auto-wiring is then done like this:

  {
    implicit val c_1: C_1 = new C_1
    ...
    implicit val c_n: C_n = new C_n

    (c_1, ..., c_n)
  }

The reason why we do not want to do this with current implicits (and the reason why MacWire exists) is that we do not want to pollute the implicit namespace of the bodies of our components C_i with the dependent components. MacWire is implemented with the kind of macros that won't be supported in Scala 3, so it would be good to find a language-level alternative.

I am bringing this up here because it would also avoid the problem that implicits require curried parameter lists. (Although in fact I am not sure that's a strong argument. For better or for worse, curried parameter lists are pretty ubiquitous in Scala. You need them not just for implicits but also for better type inference, or for passing arguments in {...}).

/cc @adamw

[odersky]

Note: If we do adopt

def f(x: Int) implicit (c: C) = ...

then the current definition syntax

def f(x: Int)(implicit c: C) = ...

can be treated as syntactic sugar for

def f(x: Int) implicit (c$: C) = { implicit val c: C = c$; ... }

[propensive]

@LPTK No, the idea is that you should never be able to supply an implicit parameter "by accident" at the call site. It should be necessary that it's distinguished in some way.

[LPTK]

the idea is that you should never be able to supply an implicit parameter "by accident" at the call site. It should be necessary that it's distinguished in some way

Is this actually a central goal of the design? At least it's not part of the original motivation explained in the opening message of this github issue. But I may be missing something (e.g., discussions happening offline). Though I can see that having to write foo()(...) is a minor annoyance compared to just foo(...), I don't know if it's that important in practice.

[lavrov]

@lihaoyi Interleaving implicit and non-implicit parameters doesn't work with current implementation of implicit function types which are encoded like this:

trait ImplicitFunction1[-T0, R] extends Function1[T0, R] {
  override def apply(implicit x: T0): R
}
implicit (T0) => R

In the case of def foo(x: Int, implicit ctx: Context) = ... how would you represent it as a function type?

[acjay]

@odersky

Now say you want to turn the first parameter into an implicit as well, since you need more flexibility:

def f(implicit x: Int, implicit y: C)

Now f() is illegal, or, depending on its result type, means something completely different!

Hmm, just to make sure I'm understanding, this is because implicit parameter lists can currently be completely elided, but we're not going be allowing no-arg and zero-arg calls to be interchangeable anymore. Is that right? If so, then in the spirit of unifying the syntax of implicit and default args, these behaviors should be rectified somehow.

While we are discussing implicit definition syntax, here's another thought. Currently when we write

   def f(x: Int)(implicit c: C) = ... implicitly[C] ...

we get "two-way" implicitness - the second argument of f is synthesized, and c is in turn available as an implicit in the body of f. It would be nice if there was a way to disentangle these two functions. One possible syntax to express this would be to also allow implicit in front of a parameter list:

I often use an idiom where services are modeled by case class, and I use implicits for my wiring, as you say. In this case, I have to put val by implicits that I want to be propagated to the inner scope. So, in this particular case, there's already a way to differentiate one-way and two-way implicits. Could this be expanded by requiring val for two-way implicits elsewhere?

Lastly, on a meta level, one question folks seem to be dancing around a bit is whether it makes sense to continue to think of implicits positionally at all.

(I do get that you're largely talking about things that are additive to today's Scala, and I'm talking about breaking changes.)

[odersky]

Hmm, just to make sure I'm understanding, this is because implicit parameter lists can currently be completely elided, but we're not going be allowing no-arg and zero-arg calls to be interchangeable anymore. Is that right?

Sort of. They never were interchangeable. There was a one-way automatic conversion from one to the other, but that one's gone now as well.

I believe the driving force here is that we want to be very clear about to which parameter(s) arguments are passed to. Implicits are hard enough, no need to throw an additional puzzler in the mix.

[odersky]

I often use an idiom where services are modeled by case class, and I use implicits for my wiring, as you say. In this case, I have to put val by implicits that I want to be propagated to the inner scope. So, in this particular case, there's already a way to differentiate one-way and two-way implicits. Could this be expanded by requiring val for two-way implicits elsewhere?

Can you show an example? I don't see how having val or not would affect the scope.

[lihaoyi]

We still need something else, and foo(x).explicitly(y) is a possible candidate which has the advantage that it fits the current syntax well.

Yep that seems right!

I also wanted to raise another red flag. All the examples we gave in this thread are extremely bad code since they pretend it's OK to define an implicit for a common type like Int. Most default arguments do have common types, but implicits should never have them. So it looks like moving from default parameters to implicits should specifically not be seamless and require some effort from the programmer.

This is true, but can be mitigated mechanically: rather than having the implicit be Int, have the implicit be FooInt with an implicit constructor:

case class FooInt(value: Int)
object FooInt{
  implicit def create(value: Int) = FooInt(value)
}

This gives you the nice default-parameter syntax along with the nice specific type for implicit-resolution. It's a pattern I use pretty commonly (e.g. from String to fansi.Str, from Int to sourcecode.Line). It's a bit of boilerplate, but not too bad overall

[acjay]

@odersky Ah, you're right. I forgot the real reason I had needed val. In my case, is is because my services sometimes extend abstract traits, in a mini cake-pattern sort of way, and those traits sometimes need implicits. So I define them as abstract implicit members in the trait, which can only be fulfilled in a case class by using val.

[ryantheleach]

Sorry in advanced for my inexperience. But 'implicitly[Foo[A]]' inside a function/method body is essentially summoning something in implicit scope, without it being Available in the visible function definition right?

So, that's essentially a hidden, anonymous parameter (it's not named until you name it) that's not immediately obvious?

If we call the concept of summoning a parameter in this form, without following implicit resolution, 'anonymous parameter passing' in that, it 1. Has no name/keyword 2. Has no position in the function definition. Then what would an anonymous parameter without implicits look like? (In that it's required to be passed in (has no default value defined) what would syntax look like to pass it in?

I think if that was introduced orthogonal to implicit resolution, then you could have default values, implicits, and anonymous parameter passing all orthogonal to each other.

[LPTK]

@ryantheleach inside a method body, implicits are resolved based on the static scope of the body, so they do not behave like parameters. The resolution won't change depending on the call sites.

[gabriel-bezerra]

@odersky 's suggestion for implicit in and out of the parameter list reminds this (badly named) suggestion on Scala Contributors: https://contributors.scala-lang.org/t/more-on-duality-and-homonyms-in-the-language/1775

[nicolasstucki]

This was implemented in #5458 and then the syntax changed in #5825