Typechecking of a macro expansion infers wrong type for List.map

[scabug]

As per http://stackoverflow.com/questions/11681631/macro-return-type-and-higher-order-functions:

Macro definition:

def test(s:String) = macro testImpl

def testImpl(c:Context)(s:c.Expr[String]):c.Expr[Any] = {
  import c.universe._
  val list = reify(List(1)).tree

  val function = reify((x:Int) => x).tree

  val res = 
    Apply(
      Select(
        list,
        newTermName("map")),
      List(function)
    )

  c.Expr(res)
}

Macro call:

val list:List[Int] = test("")

Error message:

[error]  found   : Any
[error]  required: List[Int]
[error]     val list:List[Int] = test("")
[error]                              ^

[scabug]

Imported From: https://issues.scala-lang.org/browse/SI-6155?orig=1 Reporter: @xeno-by Affected Versions: 2.10.0, 2.11.0

[scabug]

@xeno-by said: This is the result of typechecking the macro expansion:

immutable.this.List.apply[Int](1).map[Int, Any](((x: Int) => x))(immutable.this.List.canBuildFrom[Int])

Interestingly enough, c.typeCheck works correctly producing:

immutable.this.List.apply[Int](1).map[Int, List[Int]](((x: Int) => x))(immutable.this.List.canBuildFrom[Int])

[scabug]

@xeno-by said: Hence there's a workaround. Replace c.Expr(res) with c.Expr(c.typeCheck(res)), and everything works!

[scabug]

Kim Stebel (kimstebel) said: Btw, the same problem occurs with flatMap and collect, but not with filter or reduce.

[scabug]

@xeno-by said (edited on Aug 11, 2012 10:10:12 AM UTC): The problem is immediately caused by macro expansions having an elaborate scheme of typechecking, but macros themselves are not at fault here. First an expansion is typechecked against the return type of a macro impl, in that case Any. Then it's typechecked against the required type, in that case, List[Int].

When immutable.this.List.apply(1).map(((x: Int) => x)) is typechecked against Any, the result gets these weird [Int, Any] type arguments inferred for map (one would expect [Int, List[Int]] instead, which are indeed inferred if we typecheck against WildcardType). This is not macro-specific. If one writes "val foo:Any = List.apply(1).map((x: Int) => x)", the inference result will be wrong as well.

Therefore I suggest there's something wrong with the inferrer.

[scabug]

@Blaisorblade said (edited on Aug 30, 2012 5:52:23 PM UTC): As discussed on scala-internals (https://groups.google.com/d/topic/scala-internals/5mebTX1bqDU/discussion), (a variant of) the problem is even easier to trigger. It seems that Any should never be used as parameter type.

The following looks like an identity macro but isn't:

def macroId(arg: Any) = macro macroId_impl
def macroId_impl(c: Context)(arg: c.Expr[Any]): c.Expr[Any] = arg

for the same reason for which def idAny(v: Any): Any = v is not an identity function: its argument gets typechecked against Any.

The correct version of the macro is:

def macroId[T](arg: T): T = macro macroId_impl[T]
def macroId_impl[T: c.AbsTypeTag](c: Context)(arg: c.Expr[T]): c.Expr[T] = arg

The problem could be reduced by having the typechecker special-case Any as a target type, and treating it as WildcardType. Of course, this doesn't solve the general problem, it just makes it less common; but since using the expected type is part of bidirectional type inference, it seems that a general solution could be quite complex.

[scabug]

@Blaisorblade said: This might not be a bug after all, if not a documentation bug. See below.

As Eugene said, macros are not at fault because type inference in this code does the same thing, in all calls to map - they are inferred as map[Int, Any]:

def id(x: Any): Any = x
id(List.apply(1).map((x: Int) => x))
val foo:Any = List.apply(1).map((x: Int) => x)
println(List.apply(1).map((x: Int) => x))

However, all that code works!

What strikes me as odd is another thing. Given that test returns Any:

def test(s:String) = macro testImpl
def testImpl(c:Context)(s:c.Expr[String]):c.Expr[Any] = ...

why should one expect this code:

val list:List[Int] = test("")

to work? Why isn't it an error that it compiles, after adding the call to typecheck? Apparently, the macro output can statically refine the declared type - but how should this work, are there guarantees on that? I'm not sure anymore that this is a bug - unless you can point me to an explanation for users of how things are supposed to work, we might have a documentation bug here.

[scabug]

@xeno-by said: Related: http://stackoverflow.com/questions/13669974/static-return-type-of-scala-macros.

[scabug]

@xeno-by said: Let's keep this one in mind for 2.12, especially given the progress made by quasiquote implementation macros.

[scabug]

@paulp said: For another fascinating wrinkle, see https://github.com/puffnfresh/wartremover/pull/46 .

class A {
  def f1 = Vector(1,2,3) == List(1).map(_ + 1)
  def f2 = Vector(1,2,3) equals List(1).map(_ + 1)
}

The first map call infers List[Int] for the type argument, the second infers Any. Since I had already observed that arguments to overloaded targets are typed more accurately than arguments to not-overloaded targets, I looked and indeed 'def ==' is overloaded between Any and AnyRef whereas 'def equals' is overridden in AnyRef.

[scabug]

@retronym said: Funny, I noticed that overloading a few days back in #8219.

[scabug]

@retronym said: I've diagnosed the unwanted overloading in https://github.com/scala/scala/pull/3460

AnyRef and Object don't fit into traditional binary notions of isJava.

[SethTisue]

it's not clear to me if there's anything actionable here

someone can reopen if they can clearly summarize the state of the ticket (or perhaps it would be better to open one or more new tickets)

[Blaisorblade]

Following up on this comment of mine, the documentation issue has arguably been fixed.

With blackbox macros, the original code is clearly ill-typed, because a function returning Any can't be used in a context expecting List[Int].

Instead, for whitebox macros, IIUC it's nowadays clear enough that "all bets are off", isn't it?

So IIUC this isn't even a bug.