scala / scala3

The Scala 3 compiler, also known as Dotty.
https://dotty.epfl.ch
Apache License 2.0
5.88k stars 1.06k forks source link

Problems with jsoniter compilation after 3.3.1 #19098

Open Kordyjan opened 11 months ago

Kordyjan commented 11 months ago

Compiler version

Everything after 3.3.1. 3.3.2-RC1-bin-20230530-28915c4-NIGHTLY - 3.3.2-RC1-bin-20230630-c629090-NIGHTLY - xcheck-macros error from 3.3.2-RC1-bin-20230703-0a21ecf-NIGHTLY - typer crash

Reproduction

git clone --branch v2.24.4 --depth 1 https://github.com/plokhotnyuk/jsoniter-scala.git
cd jsoniter-scala
sbt
++3.3.2-RC1-bin-20230630-c629090-NIGHTLY!
jsoniter-scala-macrosJVM/Test/compile

Output (click arrow to expand)

Up to 3.3.2-RC1-bin-20230630-c629090-NIGHTLY - xcheck-macros assertion fail about missing position From 3.3.2-RC1-bin-20230703-0a21ecf-NIGHTLY:

``` [error] java.lang.AssertionError: assertion failed [error] scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:11) [error] dotty.tools.dotc.core.Types$LazyRef.ref(Types.scala:3111) [error] dotty.tools.dotc.core.Types$Type.dealias1(Types.scala:1433) [error] dotty.tools.dotc.core.Types$Type.dealias(Types.scala:1441) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:171) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.op$proxy25$1(Types.scala:6246) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldArgs$3(Types.scala:6246) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6250) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6284) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6284) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:172) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.op$proxy25$1(Types.scala:6246) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldArgs$3(Types.scala:6246) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6250) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6284) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [error] dotty.tools.dotc.core.Types$TypeAccumulator.foldOver(Types.scala:6284) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:198) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:172) [error] dotty.tools.dotc.typer.Inferencing$IsFullyDefinedAccumulator.apply(Inferencing.scala:169) [...] ```
nicolasstucki commented 11 months ago

Up to 3.3.2-RC1-bin-20230705-13f877b-NIGHTLY - xcheck-macros assertion fail about missing position

I did not manage to reproduce this one. I get the same failure that is in the details.

Kordyjan commented 11 months ago

I fixed the ranges in the original comment.

WojciechMazur commented 11 months ago

It might be possible that it's related to https://github.com/lampepfl/dotty/issues/18257 based on JsonCodecMakerNewTypeSpec test which defined recursive jsons. Altough there is was either failing or crashing with stack overflow depending on the complexity of match type.

nicolasstucki commented 11 months ago

The missing position was fixed in #18178.

nicolasstucki commented 11 months ago

I have no insight on the LazyRef.ref assertion failure.

WojciechMazur commented 7 months ago

The compilation crash is still present in the latest 3.5.0-RC1-nightly. The code that triggers the crash is a single test https://github.com/plokhotnyuk/jsoniter-scala/blob/84d57daeada31c0cce2631489e6d7c71e57be37e/jsoniter-scala-macros/shared/src/test/scala-3/com/github/plokhotnyuk/jsoniter_scala/macros/JsonCodecMakerNewTypeSpec.scala#L97-L205 One of the reasons for the crash is the used recursive match type - this kind of constructs are not allowed. It's reproduction can be found in here https://github.com/scala/scala3/issues/18257#issuecomment-1777010537 - however what's important in this form it leads to infinite loop instead of crash. The one thing that Scala compiler should do is to detect and break the cycle early. The recursive match type detection works only for simpler types, but in the complex ones, as the one in jsoniter, we never get any hints from compiler what goes wrong.

Edit

Actually in many cases the match type recursion would be reported as error, however, there are some cases when we get crash with no errors, eg. this one leads to stack overflow and no context about recursion being involved when the recursion match is defined in call-by-name block

import scala.collection.mutable
trait JsonValueCodec[T]
trait JsonReader
trait JsonWriter

def verifySerDeser[T](codec: JsonValueCodec[T], obj: T, json: String): Unit = ???
def test(block: => Unit) = ???

def Test = test{
    type JsonPrimitive = String | Int | Double | Boolean | None.type
    type Rec[JA[_], JO[_], A] = A match { // FIXME: remove this workaround after adding support of recursive types
      case JsonPrimitive => JsonPrimitive | JA[Rec[JA, JO, JsonPrimitive]] | JO[Rec[JA, JO, JsonPrimitive]]
      case _ => A | JA[Rec[JA, JO, A]] | JO[Rec[JA, JO, A]]
    }

    type Json = Rec[[A] =>> mutable.Buffer[A], [A] =>> mutable.Map[String, A], JsonPrimitive]
    type JsonObject = mutable.Map[String, Json]
    type JsonArray = mutable.Buffer[Json]

    val jsonCodec: JsonValueCodec[Json] = new JsonValueCodec[Json] {
      val nullValue: Json = None
    }

    def obj(values: (String, Json)*): Json = ???
    def arr(values: Json*): Json = mutable.ArrayBuffer[Json](values: _*)

    verifySerDeser(
      jsonCodec,
      arr("VVV", 1.2, true, obj("WWW" -> None, "XXX" -> 777)),
      """["VVV",1.2,true,{"WWW":null,"XXX":777}]"""
    )
  }