an unexpected type representation reached the compiler backend

[channingwalton]

Hi, we experience (somewhat randomly) this error:

[error] an unexpected type representation reached the compiler backend while compiling Operations.scala: (result: => doobie.imports.ConnectionIO[Either[String,T]])cobam.database.package.OperationResultConnectionIOEitherLiftSyntax[T]. If possible, please file a bug on https://github.com/scala/bug/issues.
scala.MatchError: (result: Function0)cobam.database.package.package$OperationResultConnectionIOEitherLiftSyntax[T] (of class scala.reflect.internal.Types$MethodType)
    at scala.tools.nsc.backend.jvm.BTypesFromSymbols.typeToBType(BTypesFromSymbols.scala:195)
    at scala.tools.nsc.backend.jvm.BTypesFromSymbols.methodBTypeFromMethodType(BTypesFromSymbols.scala:129)
    at scala.tools.nsc.backend.jvm.BTypesFromSymbols.methodBTypeFromSymbol(BTypesFromSymbols.scala:120)
    at scala.tools.nsc.backend.jvm.BTypesFromSymbols.$anonfun$buildInlineInfoFromClassSymbol$6(BTypesFromSymbols.scala:577)

(complete error attached). error.txt

I've attached the offending code too.

code.zip

[SethTisue]

is it possible to demonstrate the problem with a MCVE?

[channingwalton]

I’ll attempt it but this bug seems to exhibit in peculiar circumstances and randomly.

[lrytz]

Could it be that it shows up under separate compilation, in certain orders? Maybe you can experiment in that direction to find a reproducer.

For the stack trace: methodBTypeFromMethodType converts a MethodType, which is supposed to have a single parameter list in the backend phase. It invokes typeToBType(tpe.resultType), and typeToBType converts type references. But in this case it gets a MethodType, (result: => ConnectionIO[Either[String,T]])OperationResultConnectionIOEitherLiftSyntax[T]. So for some reason the outer MethodType is still curried.

[tpolecat]

We're running into this too, also randomly :-\

[som-snytt]

TFW you say how much pain a bug is causing you, and Seth is like oh yeah I meant to put that one on the Backlog.

[som-snytt]

How often does a type representation have to reach the backend in order to become expected? Maybe the error should say, "uninvited".

[tpolecat]

The project in question is here https://github.com/gemini-hlsw/lucuma-core at 30f412795195b54a3966917b2f28160ddd21e760 and we have been unable to narrow the failure down at all. The choice of JVM/JDK (8/11/17) doesn't seem to matter, nor do stack/heap options. What we're seeing is that the failure occurs randomly, in random places in the codebase, whether clean* or incremental, and disappears on retry. There is another problem that's also happening randomly (Bad superClass for class #11115) which may or may not be related.

I realize this isn't actionable yet, but I'll try to fill in clues as they accumulate and maybe things will become more clear.

*I'm going to try to verify this, we're not 100% sure yet.

[SethTisue]

if you're certain you've seen it happen with a clean compile, that's important information, as it rules out an entire category of possible causes

[SethTisue]

is the problem reliably reproducible by just clean-compiling that codebase over and over again until it fails?

[SethTisue]

also, just checking: this is on 2.13.7, yes?

[tpolecat]

Yes, 2.13.7.

We have seen this and related issues in GH CI, which should be clean although we may add git clean -xdf to the build just to be extra sure. I added an asterisk to that assertion. I'm going to build clean on a loop for a while and will report back.

[tpolecat]

Ok, so yes it absolutely happens with clean builds, as do at least three other distinct kinds of compiler crashes. I'm cataloging them here https://github.com/tpolecat/lucuma-core-crashes and will keep building in a loop until it seems like I have exhausted the possible flavors of crash.

I think I'm going to have to bisect and figure out when it started going nuts. Something very bad is happening somewhere.

[SethTisue]

it would be interesting to know if any of the following make any difference:

• rewinding to 2.13.6
• advancing to the 2.13.8 release candidate
  • (though this might require locally publishing e.g. kind projector, which is a bit of a pain)
• cutting back on compiler flags (the fewer you use, the better-tested territory you're in)

or even, and god forbid this should turn out to be relevant, but maybe try Temurin instead of GraalVM

[som-snytt]

Also curious if it rhymes with "Look, ma!", "lacuna", "hakuna (matata)", "lukas (rytz)", etc.

[som-snytt]

I think that was my second attempt:

➜  lucuma-core git:(master) git clean -dfx ; sbt compile
Removing .bsp/
Removing modules/core/js/target/
Removing modules/core/jvm/target/
Removing modules/testkit/.js/
Removing modules/testkit/.jvm/
Removing modules/tests/js/target/
Removing modules/tests/jvm/target/
Removing project/project/project/
Removing project/project/target/
Removing project/target/
Removing target/
[info] welcome to sbt 1.6.1 (Oracle Corporation Java 17.0.1)
[info] loading settings for project lucuma-core-build-build from plugins.sbt ...
[info] loading project definition from /home/amarki/projects/lucuma-core/project/project
[info] loading settings for project lucuma-core-build from plugins.sbt ...
[info] loading project definition from /home/amarki/projects/lucuma-core/project
[info] loading settings for project lucuma-core from build.sbt ...
[info] set current project to lucuma-core (in build file:/home/amarki/projects/lucuma-core/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] compiling 235 Scala sources to /home/amarki/projects/lucuma-core/modules/core/jvm/target/scala-2.13/classes ...
[info] compiling 234 Scala sources to /home/amarki/projects/lucuma-core/modules/core/js/target/scala-2.13/classes ...
[info] compiling 64 Scala sources to /home/amarki/projects/lucuma-core/modules/testkit/.jvm/target/scala-2.13/classes ...
[error] Error while emitting CatalogInfo.scala
[error] (): monocle.PLens[(C, lucuma.core.model.CatalogInfo),(C, lucuma.core.model.CatalogInfo),(C, lucuma.core.enum.CatalogName),(C, lucuma.core.enum.CatalogName)] (of class scala.reflect.internal.Types$MethodType)
[warn] an unexpected type representation reached the compiler backend while compiling CatalogInfo.scala: (): monocle.PLens[(C, lucuma.core.model.CatalogInfo),(C, lucuma.core.model.CatalogInfo),(C, lucuma.core.enum.CatalogName),(C, lucuma.core.enum.CatalogName)]. If possible, please file a bug on https://github.com/scala/bug/issues.
[warn] one warning found
[error] one error found
[info] compiling 1 Scala source to /home/amarki/projects/lucuma-core/modules/tests/jvm/target/scala-2.13/classes ...
[error] (coreJS / Compile / compileIncremental) Compilation failed
[error] Total time: 60 s, completed Jan 4, 2022, 10:35:17 AM

Hakuna matata!

[tpolecat]

[som-snytt]

It looks like a smiley face except screaming like the famous painting.

[tpolecat]

I have tried 3 JVMs as well as 2.13.6 and 2.13.5 and all crash more or less immediately in one of 7 (so far) ways.

I will remove all compiler flags next, whee!

This is more fun than I anticipated.

[tpolecat]

No, setting scalacOptions := Nil didn't help. I'm going to bisect and see if I can figure out where it started going bad.

[eed3si9n]

Both in Som's repro and 1.txt, the error compiler seems to be JS?

[info] compiling 1 Scala source to /Users/rnorris/Scala/lucuma-core/modules/tests/jvm/target/scala-2.13/classes ...
[error] (coreJS / Compile / compileIncremental) Compilation failed
[error] Total time: 71 s (01:11), completed Jan 4, 2022, 11:00:38 AM

I wonder if this has something to do with doing both Scalac JVM compilation and Scalac JS compilation in parallel on the same source code or something, which could explain the seemingly random nature.

[tpolecat]

@eed3si9n that's an interesting theory. Is there a way to turn parallel compilation off?

[som-snytt]

Also why does testkit have .js and .jvm? and .crossType(CrossType.Pure)

[tpolecat]

I don't know how any of that works. I guess it needs different places for target/ output?

[eed3si9n]

@tpolecat I think (https://www.scala-sbt.org/1.x/docs/Parallel-Execution.html):

Global / concurrentRestrictions += Tags.limit(Tags.Compile, 1)

[som-snytt]

I proposed a new stacktrace on the tracking project, using the "add new file" feature in github, how easy!

Something using reflect toolbox and eval says

[error] java.lang.AssertionError: assertion failed: import failure: cannot determine unique overloaded method alternative from
[error]  final package eu
[error] final package eu

The beauty of it is that now we know that soon, tpolecat will be on Scala 3.

[som-snytt]

It's some thorough background doc that begins, "Prior to sbt 0.12, ...", which continues, "in a galaxy far, far away."

[tpolecat]

That has to be eu.timepit.refined and there are some known old sporadic crashes.

[som-snytt]

I can't believe I got all excited about a known old sporadic crash.

Edit: it compiled with concurrentRestrictions

[tpolecat]

Yeah so far so good with concurrentRestrictions. Letting it run for a while...

[tpolecat]

Ok thanks a bunch @SethTisue @som-snytt @eed3si9n, removing parallel compilation seems to have resolved the problem.

@channingwalton sorry for polluting your issue with all this noise!

[SethTisue]

in case anyone lands here looking for the refined thing, here's the link: https://github.com/fthomas/refined/issues/260

[channingwalton]

@channingwalton sorry for polluting your issue with all this noise!

no worries, it was interesting!