search

scalanlp / breeze

Breeze is/was a numerical processing library for Scala.
https://www.scalanlp.org
Apache License 2.0
3.45k stars 692 forks source link

diverging implicit expansion on squaredDistance #810

Open pan3793 opened 3 years ago

pan3793 commented 3 years ago

Hi,

I'm trying to bump breeze from 1.0 to 1.2 in Apache Spark, but got such compile exceptions. And I noticed that breeze migrated to new codegen framework recently, is it relative?

[error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:60:46: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR]
[error] starting with method castOps in trait VectorOps
[error]       val squaredDist = breezeSquaredDistance(v1.asBreeze, v2.asBreeze)
[error]                                              ^
[error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:66:47: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR]
[error] starting with method castOps in trait VectorOps
[error]       val squaredDist2 = breezeSquaredDistance(v2.asBreeze, v3.asBreeze)
[error]                                               ^
[error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:74:48: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR]
[error] starting with method castOps in trait VectorOps
[error]         val squaredDist = breezeSquaredDistance(v2.asBreeze, v4.asBreeze)
[error]                                                ^
[error] four errors found
[warn] In the last 9 seconds, 5.224 (60.3%) were spent in GC. [Heap: 2.85GB free of 3.80GB, max 3.80GB] Consider increasing the JVM heap using `-Xmx` or try a different collector, e.g. `-XX:+UseG1GC`, for better performance.
[error] (mllib / Test / compileIncremental) Compilation failed
[error] Total time: 341 s (05:41), completed 2021-5-4 20:10:48

The compile failed code is here https://github.com/pan3793/spark/blob/SPARK-35310/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala

And to reproduce the issue, run build/mvn clean compile -DskipTests or build/sbt compile test:compile.

dlwh commented 3 years ago

Interesting. I imagine something got messed up. I'm currently doing a major refactor to try to get Scala 3 support in and I'm fighting a lot of issues like this. In particular the behavior around castOps is getting changed quite a lot, so I hope this will be fixed soon.

On Tue, May 4, 2021 at 5:44 AM Cheng Pan @.***> wrote:

Hi,

I'm trying to bump breeze from 1.0 to 1.2 in Apache Spark, but got such compile exceptions. And I noticed that breeze migrated to new codegen framework recently, is it relative?

[error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:60:46: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR] [error] starting with method castOps in trait VectorOps [error] val squaredDist = breezeSquaredDistance(v1.asBreeze, v2.asBreeze) [error] ^ [error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:66:47: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR] [error] starting with method castOps in trait VectorOps [error] val squaredDist2 = breezeSquaredDistance(v2.asBreeze, v3.asBreeze) [error] ^ [error] /Users/chengpan/Projects/apache-spark/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala:74:48: diverging implicit expansion for type breeze.linalg.squaredDistance.Impl2[breeze.linalg.Vector[Double],breeze.linalg.Vector[Double],VR] [error] starting with method castOps in trait VectorOps [error] val squaredDist = breezeSquaredDistance(v2.asBreeze, v4.asBreeze) [error] ^ [error] four errors found [warn] In the last 9 seconds, 5.224 (60.3%) were spent in GC. [Heap: 2.85GB free of 3.80GB, max 3.80GB] Consider increasing the JVM heap using -Xmx or try a different collector, e.g. -XX:+UseG1GC, for better performance. [error] (mllib / Test / compileIncremental) Compilation failed [error] Total time: 341 s (05:41), completed 2021-5-4 20:10:48

The compile failed code is here https://github.com/pan3793/spark/blob/SPARK-35310/mllib/src/test/scala/org/apache/spark/mllib/util/MLUtilsSuite.scala

And to reproduce the issue, run build/mvn clean compile -DskipTests or build/sbt compile test:compile.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/scalanlp/breeze/issues/810, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACLIN47SGUB6IHRXAVDZDTL7TZHANCNFSM44CUNJXA .