search

EpiQuant / SEMS-Spark

0 stars 0 forks source link

Breeze library dependency collision #1

Open jacobrh91 opened 7 years ago

jacobrh91 commented 7 years ago

When testing the RDD prototype on aForge, it appears that two different versions of breeze are part of the dependency tree (one direct and one is a dependency of another library we explicitly depend on).

The jar file includes only on of the two, and apparently the one it includes apparently does not have the inv function used in the prototype (the error below is given when the program runs)

Caused by: java.lang.NoSuchMethodError: breeze.linalg.inv$.canInvUsingLU_Double(Lbreeze/generic/UFunc$UImpl;)Lbreeze/generic/UFunc$UImpl; at statistics.OLSRegression.<init>(OLSRegression.scala:35)

It may be that its looking for a native implementation although we need it to default to the java-based implementation. When running locally this happens automatically. Maybe it doesn't in the context of a real spark cluster

jacobrh91 commented 7 years ago

Possible clue: When I add breeze 0.11.2 explicitly (the same version in the spark-core 2.0.0 installed on aForge) I get the same error that I get if I do not explicitly import any library of breeze (Confirming that spark-core 2.0.0 has this breeze installation present on aForge)

But, when I explicitly include the latest version of breeze (0.13.2), it compiles but then gets the runtime error. My hypothesis is that during compilation it uses the newest breeze, but then looks at the other one during runtime and this is causing the error.