search

scalanlp / breeze

Breeze is a numerical processing library for Scala.
www.scalanlp.org
Apache License 2.0
3.44k stars 691 forks source link

Not sure where this particular issue lies: either in Spark, Breeze or Netlib. But using breeze functions in EMR doesn't load native libraries. #856

Open dineshdharme opened 9 months ago

dineshdharme commented 9 months ago

I had to jump through hoops to get native libraries to load when calling breeze functions (like leastSquares) in Apache Spark. Not sure where the exact issue is. I suppose Spark uses its own shaded breeze. Locally there was no issue. But on EMR, I got warning like

WARNING: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS

Here's the work around that made it work.

https://gist.github.com/dineshdharme/8bd39cdbc35a09033b9ba2cfd1bdf146