Loading and saving matrices does not seem to be working due to NoClassDefFoundError: net/jpountz/lz4/LZ4BlockInputStream

[DanielTakeshi]

John,

There seems to be some new issue with loading/saving matrices. Currently on the master branch of BIDMach/BIDMat (for both my Mac 10.9 laptop and on stout), I get some NoClassDefFoundErrors when loading/saving matrices by following the BIDMat Loading/Saving Wiki page:

[seita@stout BIDMach]$ ./bidmach
Loading /home/seita/BIDMach/lib/bidmach_init.scala...
import BIDMat.{CMat, CSMat, DMat, Dict, FMat, FND, GMat, GDMat, GIMat, GLMat, GSMat, GSDMat, HMat, IDict, Image, IMat, LMat, Mat, SMat, SBMat, SDMat}
import BIDMat.MatFunctions._
import BIDMat.SciFunctions._
import BIDMat.Solvers._
import BIDMat.Plotting._
import BIDMach.Learner
import BIDMach.models.{FM, GLM, KMeans, KMeansw, LDA, LDAgibbs, Model, NMF, SFA, RandomForest}
import BIDMach.networks.DNN
import BIDMach.datasources.{DataSource, MatDS, FilesDS, SFilesDS}
import BIDMach.mixins.{CosineSim, Perplexity, Top, L1Regularizer, L2Regularizer}
import BIDMach.updaters.{ADAGrad, Batch, BatchNorm, IncMult, IncNorm, Telescoping}
import BIDMach.causal.IPTW
4 CUDA devices found, CUDA version 6.5

Welcome to Scala version 2.11.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51).
Type in expressions to have them evaluated.
Type :help for more information.

scala> saveFMat("hi", rand(3,3))
java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4BlockInputStream
  at BIDMat.MatFunctions$.saveFMat(MatFunctions.scala:1852)
  ... 33 elided
Caused by: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4BlockInputStream
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  ... 34 more

This is on my laptop:

scala> val a = loadSMat("moods_data/moodsByUser.smat.lz4")
java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4BlockInputStream
  at BIDMat.MatFunctions$.loadSMat(MatFunctions.scala:1846)
  ... 33 elided
Caused by: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4BlockInputStream
  at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 34 more

In the past, I tried saving matrices by using the HMat.saveFMatTxt("name", a) workaround but that suddenly gives me the same error.

Do you know if a recent update changed this loading/saving process, or is there something I need to download? Thanks.

-Daniel

[jcanny]

I upgraded the lz4 jar version a few days ago. The latest release (1.0.2) has the new one. You can either download and unpack 1.0.2, or run ./sbt package which will pick up the new jar and put it in your local ivy cache. You need to copy it from there into BIDMat/lib and BIDMach/lib

[DanielTakeshi]

Thank you, I think I figured it out. Apparently, when I compile with ./sbt package, it was not copying over the lz4-1.3.jar file. As a result, I had been using lz4-1.1.2.jar, which is clearly out of date. I finally found out where in the ivy cache it was stored, then I could copy:

dhcp-46-165:BIDMat danielseita$ cp ~/.ivy2/cache/net.jpountz.lz4/lz4/jars/lz4-1.3.jar lib/

And now saving/loading should work, though some methods need HMat. appended at the start.