Not able to train SOMs

[ankitksharma]

Hi,

I am trying to create a SOM model using the following code:

val model = SOM.train(xdim = 5, ydim = 5, examples = ffRDD, iterations = 10, fdim = 3)

Unfortunately I'm getting the following error. Can you please provide some documentation through which I can understand if I'm doing anything wrong in building the model.

` Exception in thread "main" java.io.NotSerializableException: com.redhat.et.silex.util.SampleSink Serialization stack:

• object not serializable (class: com.redhat.et.silex.util.SampleSink, value: SampleSink(count=0, min=Infinity, max=-Infinity, mean=0.0, variance=NaN))
• field (class: com.redhat.et.silex.som.SOM, name: mqsink, type: class com.redhat.et.silex.util.SampleSink)
• object (class com.redhat.et.silex.som.SOM, com.redhat.et.silex.som.SOM@7402c49f) at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40) at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$2.apply(TorrentBroadcast.scala:236) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$blockifyObject$2.apply(TorrentBroadcast.scala:236) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1287) at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:237) at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:107) at org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:86) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56) at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1370) at com.redhat.et.silex.som.SOM$$anonfun$train$1.apply(som.scala:204) at com.redhat.et.silex.som.SOM$$anonfun$train$1.apply(som.scala:201) at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144) at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144) at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:105) at com.redhat.et.silex.som.SOM$.train(som.scala:201) at com.datarpm.dsl.som.som$.main(som.scala:42) at com.datarpm.dsl.som.som.main(som.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) `

I'm using Spark v2.0.0 & Scala v2.10.4

Thanks for your time.

[willb]

@ankitksharma thanks for the report! It's a long, embarrassing story but it was a quick fix. I'll cut a new release ASAP.

[willb]

@ankitksharma v0.1.1 should solve this issue. Please try it out and let me know if you have any problems!

[ankitksharma]

@willb This is awesome, its working now. Thanks a lot for quickly resolving this. Appreciate the effort.

[willb]

@ankitksharma Thanks again! I'm planning some improvements to this code in the future, so please stay tuned if it turns out to be useful.

[ankitksharma]

@willb That would be great. Although I'm able to run the code but I'm not able to figure out the result yet. I am trying to do clustering using SOM and check the accuracy using a labelled dataset. Similar to example given in R library "Kohonen". If you know R language then following is the code:

data(wines)
set.seed(7)

training <- sample(nrow(wines), 120)
Xtraining <- scale(wines[training, ])
Xtest <- scale(wines[-training, ],
               center = attr(Xtraining, "scaled:center"),
               scale = attr(Xtraining, "scaled:scale"))

som.wines <- som(Xtraining, grid = somgrid(5, 5, "hexagonal"))

som.prediction <- predict(som.wines, newdata = Xtest,
                          trainX = Xtraining,
                          trainY = factor(wine.classes[training]))
table(wine.classes[-training], som.prediction$prediction)
plot(som.wines)

This gives me the following confusion matrix

SOM plot which I get from this

Now when I try to run your library on the same dataset in Spark and using the following config:

val model = SOM.train(xdim = 7, ydim = 7, examples = xtrain, iterations = 100, fdim = data.columns.length)

I get only one type of response from model.closestWithSimilarity() function

(48, 1)

I am not able to understand this result. Can you throw some light on this? If you have some sort of documentation which can help me out in getting results close to what R library does it will do the job for me.

Thanks a lot for the help by the way.