aliceranzhou
commented
8 years ago That makes sense.
If NER3Classifier were an object, we could pass the classifier file path to every instance of the classifier method. But it would be loaded many times then..
I've changed the NER3Classifier from an object to a class to be passed around. The driver 'creates' the NER3Classifier class with a specific classifier, but the classifier file is loaded 'locally' by worker nodes (when classifier.classify() is first called). Would this work?
Try this?
import org.warcbase.spark.matchbox.NER3Classifier
val classifier = new NER3Classifier("hdfs:///user/jrwiebe/english.all.3class.distsim.crf.ser.gz")
val entities = sc.textFile("hdfs:///user/jrwiebe/cpp.text-greenparty/part-m-00000")
.map(_.split("\t"))
.map(e => (e(0), e(1), classifier.classify(e(2))))
val entities = rdd.map(r => (r._1, r._2, classifier_.classify(r._3)))
entities.take(3).foreach(println)
jrwiebe
lintool
This issue refers to @aliceranzhou's ner3classifier branch. Also, I'm a novice at Scala and Spark, so forgive me if what I've labelled an issue can be chalked up to my ignorance.
I tried writing a straight port of my
extract-entities-from-scrape-text.pigscript to test NER3Classifier.scala on the cluster. It failed, it would appear because the NER3Classifier object is not serializable.I ran the following in the Spark shell:
I received the following error messages:
I eventually concluded that the problem was with evaluation of the second map closure. This seemed to be confirmed when I pasted
NER3Classifier.scala(minus thepackagedeclaration) into the Spark shell, replacing the import statement in my script. In this instance a "Task not serializable" exception was thrown instead of a NullPointerException. (I assume the difference is due to the information available to the interpreter, as opposed to what the bytecode produced by the compiler.)I tried modifying NER3Classifier, but I didn't get anywhere. It's not even clear to me that I'm taking the right approach. It seems to me that rather than the driver creating the NER3Classifier object, loading the classifier file, and somehow getting serialized and then passed to the worker nodes, each worker node should rather get a function that would create the NER3Classifier object 'locally'. I tried this by passing the map a function, but I got the same "Task not serializable" error.