Closed razrLeLe closed 7 years ago
Is your input corpus in LIBSVM format? And what's your command arguments used?
Yes, I used libsvm format, and the submit command is as follows:
spark-submit --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs://goyoo/user/yuyue/log --executor-cores 4 --num-executors 21 --driver-memory 4G --executor-memory 4G --master yarn-client --class com.github.cloudml.zen.examples.ml.LDADriver zen-examples-0.3-SNAPSHOT-spark1.6.1.jar -numPartitions=21 -LDAAlgorithm=LightLDA -numThreads=16 -numTopics=500 -alpha=0.01 -beta=0.01 -alphaAS=1.0 -totalIter=1500 hdfs://goyoo/user/yuyue/10w_doc.libsvm hdfs://goyoo/user/yuyue/zen_10w_result
@bhoppi Thanks so much for your help, I finally find out there is something wrong with pretreatment of the corpus, which caused every word count of every document is zero, and then got the exception.
I tried with 10k-80k doc-word dataset, the program worked, but when I tested with 100k-160k doc-word dataset the program failed with exception:
I am wondering whether it is necessary to specify some configuration or meet up with some equipment, so could you please tell me more detail about your configuration when works with billions of documents?
And thanks so much for your awesome work!