search

IBMPredictiveAnalytics / K_Means_with_MLlib

SPSS Modeler Extension to execute PySpark MLlib implementation of K-Means Clustering
Apache License 2.0
2 stars 5 forks source link

Error in SparkListener #1

Closed yiru01 closed 8 years ago

yiru01 commented 8 years ago

Hello Sir, I tried to run sample of K Means with MLlib on SPSS 18 locally, but it didn't work. Would you tell if I need to install Spark in advance? Thank you.

Error

ghost commented 8 years ago

Hi,

You don't need to install spark, a version of spark is installed with Modeler 18

Please can you let me know what you have in your config/options.cfg for setting eas_pyspark_python which appears at the end of the file and is set to empty string by default. If connecting to modeler server, you'll need to modify this file on modeler server.

I have the following (I believe that forward slashes should be used)

eas_pyspark_python_path, "C:/Anaconda2/python.exe"

Best Niall

yiru01 commented 8 years ago

Hi Niall,

Thanks for your quick response. I set a path in the end of eas_pyspark_python in advance, but the result was error of SparkListener.

I checked python 2.7.10 / Anaconda 2.3.0(64-bit) can be run through the path. My environment is SPSS Modeler standalone without Modeler Server. I executed 5 provided MLlib models, and most of them have same error.

Best, Yiru

ghost commented 8 years ago

Hi Yiru,

Thanks for the information - your setting of eas_pyspark_python_path in options.cfg looks correct so that is not the problem... I will try and figure out what else could cause this error and get back to you with next steps...

Best Niall

yiru01 commented 8 years ago

Hi Niall,

Thank you. I'll try installing SPSS in different PC to find out if there is same problem. Spark on SPSS is really great function that I'm looking forward to use it.

Best,

Yiru

ghost commented 8 years ago

Hi Yiru,

Any luck installing on another machine? We hope to try and figure this out soon.

Would you be able to tell me which version of windows you have (and which locale you are running in) please?

Also please can you paste in a listing of the files in folder /spark/lib

Thanks and Best Regards Niall

yiru01 commented 8 years ago

Hi Niall,

I figured out the error was caused by VirtualStore. So I set eas_pyspark_python_path again and it works well. Thank you for taking time.

FYI: I tried all 5 models(KMeans, CF, Pagerank, Naive Bayes, Boosted Trees) on Windows7 PC locally. All shown good result. For the next step, I'll try connecting to modeler server.

Best,

Yiru

ghost commented 8 years ago

Hi Yiru,

OK thats great news.... I will close this issue.

Let us know if you hit other problems or if you have any ideas or requests....

Best Niall

brpalave commented 4 years ago

Hi Niall,

I figured out the error was caused by VirtualStore. So I set eas_pyspark_python_path again and it works well. Thank you for taking time.

FYI: I tried all 5 models(KMeans, CF, Pagerank, Naive Bayes, Boosted Trees) on Windows7 PC locally. All shown good result. For the next step, I'll try connecting to modeler server.

Best,

Yiru

excuse me , i have the same problem , how did you solve it? Thanks in advanced.