Stack overflow error during training

Hi, I'm going through the Quick Start material in the Apache PredictionIO site, and managed to get into a successful build. However, the next step - train - had an error. I did not deviate from the instructions.

Can anyone help pinpoint the root cause and how to fix? TIA.

et@et-VirtualBox:~/apache-predictionio/PredictionIO-0.12.0-incubating/engines/MyRecommendation$ pio train [INFO] [Runner$] Submission command: /home/et/apache-predictionio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6/bin/spark-submit --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/postgresql-42.0.0.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/engines/MyRecommendation/target/scala-2.11/template-scala-parallel-recommendation_2.11-0.1-SNAPSHOT.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/engines/MyRecommendation/target/scala-2.11/template-scala-parallel-recommendation-assembly-0.1-SNAPSHOT-deps.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-elasticsearch-assembly-0.12.0-incubating.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-hbase-assembly-0.12.0-incubating.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-hdfs-assembly-0.12.0-incubating.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-jdbc-assembly-0.12.0-incubating.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-s3-assembly-0.12.0-incubating.jar,file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/spark/pio-data-localfs-assembly-0.12.0-incubating.jar --files file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/conf/log4j.properties --driver-class-path /home/et/apache-predictionio/PredictionIO-0.12.0-incubating/conf:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/postgresql-42.0.0.jar --driver-java-options -Dpio.log.dir=/home/et file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/lib/pio-assembly-0.12.0-incubating.jar --engine-id org.example.recommendation.RecommendationEngine --engine-version c45b9862c196bf5bf27f8b706da661e1e50e9bc1 --engine-variant file:/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/engines/MyRecommendation/engine.json --verbosity 0 --json-extractor Both --env PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/et/.pio_store,PIO_HOME=/home/et/apache-predictionio/PredictionIO-0.12.0-incubating,PIO_FS_ENGINESDIR=/home/et/.pio_store/engines,PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio,PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc,PIO_FS_TMPDIR=/home/et/.pio_store/tmp,PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=PGSQL,PIO_CONF_DIR=/home/et/apache-predictionio/PredictionIO-0.12.0-incubating/conf [INFO] [Engine] Extracting datasource params... [INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be used. [INFO] [Engine] Datasource params: (,DataSourceParams(MyApp1,None)) [INFO] [Engine] Extracting preparator params... [INFO] [Engine] Preparator params: (,Empty) [INFO] [Engine] Extracting serving params... [INFO] [Engine] Serving params: (,Empty) [WARN] [Utils] Your hostname, et-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3) [WARN] [Utils] Set SPARK_LOCAL_IP if you need to bind to another address [INFO] [log] Logging initialized @22682ms [INFO] [Server] jetty-9.2.z-SNAPSHOT [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5261ec9{/jobs,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@65bb6275{/jobs/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@64da6cbd{/jobs/job,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@72a2312e{/jobs/job/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@47c7a9e5{/stages,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7951c3a2{/stages/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4ed9f7b1{/stages/stage,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2d119405{/stages/stage/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@57e388c3{/stages/pool,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@20580d4e{/stages/pool/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@21bd128b{/storage,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2d0b05{/storage/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4b432e53{/storage/rdd,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2b916808{/storage/rdd/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@62f11ebb{/environment,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@a6c54c3{/environment/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@580fd26b{/executors,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@78fe204a{/executors/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1290ed28{/executors/threadDump,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@7a3269f5{/executors/threadDump/json,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5c887052{/static,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@55fdf7f9{/,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4091b9c3{/api,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@29050de5{/jobs/job/kill,null,AVAILABLE,@Spark} [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3610f277{/stages/stage/kill,null,AVAILABLE,@Spark} [INFO] [ServerConnector] Started Spark@27ca01e6{HTTP/1.1}{0.0.0.0:4040} [INFO] [Server] Started @23583ms [INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@11787b64{/metrics/json,null,AVAILABLE,@Spark} [INFO] [Engine$] EngineWorkflow.train [INFO] [Engine$] DataSource: org.example.recommendation.DataSource@58833798 [INFO] [Engine$] Preparator: org.example.recommendation.Preparator@64e7d698 [INFO] [Engine$] AlgorithmList: List(org.example.recommendation.ALSAlgorithm@401c4250) [INFO] [Engine$] Data sanity check is on. [INFO] [Engine$] org.example.recommendation.TrainingData does not support data sanity check. Skipping check. [INFO] [Engine$] org.example.recommendation.PreparedData does not support data sanity check. Skipping check. [Stage 1:=======================================> (2 + 1) / 3][WARN] [Executor] 1 block locks were not released by TID = 3: [rdd_1_3] [WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS [WARN] [BLAS] Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS [WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK [WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK [Stage 44:> (0 + 0) / 2][ERROR] [Executor] Exception in task 0.0 in stage 54.0 (TID 114) [ERROR] [SparkUncaughtExceptionHandler] Uncaught exception in thread Thread[Executor task launch worker for task 114,5,main] [WARN] [TaskSetManager] Lost task 0.0 in stage 54.0 (TID 114, localhost, executor driver): java.lang.StackOverflowError at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:3120) at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1676) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1548) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2282) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2206) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2064) pio train error

Sorry this repo is still visible. The code here has been moved and updated. PredictionIO moved to Apache and ActionML took over the Universal Recommender.

Find the PIO 0.12.0 version here in the UR 0.7.0-SNAPSHOT branch. We are waiting for a new Apache Mahout build before we release the UR v0.7.0 but you can build everything following the instructions in the README.md here: https://github.com/actionml/universal-recommender/tree/0.7.0-SNAPSHOT https://github.com/actionml/universal-recommender/tree/0.7.0-SNAPSHOT

Notice that for now you have to build our branch of Apache Mahout. I’m a committer and the code is in the main Apache repo but best to build from the instructions here: https://github.com/actionml/mahout https://github.com/actionml/mahout

Note that in addition to pio 0.12.0 we require Elasticsearch 5.x, Spark 2.2, and Scala 2.11

On Dec 5, 2017, at 9:23 PM, etandes notifications@github.com wrote:

Can anyone help pinpoint the root cause and how to fix? TIA.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/PredictionIO/template-scala-parallel-universal-recommendation/issues/62, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT8SxeWpU8qa2NBDwiBgoUd-_FyZ4T7ks5s9iTtgaJpZM4Q3Y3k.

PredictionIO / template-scala-parallel-universal-recommendation

Stack overflow error during training #62