I am trying to use sparkxgb package in a yarn session, the problem is that when I try to create a spark session it fails with the following message:
spark <- spark_connect(master = "yarn-client",version = "2.3.0",app_name = "test", config=config) [main] INFO com.unraveldata.agent.ResourceCollector - Unravel Sensor 4.5.0.8_dev+origin_fix_hive_on_spark_4.5.0.8+103/1.3.11.3 initializing. Ivy Default Cache set to: /home/cdsw/.ivy2/cache The jars for the packages stored in: /home/cdsw/.ivy2/jars :: loading settings :: url = jar:file:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml ml.dmlc#xgboost4j-spark added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-0e80d9c5-936d-4e4a-b31b-6b950c29d1f9;1.0 confs: [default]
`Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (96811): Gateway in localhost:8880 did not respond.
Path: /opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/bin/spark-submit
Parameters: --class, sparklyr.Shell, --jars, '/home/cdsw/R/sparkxgb/java/sparkxgb-2.3-2.11.jar', --packages, 'ml.dmlc:xgboost4j-spark:0.81', '/home/cdsw/R/sparklyr/java/sparklyr-2.3-2.11.jar', 8880, 96811
Log: /tmp/RtmpesTPnW/file13d1275ad76_spark.log
Hello,
I am trying to use sparkxgb package in a yarn session, the problem is that when I try to create a spark session it fails with the following message:
spark <- spark_connect(master = "yarn-client",version = "2.3.0",app_name = "test", config=config) [main] INFO com.unraveldata.agent.ResourceCollector - Unravel Sensor 4.5.0.8_dev+origin_fix_hive_on_spark_4.5.0.8+103/1.3.11.3 initializing. Ivy Default Cache set to: /home/cdsw/.ivy2/cache The jars for the packages stored in: /home/cdsw/.ivy2/jars :: loading settings :: url = jar:file:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml ml.dmlc#xgboost4j-spark added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-0e80d9c5-936d-4e4a-b31b-6b950c29d1f9;1.0 confs: [default]`Error in force(code) : Failed while connecting to sparklyr to port (8880) for sessionid (96811): Gateway in localhost:8880 did not respond. Path: /opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/bin/spark-submit Parameters: --class, sparklyr.Shell, --jars, '/home/cdsw/R/sparkxgb/java/sparkxgb-2.3-2.11.jar', --packages, 'ml.dmlc:xgboost4j-spark:0.81', '/home/cdsw/R/sparklyr/java/sparklyr-2.3-2.11.jar', 8880, 96811 Log: /tmp/RtmpesTPnW/file13d1275ad76_spark.log
---- Output Log ----
---- Error Log ----`