rsparkling requires internet access for installation?

[exalate-issue-sync[bot]]

Reported by a user:

We are contacting you because we are experiencing issues installing and executing rsparkling.

For security reasons, we have to work on a closed environment, therefore we cannot access and download packages via classic gradle or maven. We have downloaded and install rsparkling and all dependencies (sparklyr, sparkling-water…) manually.

Without this add-on our spark jobs runs just fine. But when using it, rsparkling tries to get some packages from distant repositories, which it cannot access.

When running the R command with rsparkling package attached : “sc <- spark_connect (master = "yarn-client")”, we obtain the following error:

{code} Error in start_shell(master = master, spark_home = spark_home, spark_version = version, :

Failed to launch Spark shell. Ports file does not exist.

Path: /usr/hdp/2.4.2.0-258/spark/bin/spark-submit

Parameters: --class, sparklyr.Backend, --packages, 'ai.h2o:sparkling-water-core_2.10:1.6.7','ai.h2o:sparkling-water-ml_2.10:1.6.7','ai.h2o:sparkling-water-repl_2.10:1.6.7', '/usr/lib64/RRO-3.2.2/R-3.2.2/lib64/R/library/sparklyr/java/sparklyr-1.6-2.10.jar', /tmp/RtmpG6NNO8/fileb1211dcb5243.out

Ivy Default Cache set to: /home/a454275/.ivy2/cache

The jars for the packages stored in: /home/a454275/.ivy2/jars

:: loading settings :: url = jar:file:/usr/hdp/2.4.2.0-258/spark/lib/spark-assembly-1.6.1.2.4.2.0-258-hadoop2.7.1.2.4.2.0-258.jar!/org/apache/ivy/core/settings/ivysettings.xml

ai.h2o#sparkling-water-core_2.10 added as a dependency

ai.h2o#sparkling-water-ml_2.10 added as a dependency

ai.h2o#sparkling-water-repl_2.10 added as a dependency

:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0

            confs: [default]

You probably access the destination {code}

[exalate-issue-sync[bot]]

Navdeep commented: [~accountid:557058:afd6e9a4-1891-4845-98ea-b5d34a2bc42c] Don't totally understand this JIRA. It seems it is Sparklyr related, i.e., the port file

[exalate-issue-sync[bot]]

Navdeep commented: Next release will have the ability to point to a custom jar. So hopefully that helps this problem.

[DinukaH2O]

JIRA Issue Migration Info

Jira Issue: SW-300 Assignee: Navdeep Gill Reporter: Erin LeDell State: Resolved Fix Version: N/A Attachments: N/A Development PRs: N/A

[hasithjp]

JIRA Issue Migration Info Cont'd

Jira Issue Created Date: 2016-12-31T18:24:05.313-0800