search

etiennebr / sparksf

Enable spatial functions in Spark through the `sparklyr` package
8 stars 1 forks source link

Unresolved dependencies #1

Open etiennebr opened 5 years ago

etiennebr commented 5 years ago

Running 

library(sparksf)
library(sparklyr)

sc <- spark_connect(master = "local")

I get an error because of unresolved dependencies

[...]
                ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: org.geotools#gt-main;18.0: not found

        :: org.geotools#gt-api;18.0: not found

        :: org.geotools#gt-cql;18.0: not found

        :: org.geotools#gt-transform;18.0: not found

        :: org.locationtech.sfcurve#sfcurve-zorder_2.11;0.2.0: not found
[...]
etiennebr commented 5 years ago

I turned to the console and it seems I can get some dependencies resolved by adding repos.

/home/etienne/spark/spark-2.3.2-bin-hadoop2.7/bin/spark-submit --class sparklyr.Shell --repositories 'https://repo.boundlessgeo.com/main','http://download.osgeo.org/webdav/geotools','http://conjars.org/repo','http://maven.geotoolkit.org','https://repo.locationtech.org/content/groups/releases' --packages 'org.locationtech.geomesa:geomesa-fs-dist_2.11:2.1.1' '/home/etienne/R/x86_64-pc-linux-gnu-library/3.5/sparklyr/java/sparklyr-2.3-2.11.jar' 8880 78052

However I still have an issue with finding javax.media#jai_core;1.1.3!jai_core.jar (only tries one repo)

Seems to be related to https://github.com/rstudio/sparklyr/issues/#1476

etiennebr commented 5 years ago

Isolating the jai_core ...

/home/etienne/spark/spark-2.3.2-bin-hadoop2.7/bin/spark-submit \
  --class sparklyr.Shell \
  --repositories 'http://download.osgeo.org/webdav/geotools' \
  --packages 'javax.media:jai_core:1.1.3' \
  '/home/etienne/R/x86_64-pc-linux-gnu-library/3.5/sparklyr/java/sparklyr-2.3-2.11.jar' 8880 78052

It seems it doesn't try the other repos provided... but it's there: http://download.osgeo.org/webdav/geotools/javax/media/jai_core/1.1.3/

http://download.osgeo.org/webdav/geotools added as a remote repository with the name: repo-1
Ivy Default Cache set to: /home/etienne/.ivy2/cache
The jars for the packages stored in: /home/etienne/.ivy2/jars
:: loading settings :: url = jar:file:/home/etienne/spark/spark-2.3.2-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
javax.media#jai_core added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-33f4fc38-54df-4e19-a3c3-a06b5772db61;1.0
        confs: [default]
        found javax.media#jai_core;1.1.3 in central
:: resolution report :: resolve 236ms :: artifacts dl 732ms
        :: modules in use:
        javax.media#jai_core;1.1.3 from central in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
                [NOT FOUND  ] javax.media#jai_core;1.1.3!jai_core.jar (724ms)

        ==== central: tried

          https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

                ::              FAILED DOWNLOADS            ::

                :: ^ see resolution messages for details  ^ ::

                ::::::::::::::::::::::::::::::::::::::::::::::

                :: javax.media#jai_core;1.1.3!jai_core.jar

                ::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: javax.media#jai_core;1.1.3!jai_core.jar]
        at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1303)
        at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:53)
        at org.apache.spark.deploy.SparkSubmit$.doPrepareSubmitEnvironment(SparkSubmit.scala:364)
        at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:250)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:171)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Any idea @nicolasessisbreton can you replicate that?