search

AbsaOSS / spline

Data Lineage Tracking And Visualization Solution
https://absaoss.github.io/spline/
Apache License 2.0
599 stars 154 forks source link

Support for SBT-based client projects #160

Closed jcnorman48 closed 4 years ago

jcnorman48 commented 5 years ago

I get the error: unresolved dependency: org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version}: not found

when adding the dependencies for za.co.absa.spline:spline-core:0.3.6 I don't get this error with 0.3.5. I've tested this in SBT and gradle.

wajda commented 5 years ago

Can you give us more context? pom.xml, command line etc.

bhintermeier commented 5 years ago

Hi,

I get the same error for za.co.absa.spline:spline-core:0.3.6

My setup as follows:

Here´s my build.sbt:

name := "sparkSbtAssemblyTest"

version := "0.1"

scalaVersion := "2.11.12"

sparkVersion := "2.4.0"

sparkComponents ++= Seq("sql")

libraryDependencies += "za.co.absa.spline" % "spline-core" % "0.3.6"

libraryDependencies += "za.co.absa.spline" % "spline-core-spark-adapter-2.4" % "0.3.6"

libraryDependencies += "za.co.absa.spline" % "spline-persistence-mongo" % "0.3.6"

assemblyMergeStrategy in assembly := { case PathList("META-INF", xs @ _*) => MergeStrategy.discard case x => MergeStrategy.first }

Here the logs (after 'Refresh all sbt projects' in IntelliJ):


[info] Loading global plugins from /Users/benedikthintermeier/.sbt/0.13/plugins
[info] Loading project definition from /Users/benedikthintermeier/sparkSbtAssemblyTest/project
[info] Updating {file:/Users/benedikthintermeier/sparkSbtAssemblyTest/project/}sparksbtassemblytest-build...
[info] Resolving org.scala-sbt#global-plugins;0.0 ...
[info] Resolving org.spark-packages#sbt-spark-package;0.2.6 ...
[info] Resolving org.scalaj#scalaj-http_2.10;1.1.4 ...
[info] Resolving org.scala-lang#scala-library;2.10.7 ...
[info] Resolving org.apache.directory.studio#org.apache.commons.codec;1.8 ...
[info] Resolving commons-codec#commons-codec;1.8 ...
[info] Resolving com.eed3si9n#sbt-assembly;0.14.6 ...
[info] Resolving org.scalactic#scalactic_2.10;3.0.1 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.7 ...
[info] Resolving org.pantsbuild#jarjar;1.6.5 ...
[info] Resolving org.apache.ant#ant;1.9.9 ...
[info] Resolving org.apache.ant#ant-launcher;1.9.9 ...
[info] Resolving org.ow2.asm#asm;6.0 ...
[info] Resolving org.ow2.asm#asm-commons;6.0 ...
[info] Resolving org.ow2.asm#asm-tree;6.0 ...
[info] Resolving org.apache.maven#maven-plugin-api;3.3.9 ...
[info] Resolving org.apache.maven#maven-model;3.3.9 ...
[info] Resolving org.codehaus.plexus#plexus-utils;3.0.22 ...
[info] Resolving org.apache.commons#commons-lang3;3.4 ...
[info] Resolving org.apache.maven#maven-artifact;3.3.9 ...
[info] Resolving org.eclipse.sisu#org.eclipse.sisu.plexus;0.3.2 ...
[info] Resolving javax.enterprise#cdi-api;1.0 ...
[info] Resolving javax.annotation#jsr250-api;1.0 ...
[info] Resolving javax.inject#javax.inject;1 ...
[info] Resolving org.eclipse.sisu#org.eclipse.sisu.inject;0.3.2 ...
[info] Resolving org.codehaus.plexus#plexus-component-annotations;1.5.5 ...
[info] Resolving org.codehaus.plexus#plexus-classworlds;2.5.2 ...
[info] Resolving org.scala-sbt#sbt;0.13.17 ...
[info] Resolving org.scala-sbt#main;0.13.17 ...
[info] Resolving org.scala-sbt#actions;0.13.17 ...
[info] Resolving org.scala-sbt#classpath;0.13.17 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.7 ...
[info] Resolving org.scala-sbt#interface;0.13.17 ...
[info] Resolving org.scala-sbt#io;0.13.17 ...
[info] Resolving org.scala-sbt#control;0.13.17 ...
[info] Resolving org.scala-sbt#launcher-interface;1.0.1 ...
[info] Resolving org.scala-sbt#completion;0.13.17 ...
[info] Resolving org.scala-sbt#collections;0.13.17 ...
[info] Resolving jline#jline;2.14.5 ...
[info] Resolving org.scala-sbt#api;0.13.17 ...
[info] Resolving org.scala-sbt#classfile;0.13.17 ...
[info] Resolving org.scala-sbt#logging;0.13.17 ...
[info] Resolving org.scala-sbt#process;0.13.17 ...
[info] Resolving org.scala-sbt#compiler-integration;0.13.17 ...
[info] Resolving org.scala-sbt#incremental-compiler;0.13.17 ...
[info] Resolving org.scala-sbt#relation;0.13.17 ...
[info] Resolving org.scala-sbt#compile;0.13.17 ...
[info] Resolving org.scala-sbt#persist;0.13.17 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.10;0.4.2 ...
[info] Resolving org.scala-sbt#compiler-ivy-integration;0.13.17 ...
[info] Resolving org.scala-sbt#ivy;0.13.17 ...
[info] Resolving org.scala-sbt#cross;0.13.17 ...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-b18f59ea3bc914a297bb6f1a4f7fb0ace399e310 ...
[info] Resolving com.jcraft#jsch;0.1.50 ...
[info] Resolving org.scala-sbt#serialization_2.10;0.1.2 ...
[info] Resolving org.scala-lang.modules#scala-pickling_2.10;0.10.1 ...
[info] Resolving org.scalamacros#quasiquotes_2.10;2.0.1 ...
[info] Resolving org.json4s#json4s-core_2.10;3.2.10 ...
[info] Resolving org.json4s#json4s-ast_2.10;3.2.10 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving org.spire-math#jawn-parser_2.10;0.6.0 ...
[info] Resolving org.spire-math#json4s-support_2.10;0.6.0 ...
[info] Resolving org.scala-sbt#run;0.13.17 ...
[info] Resolving org.scala-sbt#task-system;0.13.17 ...
[info] Resolving org.scala-sbt#tasks;0.13.17 ...
[info] Resolving org.scala-sbt#tracking;0.13.17 ...
[info] Resolving org.scala-sbt#cache;0.13.17 ...
[info] Resolving org.scala-sbt#testing;0.13.17 ...
[info] Resolving org.scala-sbt#test-agent;0.13.17 ...
[info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] Resolving org.scala-sbt#main-settings;0.13.17 ...
[info] Resolving org.scala-sbt#apply-macro;0.13.17 ...
[info] Resolving org.scala-sbt#command;0.13.17 ...
[info] Resolving org.scala-sbt#template-resolver;0.1 ...
[info] Resolving org.scala-sbt#logic;0.13.17 ...
[info] Resolving org.scala-sbt#compiler-interface;0.13.17 ...
[info] Resolving org.scala-lang#jline;2.10.7 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]
[warn]  * com.eed3si9n:sbt-assembly:0.14.6 is selected over 0.14.3
[warn]      +- default:sparksbtassemblytest-build:0.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13) (depends on 0.14.6)
[warn]      +- org.spark-packages:sbt-spark-package:0.2.6 (scalaVersion=2.10, sbtVersion=0.13) (depends on 0.14.3)
[warn]
[warn] Run 'evicted' to see detailed eviction warnings
[info] Set current project to sparkSbtAssemblyTest (in build file:/Users/benedikthintermeier/sparkSbtAssemblyTest/)
>
[info] Defining */*:sbtStructureOptions, */*:sbtStructureOutputFile and 1 others.
[info] The new values will be used by *:cleanKeepFiles
[info]  Run `last` for details.
[info] Reapplying settings...
[info] Set current project to sparkSbtAssemblyTest (in build file:/Users/benedikthintermeier/sparkSbtAssemblyTest/)
[info] Applying State transformations org.jetbrains.sbt.CreateTasks from /Users/benedikthintermeier/Library/Application Support/IntelliJIdea2018.3/Scala/launcher/sbt-structure-0.13.jar
[info] Reapplying settings...
[info] Set current project to sparkSbtAssemblyTest (in build file:/Users/benedikthintermeier/sparkSbtAssemblyTest/)
[info] Updating {file:/Users/benedikthintermeier/sparkSbtAssemblyTest/}sparksbtassemblytest...
[info] Resolving org.scala-lang#scala-library;2.11.12 ...
[info] Resolving za.co.absa.spline#spline-core;0.3.6 ...
[info] Resolving za.co.absa.spline#spline-core-spark-adapter-api;0.3.6 ...
[info] Resolving za.co.absa.spline#spline-commons;0.3.6 ...
[info] Resolving commons-configuration#commons-configuration;1.10 ...
[info] Resolving commons-lang#commons-lang;2.6 ...
[info] Resolving commons-logging#commons-logging;1.1.1 ...
[info] Resolving commons-io#commons-io;2.6 ...
[info] Resolving org.scala-lang#scala-reflect;2.11.12 ...
[info] Resolving org.slf4s#slf4s-api_2.11;1.7.25 ...
[info] Resolving org.slf4j#slf4j-api;1.7.25 ...
[info] Resolving org.apache.spark#spark-core_2.11;${spark.version} ...
[info] Resolving org.apache.spark#spark-core_2.11;${spark.version} ...
[warn]  module not found: org.apache.spark#spark-core_2.11;${spark.version}
[warn] ==== local: tried
[warn]   /Users/benedikthintermeier/.ivy2/local/org.apache.spark/spark-core_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.11/${spark.version}/spark-core_2.11-${spark.version}.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /Users/benedikthintermeier/.sbt/preloaded/org.apache.spark/spark-core_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////Users/benedikthintermeier/.sbt/preloaded/org/apache/spark/spark-core_2.11/${spark.version}/spark-core_2.11-${spark.version}.pom
[warn] ==== Spark Packages Repo: tried
[warn]   https://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-core_2.11/${spark.version}/spark-core_2.11-${spark.version}.pom
[info] Resolving org.apache.spark#spark-sql_2.11;${spark.version} ...
[info] Resolving org.apache.spark#spark-sql_2.11;${spark.version} ...
[warn]  module not found: org.apache.spark#spark-sql_2.11;${spark.version}
[warn] ==== local: tried
[warn]   /Users/benedikthintermeier/.ivy2/local/org.apache.spark/spark-sql_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-sql_2.11/${spark.version}/spark-sql_2.11-${spark.version}.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /Users/benedikthintermeier/.sbt/preloaded/org.apache.spark/spark-sql_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////Users/benedikthintermeier/.sbt/preloaded/org/apache/spark/spark-sql_2.11/${spark.version}/spark-sql_2.11-${spark.version}.pom
[warn] ==== Spark Packages Repo: tried
[warn]   https://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql_2.11/${spark.version}/spark-sql_2.11-${spark.version}.pom
[info] Resolving org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version} ...
[info] Resolving org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version} ...
[warn]  module not found: org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version}
[warn] ==== local: tried
[warn]   /Users/benedikthintermeier/.ivy2/local/org.apache.spark/spark-sql-kafka-0-10_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/${spark.version}/spark-sql-kafka-0-10_2.11-${spark.version}.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /Users/benedikthintermeier/.sbt/preloaded/org.apache.spark/spark-sql-kafka-0-10_2.11/${spark.version}/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////Users/benedikthintermeier/.sbt/preloaded/org/apache/spark/spark-sql-kafka-0-10_2.11/${spark.version}/spark-sql-kafka-0-10_2.11-${spark.version}.pom
[warn] ==== Spark Packages Repo: tried
[warn]   https://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql-kafka-0-10_2.11/${spark.version}/spark-sql-kafka-0-10_2.11-${spark.version}.pom
[info] Resolving za.co.absa.spline#spline-model;0.3.6 ...
[info] Resolving com.github.salat#salat-util_2.11;1.11.2 ...
[info] Resolving org.scala-lang#scalap;2.11.12 ...
[info] Resolving org.scala-lang#scala-compiler;2.11.12 ...
[info] Resolving org.scala-lang.modules#scala-xml_2.11;1.0.5 ...
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.0.4 ...
[info] Resolving org.json4s#json4s-native_2.11;3.2.9 ...
[info] Resolving org.json4s#json4s-core_2.11;3.2.9 ...
[info] Resolving org.json4s#json4s-ast_2.11;3.2.9 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.6 ...
[info] Resolving za.co.absa.spline#spline-persistence-api;0.3.6 ...
[info] Resolving commons-codec#commons-codec;1.11 ...
[info] Resolving com.databricks#spark-xml_2.11;0.4.1 ...
[info] Resolving org.scalaz#scalaz-core_2.11;7.2.27 ...
[info] Resolving za.co.absa.spline#spline-core-spark-adapter-2.4;0.3.6 ...
[info] Resolving za.co.absa.spline#spline-persistence-mongo;0.3.6 ...
[info] Resolving com.github.salat#salat-core_2.11;1.11.2 ...
[info] Resolving org.mongodb#casbah-core_2.11;3.1.1 ...
[info] Resolving org.mongodb#casbah-commons_2.11;3.1.1 ...
[info] Resolving com.github.nscala-time#nscala-time_2.11;1.0.0 ...
[info] Resolving joda-time#joda-time;2.3 ...
[info] Resolving org.joda#joda-convert;1.2 ...
[info] Resolving org.mongodb#mongo-java-driver;3.2.2 ...
[info] Resolving org.mongodb#casbah-query_2.11;3.1.1 ...
[info] Resolving org.apache.spark#spark-sql_2.11;2.4.0 ...
[info] Resolving com.univocity#univocity-parsers;2.7.3 ...
[info] Resolving org.apache.spark#spark-sketch_2.11;2.4.0 ...
[info] Resolving org.apache.spark#spark-tags_2.11;2.4.0 ...
[info] Resolving org.spark-project.spark#unused;1.0.0 ...
[info] Resolving org.apache.spark#spark-core_2.11;2.4.0 ...
[info] Resolving org.apache.avro#avro;1.8.2 ...
[info] Resolving org.codehaus.jackson#jackson-core-asl;1.9.13 ...
[info] Resolving org.codehaus.jackson#jackson-mapper-asl;1.9.13 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.7 ...
[info] Resolving org.apache.commons#commons-compress;1.8.1 ...
[info] Resolving org.tukaani#xz;1.5 ...
[info] Resolving org.apache.avro#avro-mapred;1.8.2 ...
[info] Resolving org.apache.avro#avro-ipc;1.8.2 ...
[info] Resolving commons-codec#commons-codec;1.9 ...
[info] Resolving com.twitter#chill_2.11;0.9.3 ...
[info] Resolving com.twitter#chill-java;0.9.3 ...
[info] Resolving com.esotericsoftware#kryo-shaded;4.0.2 ...
[info] Resolving com.esotericsoftware#minlog;1.3.0 ...
[info] Resolving org.objenesis#objenesis;2.5.1 ...
[info] Resolving org.apache.xbean#xbean-asm6-shaded;4.8 ...
[info] Resolving org.apache.hadoop#hadoop-client;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-common;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-annotations;2.6.5 ...
[info] Resolving com.google.guava#guava;11.0.2 ...
[info] Resolving com.google.code.findbugs#jsr305;1.3.9 ...
[info] Resolving commons-cli#commons-cli;1.2 ...
[info] Resolving xmlenc#xmlenc;0.52 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-io#commons-io;2.4 ...
[info] Resolving commons-net#commons-net;3.1 ...
[info] Resolving commons-collections#commons-collections;3.2.2 ...
[info] Resolving log4j#log4j;1.2.17 ...
[info] Resolving commons-configuration#commons-configuration;1.6 ...
[info] Resolving commons-digester#commons-digester;1.8 ...
[info] Resolving commons-beanutils#commons-beanutils;1.7.0 ...
[info] Resolving commons-beanutils#commons-beanutils-core;1.8.0 ...
[info] Resolving com.google.protobuf#protobuf-java;2.5.0 ...
[info] Resolving com.google.code.gson#gson;2.2.4 ...
[info] Resolving org.apache.hadoop#hadoop-auth;2.6.5 ...
[info] Resolving org.apache.httpcomponents#httpclient;4.2.5 ...
[info] Resolving org.apache.httpcomponents#httpcore;4.2.4 ...
[info] Resolving org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 ...
[info] Resolving org.apache.directory.server#apacheds-i18n;2.0.0-M15 ...
[info] Resolving org.apache.directory.api#api-asn1-api;1.0.0-M20 ...
[info] Resolving org.apache.directory.api#api-util;1.0.0-M20 ...
[info] Resolving org.apache.curator#curator-framework;2.6.0 ...
[info] Resolving org.apache.curator#curator-client;2.6.0 ...
[info] Resolving org.apache.zookeeper#zookeeper;3.4.6 ...
[info] Resolving jline#jline;0.9.94 ...
[info] Resolving junit#junit;3.8.1 ...
[info] Resolving org.apache.curator#curator-recipes;2.6.0 ...
[info] Resolving org.htrace#htrace-core;3.0.4 ...
[info] Resolving org.apache.hadoop#hadoop-hdfs;2.6.5 ...
[info] Resolving org.mortbay.jetty#jetty-util;6.1.26 ...
[info] Resolving xerces#xercesImpl;2.9.1 ...
[info] Resolving xml-apis#xml-apis;1.3.04 ...
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-app;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-common;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-yarn-common;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-yarn-api;2.6.5 ...
[info] Resolving javax.xml.bind#jaxb-api;2.2.2 ...
[info] Resolving javax.xml.stream#stax-api;1.0-2 ...
[info] Resolving org.codehaus.jackson#jackson-jaxrs;1.9.13 ...
[info] Resolving org.codehaus.jackson#jackson-xc;1.9.13 ...
[info] Resolving com.google.inject#guice;3.0 ...
[info] Resolving javax.inject#javax.inject;1 ...
[info] Resolving aopalliance#aopalliance;1.0 ...
[info] Resolving org.sonatype.sisu.inject#cglib;2.2.1-v20090111 ...
[info] Resolving org.apache.hadoop#hadoop-yarn-client;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-core;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-yarn-server-common;2.6.5 ...
[info] Resolving org.fusesource.leveldbjni#leveldbjni-all;1.8 ...
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.6.5 ...
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.6.5 ...
[info] Resolving org.apache.spark#spark-launcher_2.11;2.4.0 ...
[info] Resolving org.apache.spark#spark-kvstore_2.11;2.4.0 ...
[info] Resolving com.fasterxml.jackson.core#jackson-core;2.6.7 ...
[info] Resolving com.fasterxml.jackson.core#jackson-databind;2.6.7.1 ...
[info] Resolving com.fasterxml.jackson.core#jackson-annotations;2.6.7 ...
[info] Resolving org.apache.spark#spark-network-common_2.11;2.4.0 ...
[info] Resolving io.netty#netty-all;4.1.17.Final ...
[info] Resolving org.apache.commons#commons-lang3;3.5 ...
[info] Resolving io.dropwizard.metrics#metrics-core;3.1.5 ...
[info] Resolving org.apache.commons#commons-crypto;1.0.0 ...
[info] Resolving org.apache.spark#spark-network-shuffle_2.11;2.4.0 ...
[info] Resolving org.apache.spark#spark-unsafe_2.11;2.4.0 ...
[info] Resolving javax.activation#activation;1.1.1 ...
[info] Resolving javax.servlet#javax.servlet-api;3.1.0 ...
[info] Resolving org.apache.commons#commons-math3;3.4.1 ...
[info] Resolving org.slf4j#slf4j-api;1.7.16 ...
[info] Resolving org.slf4j#jul-to-slf4j;1.7.16 ...
[info] Resolving org.slf4j#jcl-over-slf4j;1.7.16 ...
[info] Resolving org.slf4j#slf4j-log4j12;1.7.16 ...
[info] Resolving com.ning#compress-lzf;1.0.3 ...
[info] Resolving org.xerial.snappy#snappy-java;1.1.7.1 ...
[info] Resolving org.lz4#lz4-java;1.4.0 ...
[info] Resolving com.github.luben#zstd-jni;1.3.2-2 ...
[info] Resolving org.roaringbitmap#RoaringBitmap;0.5.11 ...
[info] Resolving org.json4s#json4s-jackson_2.11;3.5.3 ...
[info] Resolving org.json4s#json4s-core_2.11;3.5.3 ...
[info] Resolving org.json4s#json4s-ast_2.11;3.5.3 ...
[info] Resolving org.json4s#json4s-scalap_2.11;3.5.3 ...
[info] Resolving com.thoughtworks.paranamer#paranamer;2.8 ...
[info] Resolving org.scala-lang.modules#scala-xml_2.11;1.0.6 ...
[info] Resolving org.glassfish.jersey.core#jersey-client;2.22.2 ...
[info] Resolving javax.ws.rs#javax.ws.rs-api;2.0.1 ...
[info] Resolving org.glassfish.jersey.core#jersey-common;2.22.2 ...
[info] Resolving javax.annotation#javax.annotation-api;1.2 ...
[info] Resolving org.glassfish.jersey.bundles.repackaged#jersey-guava;2.22.2 ...
[info] Resolving org.glassfish.hk2#hk2-api;2.4.0-b34 ...
[info] Resolving org.glassfish.hk2#hk2-utils;2.4.0-b34 ...
[info] Resolving org.glassfish.hk2.external#aopalliance-repackaged;2.4.0-b34 ...
[info] Resolving org.glassfish.hk2.external#javax.inject;2.4.0-b34 ...
[info] Resolving org.glassfish.hk2#hk2-locator;2.4.0-b34 ...
[info] Resolving org.javassist#javassist;3.18.1-GA ...
[info] Resolving org.glassfish.hk2#osgi-resource-locator;1.0.1 ...
[info] Resolving org.glassfish.jersey.core#jersey-server;2.22.2 ...
[info] Resolving org.glassfish.jersey.media#jersey-media-jaxb;2.22.2 ...
[info] Resolving javax.validation#validation-api;1.1.0.Final ...
[info] Resolving org.glassfish.jersey.containers#jersey-container-servlet;2.22.2 ...
[info] Resolving org.glassfish.jersey.containers#jersey-container-servlet-core;2.22.2 ...
[info] Resolving io.netty#netty;3.9.9.Final ...
[info] Resolving com.clearspring.analytics#stream;2.7.0 ...
[info] Resolving io.dropwizard.metrics#metrics-jvm;3.1.5 ...
[info] Resolving io.dropwizard.metrics#metrics-json;3.1.5 ...
[info] Resolving io.dropwizard.metrics#metrics-graphite;3.1.5 ...
[info] Resolving com.fasterxml.jackson.module#jackson-module-scala_2.11;2.6.7.1 ...
[info] Resolving com.fasterxml.jackson.module#jackson-module-paranamer;2.7.9 ...
[info] Resolving org.apache.ivy#ivy;2.4.0 ...
[info] Resolving oro#oro;2.0.8 ...
[info] Resolving net.razorvine#pyrolite;4.13 ...
[info] Resolving net.sf.py4j#py4j;0.10.7 ...
[info] Resolving org.apache.hadoop#hadoop-yarn-server-nodemanager;2.6.5 ...
[info] Resolving org.codehaus.jettison#jettison;1.1 ...
[info] Resolving org.apache.spark#spark-catalyst_2.11;2.4.0 ...
[info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11;1.1.0 ...
[info] Resolving org.codehaus.janino#janino;3.0.9 ...
[info] Resolving org.codehaus.janino#commons-compiler;3.0.9 ...
[info] Resolving org.antlr#antlr4-runtime;4.7 ...
[info] Resolving commons-codec#commons-codec;1.10 ...
[info] Resolving org.apache.orc#orc-core;1.5.2 ...
[info] Resolving org.apache.orc#orc-shims;1.5.2 ...
[info] Resolving io.airlift#aircompressor;0.10 ...
[info] Resolving org.apache.orc#orc-mapreduce;1.5.2 ...
[info] Resolving org.apache.parquet#parquet-column;1.10.0 ...
[info] Resolving org.apache.parquet#parquet-common;1.10.0 ...
[info] Resolving org.apache.parquet#parquet-format;2.4.0 ...
[info] Resolving org.slf4j#slf4j-api;1.7.22 ...
[info] Resolving org.apache.parquet#parquet-encoding;1.10.0 ...
[info] Resolving org.apache.parquet#parquet-hadoop;1.10.0 ...
[info] Resolving org.apache.parquet#parquet-jackson;1.10.0 ...
[info] Resolving org.apache.arrow#arrow-vector;0.10.0 ...
[info] Resolving org.apache.arrow#arrow-format;0.10.0 ...
[info] Resolving com.vlkan#flatbuffers;1.2.0-3f79e055 ...
[info] Resolving org.apache.arrow#arrow-memory;0.10.0 ...
[info] Resolving com.google.code.findbugs#jsr305;3.0.2 ...
[info] Resolving joda-time#joda-time;2.9.9 ...
[info] Resolving com.fasterxml.jackson.core#jackson-core;2.7.9 ...
[info] Resolving com.carrotsearch#hppc;0.7.2 ...
[info] Resolving jline#jline;2.14.3 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-core_2.11;${spark.version}: not found
[warn]  :: org.apache.spark#spark-sql_2.11;${spark.version}: not found
[warn]  :: org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version}: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn]  Note: Unresolved dependencies path:
[warn]      org.apache.spark:spark-core_2.11:${spark.version}
[warn]        +- za.co.absa.spline:spline-core-spark-adapter-api:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- za.co.absa.spline:spline-core:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- default:sparksbtassemblytest_2.11:0.1
[warn]      org.apache.spark:spark-sql_2.11:${spark.version}
[warn]        +- za.co.absa.spline:spline-core-spark-adapter-api:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- za.co.absa.spline:spline-core:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- default:sparksbtassemblytest_2.11:0.1
[warn]      org.apache.spark:spark-sql-kafka-0-10_2.11:${spark.version}
[warn]        +- za.co.absa.spline:spline-core-spark-adapter-api:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- za.co.absa.spline:spline-core:0.3.6 (/Users/benedikthintermeier/sparkSbtAssemblyTest/build.sbt#L11-12)
[warn]        +- default:sparksbtassemblytest_2.11:0.1
[trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output.
[trace] Stack trace suppressed: run 'last *:update' for the full output.
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11;${spark.version}: not found
[error] unresolved dependency: org.apache.spark#spark-sql_2.11;${spark.version}: not found
[error] unresolved dependency: org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version}: not found
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11;${spark.version}: not found
[error] unresolved dependency: org.apache.spark#spark-sql_2.11;${spark.version}: not found
[error] unresolved dependency: org.apache.spark#spark-sql-kafka-0-10_2.11;${spark.version}: not found
[error] Total time: 3 s, completed 02.04.2019 11:22:12```

Spark without Spline, with the above mentioned setup works fine. I´ve also tried all supported Spark Versions with Spline 0.3.5 without any problems.

Thanks in advance!

lmerchante commented 5 years ago

Hi wajda, same issue here. Working with scala sbt projects. Versions:

val scalaVersion = "2.11.12" val sparkVersion ="2.3.0"

Including your package on the build.sbt as:

    libraryDependencies += "za.co.absa.spline" % "spline-core" % "0.3.6"

fails with the error posted by jcnorman48. Replacing that line for:

    libraryDependencies += "za.co.absa.spline" % "spline-core" % "0.3.5"

compiles, packages and assemblies successfully.

Thanks!

wajda commented 5 years ago

Looks like a bug in SBT - https://github.com/sbt/sbt/issues/647 We didn't test Spline with SBT based projects. Need to investigate if it's possible to place a workaround.

wajda commented 4 years ago

Related to #420

wajda commented 4 years ago

fixed in #212 Relese 0.5.0