search

AbsaOSS / spline-getting-started

Apache License 2.0
23 stars 17 forks source link

unknown resolver null #42

Open dulangaheshan opened 1 year ago

dulangaheshan commented 1 year ago

I'm follwing https://absaoss.github.io/spline/#step-by-step and trying to get understand this. so i was able to get Spline up and running using docker-compose

as per the dicumentation [http://localhost:9090/]( Spline UI). and http://localhost:8080/ is running.

so i tried to with below command

pyspark \
  --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.producer.url=http://localhost:8080/producer"

In the terminal i was able to see below error

:: problems summary ::
:::: ERRORS
    unknown resolver null

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent-1bda6b26-985d-462f-b21b-9eb6a34a5555
    confs: [default]
    0 artifacts copied, 26 already retrieved (0kB/6ms)
23/02/06 17:15:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

then i tried below command to run small pyspark code

spark-submit \
  --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.producer.url=http://localhost:8080/producer" \
  ./schema.py`

the error unknown resolver null was there and but code executed successfully but i can't see any output http://localhost:9090/app/events/list here

any suggestions ??

wajda commented 1 year ago

"unknown resolver null" error is unlikely related to Spline, please double check your Spark installation. Also try to post a full log from pyspark, so we see more context.

dulangaheshan commented 1 year ago

please find the full log herewith for below command

 pyspark \
  --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.producer.url=http://localhost:8080/producer"
(base) d5han@d5han-HP-ProBook-440-G8-Notebook-PC:~/$ pyspark   --packages za.co.absa.spline.agent.spark:spark-3.0-spline-agent-bundle_2.12:1.0.2   --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener"   --conf "spark.spline.producer.url=http://localhost:8080/producer"
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
23/02/06 19:51:46 WARN Utils: Your hostname, d5han-HP-ProBook-440-G8-Notebook-PC resolves to a loopback address: 127.0.1.1; using 192.168.8.103 instead (on interface wlp2s0)
23/02/06 19:51:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
:: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/d5han/.ivy2/cache
The jars for the packages stored in: /home/d5han/.ivy2/jars
za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-1ace4748-b1af-4ad6-bee2-9f3744ef62c0;1.0
    confs: [default]
    found za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12;1.0.2 in central
    found za.co.absa.spline.agent.spark#agent-core_2.12;1.0.2 in central
    found org.scala-lang#scala-compiler;2.12.10 in local-m2-cache
    found org.scala-lang#scala-reflect;2.12.10 in local-m2-cache
    found org.scala-lang.modules#scala-xml_2.12;1.0.6 in spark-list
    found za.co.absa.commons#commons_2.12;1.3.3 in central
    found org.scalaj#scalaj-http_2.12;2.4.1 in central
    found io.github.classgraph#classgraph;4.8.116 in central
    found org.scala-graph#graph-core_2.12;1.12.5 in central
    found com.fasterxml.uuid#java-uuid-generator;4.0.1 in central
    found org.slf4j#slf4j-api;1.7.16 in spark-list
    found com.lihaoyi#fastparse_2.12;2.3.3 in central
    found com.lihaoyi#sourcecode_2.12;0.2.3 in central
    found com.lihaoyi#geny_2.12;0.6.10 in central
    found org.yaml#snakeyaml;1.33 in local-m2-cache
    found commons-configuration#commons-configuration;1.6 in spark-list
    found commons-collections#commons-collections;3.2.2 in spark-list
    found commons-lang#commons-lang;2.6 in spark-list
    found commons-logging#commons-logging;1.1.1 in local-m2-cache
    found commons-digester#commons-digester;1.8 in spark-list
    found commons-beanutils#commons-beanutils;1.9.4 in local-m2-cache
    found commons-beanutils#commons-beanutils-core;1.8.0 in spark-list
    found com.github.bigwheel#util-backports_2.12;2.1 in central
    found org.json4s#json4s-ext_2.12;3.6.10 in central
    found joda-time#joda-time;2.9.3 in spark-list
    found org.joda#joda-convert;2.2.1 in central
:: resolution report :: resolve 4507ms :: artifacts dl 7ms
    :: modules in use:
    com.fasterxml.uuid#java-uuid-generator;4.0.1 from central in [default]
    com.github.bigwheel#util-backports_2.12;2.1 from central in [default]
    com.lihaoyi#fastparse_2.12;2.3.3 from central in [default]
    com.lihaoyi#geny_2.12;0.6.10 from central in [default]
    com.lihaoyi#sourcecode_2.12;0.2.3 from central in [default]
    commons-beanutils#commons-beanutils;1.9.4 from local-m2-cache in [default]
    commons-beanutils#commons-beanutils-core;1.8.0 from spark-list in [default]
    commons-collections#commons-collections;3.2.2 from spark-list in [default]
    commons-configuration#commons-configuration;1.6 from spark-list in [default]
    commons-digester#commons-digester;1.8 from spark-list in [default]
    commons-lang#commons-lang;2.6 from spark-list in [default]
    commons-logging#commons-logging;1.1.1 from local-m2-cache in [default]
    io.github.classgraph#classgraph;4.8.116 from central in [default]
    joda-time#joda-time;2.9.3 from spark-list in [default]
    org.joda#joda-convert;2.2.1 from central in [default]
    org.json4s#json4s-ext_2.12;3.6.10 from central in [default]
    org.scala-graph#graph-core_2.12;1.12.5 from central in [default]
    org.scala-lang#scala-compiler;2.12.10 from local-m2-cache in [default]
    org.scala-lang#scala-reflect;2.12.10 from local-m2-cache in [default]
    org.scala-lang.modules#scala-xml_2.12;1.0.6 from spark-list in [default]
    org.scalaj#scalaj-http_2.12;2.4.1 from central in [default]
    org.slf4j#slf4j-api;1.7.16 from spark-list in [default]
    org.yaml#snakeyaml;1.33 from local-m2-cache in [default]
    za.co.absa.commons#commons_2.12;1.3.3 from central in [default]
    za.co.absa.spline.agent.spark#agent-core_2.12;1.0.2 from central in [default]
    za.co.absa.spline.agent.spark#spark-3.0-spline-agent-bundle_2.12;1.0.2 from central in [default]
    :: evicted modules:
    commons-logging#commons-logging;1.2 by [commons-logging#commons-logging;1.1.1] in [default]
    commons-logging#commons-logging;1.1 by [commons-logging#commons-logging;1.1.1] in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   28  |   1   |   1   |   2   ||   26  |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: ERRORS
    unknown resolver null

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent-1ace4748-b1af-4ad6-bee2-9f3744ef62c0
    confs: [default]
    0 artifacts copied, 26 already retrieved (0kB/7ms)
23/02/06 19:51:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/06 19:51:57 WARN AutoDiscoveryPluginRegistry: Plugin za.co.absa.spline.harvester.plugin.embedded.NonPersistentActionsCapturePlugin is disabled in the configuration.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.1.1
      /_/

Using Python version 3.7.6 (default, Jan  8 2020 19:59:22)
Spark context Web UI available at http://192.168.8.103:4040
Spark context available as 'sc' (master = local[*], app id = local-1675693313073).
SparkSession available as 'spark'.
>>>
wajda commented 1 year ago

image

image

Spark versions must match.