search

microsoft / SynapseML

Simple and Distributed Machine Learning
http://aka.ms/spark
MIT License
5.06k stars 830 forks source link

java.io.FileNotFoundException - wrong jar names #1374

Closed philmassie closed 2 years ago

philmassie commented 2 years ago

Describe the bug jars not found when starting Spark session:

...
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final.jar does not exist, skipping.
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-kqueue-4.1.68.Final.jar does not exist, skipping.
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-resolver-dns-native-macos-4.1.68.Final.jar does not exist, skipping.
...
java.io.FileNotFoundException: File file:/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final.jar does not exist

To Reproduce For me:

emrconfig = SparkConf()\
    .setMaster("yarn")\
    .set('spark.emr.maximizeResourceAllocation','true')\
    .set('spark.dynamicAllocation.enabled', 'false')\
    .set("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5") \
    .set("spark.jars.repositories", "https://mmlspark.azureedge.net/maven")
spark = SparkSession \
    .builder \
    .config(conf=emrconfig)\
    .getOrCreate()

This is so simple so I assume 'm missing something

Expected behavior The three files to be downloaded with correct names. If I rename the files as follows, the application gets submitted and started successfully:

/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar > io.netty_netty-transport-native-epoll-4.1.68.Final.jar
/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-kqueue-4.1.68.Final-osx-x86_64.jar > io.netty_netty-transport-native-kqueue-4.1.68.Final.jar
/home/ec2-user/.ivy2/jars/io.netty_netty-resolver-dns-native-macos-4.1.68.Final-osx-x86_64.jar > io.netty_netty-resolver-dns-native-macos-4.1.68.Final.jar

Info (please complete the following information):

Stacktrace

[Please post the stacktrace here if applicable](https://mmlspark.azureedge.net/maven added as a remote repository with the name: repo-1
Ivy Default Cache set to: /home/ec2-user/.ivy2/cache
The jars for the packages stored in: /home/ec2-user/.ivy2/jars
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.azure#synapseml_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-c0c8a1b1-ebd8-41ae-9b2a-625051f08cc3;1.0
    confs: [default]
    found com.microsoft.azure#synapseml_2.12;0.9.5 in central
    found com.microsoft.azure#synapseml-core_2.12;0.9.5 in central
    found org.scalactic#scalactic_2.12;3.0.5 in central
    found org.scala-lang#scala-reflect;2.12.4 in central
    found io.spray#spray-json_2.12;1.3.2 in central
    found com.jcraft#jsch;0.1.54 in central
    found org.apache.httpcomponents#httpclient;4.5.6 in central
    found org.apache.httpcomponents#httpcore;4.4.10 in central
    found commons-logging#commons-logging;1.2 in central
    found commons-codec#commons-codec;1.10 in central
    found org.apache.httpcomponents#httpmime;4.5.6 in central
    found com.linkedin.isolation-forest#isolation-forest_3.2.0_2.12;2.0.8 in central
    found com.chuusai#shapeless_2.12;2.3.2 in central
    found org.typelevel#macro-compat_2.12;1.1.1 in central
    found org.apache.spark#spark-avro_2.12;3.2.0 in central
    found org.tukaani#xz;1.8 in central
    found org.spark-project.spark#unused;1.0.0 in central
    found org.testng#testng;6.8.8 in central
    found org.beanshell#bsh;2.0b4 in central
    found com.beust#jcommander;1.27 in central
    found com.microsoft.azure#synapseml-deep-learning_2.12;0.9.5 in central
    found com.microsoft.azure#synapseml-opencv_2.12;0.9.5 in central
    found org.openpnp#opencv;3.2.0-1 in central
    found com.microsoft.cntk#cntk;2.4 in central
    found com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 in central
    found com.microsoft.azure#synapseml-cognitive_2.12;0.9.5 in central
    found com.microsoft.cognitiveservices.speech#client-jar-sdk;1.14.0 in central
    found com.azure#azure-storage-blob;12.14.2 in central
    found com.azure#azure-core;1.22.0 in central
    found com.fasterxml.jackson.core#jackson-annotations;2.12.5 in central
    found com.fasterxml.jackson.core#jackson-core;2.12.5 in central
    found com.fasterxml.jackson.core#jackson-databind;2.12.5 in central
    found com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.12.5 in central
    found com.fasterxml.jackson.dataformat#jackson-dataformat-xml;2.12.5 in central
    found com.fasterxml.jackson.module#jackson-module-jaxb-annotations;2.12.5 in central
    found jakarta.xml.bind#jakarta.xml.bind-api;2.3.2 in central
    found jakarta.activation#jakarta.activation-api;1.2.1 in central
    found org.codehaus.woodstox#stax2-api;4.2.1 in central
    found com.fasterxml.woodstox#woodstox-core;6.2.4 in central
    found org.slf4j#slf4j-api;1.7.32 in central
    found io.projectreactor#reactor-core;3.4.10 in central
    found org.reactivestreams#reactive-streams;1.0.3 in central
    found io.netty#netty-tcnative-boringssl-static;2.0.43.Final in central
    found com.azure#azure-core-http-netty;1.11.2 in central
    found io.netty#netty-handler;4.1.68.Final in central
    found io.netty#netty-common;4.1.68.Final in central
    found io.netty#netty-resolver;4.1.68.Final in central
    found io.netty#netty-buffer;4.1.68.Final in central
    found io.netty#netty-transport;4.1.68.Final in central
    found io.netty#netty-codec;4.1.68.Final in central
    found io.netty#netty-handler-proxy;4.1.68.Final in central
    found io.netty#netty-codec-socks;4.1.68.Final in central
    found io.netty#netty-codec-http;4.1.68.Final in central
    found io.netty#netty-codec-http2;4.1.68.Final in central
    found io.netty#netty-transport-native-unix-common;4.1.68.Final in central
    found io.netty#netty-transport-native-epoll;4.1.68.Final in central
    found io.netty#netty-transport-native-kqueue;4.1.68.Final in central
    found io.projectreactor.netty#reactor-netty-http;1.0.11 in central
    found io.netty#netty-resolver-dns;4.1.68.Final in central
    found io.netty#netty-codec-dns;4.1.68.Final in central
    found io.netty#netty-resolver-dns-native-macos;4.1.68.Final in central
    found io.projectreactor.netty#reactor-netty-core;1.0.11 in central
    found com.azure#azure-storage-common;12.14.1 in central
    found com.azure#azure-storage-internal-avro;12.1.2 in central
    found com.azure#azure-ai-textanalytics;5.1.4 in central
    found com.microsoft.azure#synapseml-vw_2.12;0.9.5 in central
    found com.github.vowpalwabbit#vw-jni;8.9.1 in central
    found com.microsoft.azure#synapseml-lightgbm_2.12;0.9.5 in central
    found com.microsoft.ml.lightgbm#lightgbmlib;3.2.110 in central
:: resolution report :: resolve 828ms :: artifacts dl 15ms
    :: modules in use:
    com.azure#azure-ai-textanalytics;5.1.4 from central in [default]
    com.azure#azure-core;1.22.0 from central in [default]
    com.azure#azure-core-http-netty;1.11.2 from central in [default]
    com.azure#azure-storage-blob;12.14.2 from central in [default]
    com.azure#azure-storage-common;12.14.1 from central in [default]
    com.azure#azure-storage-internal-avro;12.1.2 from central in [default]
    com.beust#jcommander;1.27 from central in [default]
    com.chuusai#shapeless_2.12;2.3.2 from central in [default]
    com.fasterxml.jackson.core#jackson-annotations;2.12.5 from central in [default]
    com.fasterxml.jackson.core#jackson-core;2.12.5 from central in [default]
    com.fasterxml.jackson.core#jackson-databind;2.12.5 from central in [default]
    com.fasterxml.jackson.dataformat#jackson-dataformat-xml;2.12.5 from central in [default]
    com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.12.5 from central in [default]
    com.fasterxml.jackson.module#jackson-module-jaxb-annotations;2.12.5 from central in [default]
    com.fasterxml.woodstox#woodstox-core;6.2.4 from central in [default]
    com.github.vowpalwabbit#vw-jni;8.9.1 from central in [default]
    com.jcraft#jsch;0.1.54 from central in [default]
    com.linkedin.isolation-forest#isolation-forest_3.2.0_2.12;2.0.8 from central in [default]
    com.microsoft.azure#synapseml-cognitive_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-core_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-deep-learning_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-lightgbm_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-opencv_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-vw_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml_2.12;0.9.5 from central in [default]
    com.microsoft.cntk#cntk;2.4 from central in [default]
    com.microsoft.cognitiveservices.speech#client-jar-sdk;1.14.0 from central in [default]
    com.microsoft.ml.lightgbm#lightgbmlib;3.2.110 from central in [default]
    com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 from central in [default]
    commons-codec#commons-codec;1.10 from central in [default]
    commons-logging#commons-logging;1.2 from central in [default]
    io.netty#netty-buffer;4.1.68.Final from central in [default]
    io.netty#netty-codec;4.1.68.Final from central in [default]
    io.netty#netty-codec-dns;4.1.68.Final from central in [default]
    io.netty#netty-codec-http;4.1.68.Final from central in [default]
    io.netty#netty-codec-http2;4.1.68.Final from central in [default]
    io.netty#netty-codec-socks;4.1.68.Final from central in [default]
    io.netty#netty-common;4.1.68.Final from central in [default]
    io.netty#netty-handler;4.1.68.Final from central in [default]
    io.netty#netty-handler-proxy;4.1.68.Final from central in [default]
    io.netty#netty-resolver;4.1.68.Final from central in [default]
    io.netty#netty-resolver-dns;4.1.68.Final from central in [default]
    io.netty#netty-resolver-dns-native-macos;4.1.68.Final from central in [default]
    io.netty#netty-tcnative-boringssl-static;2.0.43.Final from central in [default]
    io.netty#netty-transport;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-epoll;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-kqueue;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-unix-common;4.1.68.Final from central in [default]
    io.projectreactor#reactor-core;3.4.10 from central in [default]
    io.projectreactor.netty#reactor-netty-core;1.0.11 from central in [default]
    io.projectreactor.netty#reactor-netty-http;1.0.11 from central in [default]
    io.spray#spray-json_2.12;1.3.2 from central in [default]
    jakarta.activation#jakarta.activation-api;1.2.1 from central in [default]
    jakarta.xml.bind#jakarta.xml.bind-api;2.3.2 from central in [default]
    org.apache.httpcomponents#httpclient;4.5.6 from central in [default]
    org.apache.httpcomponents#httpcore;4.4.10 from central in [default]
    org.apache.httpcomponents#httpmime;4.5.6 from central in [default]
    org.apache.spark#spark-avro_2.12;3.2.0 from central in [default]
    org.beanshell#bsh;2.0b4 from central in [default]
    org.codehaus.woodstox#stax2-api;4.2.1 from central in [default]
    org.openpnp#opencv;3.2.0-1 from central in [default]
    org.reactivestreams#reactive-streams;1.0.3 from central in [default]
    org.scala-lang#scala-reflect;2.12.4 from central in [default]
    org.scalactic#scalactic_2.12;3.0.5 from central in [default]
    org.slf4j#slf4j-api;1.7.32 from central in [default]
    org.spark-project.spark#unused;1.0.0 from central in [default]
    org.testng#testng;6.8.8 from central in [default]
    org.tukaani#xz;1.8 from central in [default]
    org.typelevel#macro-compat_2.12;1.1.1 from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   69  |   0   |   0   |   0   ||   69  |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-c0c8a1b1-ebd8-41ae-9b2a-625051f08cc3
    confs: [default]
    0 artifacts copied, 69 already retrieved (0kB/13ms)
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final.jar does not exist, skipping.
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-kqueue-4.1.68.Final.jar does not exist, skipping.
22/02/01 13:31:14 WARN DependencyUtils: Local jar /home/ec2-user/.ivy2/jars/io.netty_netty-resolver-dns-native-macos-4.1.68.Final.jar does not exist, skipping.
22/02/01 13:31:14 INFO SparkContext: Running Spark version 2.4.7-amzn-1
22/02/01 13:31:14 INFO SparkContext: Submitted application: pyspark-shell
22/02/01 13:31:14 INFO SecurityManager: Changing view acls to: ec2-user
22/02/01 13:31:14 INFO SecurityManager: Changing modify acls to: ec2-user
22/02/01 13:31:14 INFO SecurityManager: Changing view acls groups to: 
22/02/01 13:31:14 INFO SecurityManager: Changing modify acls groups to: 
22/02/01 13:31:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ec2-user); groups with view permissions: Set(); users  with modify permissions: Set(ec2-user); groups with modify permissions: Set()
22/02/01 13:31:14 INFO Utils: Successfully started service 'sparkDriver' on port 33873.
22/02/01 13:31:14 INFO SparkEnv: Registering MapOutputTracker
22/02/01 13:31:14 INFO SparkEnv: Registering BlockManagerMaster
22/02/01 13:31:14 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/02/01 13:31:14 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/02/01 13:31:14 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8f6e6fa5-f663-4604-8c74-05e280c93584
22/02/01 13:31:14 INFO MemoryStore: MemoryStore started with capacity 27.0 GB
22/02/01 13:31:14 INFO SparkEnv: Registering OutputCommitCoordinator
22/02/01 13:31:14 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/02/01 13:31:14 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://ip-1-2-3-4.eu-west-1.compute.internal:4040
22/02/01 13:31:15 INFO RMProxy: Connecting to ResourceManager at ip-10-3-32-74.eu-west-1.compute.internal/10.3.32.74:8032
22/02/01 13:31:15 INFO Client: Requesting a new application from cluster with 3 NodeManagers
22/02/01 13:31:15 INFO Configuration: resource-types.xml not found
22/02/01 13:31:15 INFO ResourceUtils: Unable to find 'resource-types.xml'.
22/02/01 13:31:15 INFO ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
22/02/01 13:31:15 INFO ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
22/02/01 13:31:15 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (57344 MB per container)
22/02/01 13:31:15 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
22/02/01 13:31:15 INFO Client: Setting up container launch context for our AM
22/02/01 13:31:15 INFO Client: Setting up the launch environment for our AM container
22/02/01 13:31:15 INFO Client: Preparing resources for our AM container
22/02/01 13:31:15 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
22/02/01 13:31:17 INFO Client: Uploading resource file:/tmp/spark-534569b4-be5f-4337-9a66-2d1dca59fbee/__spark_libs__5179491898158020782.zip -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/__spark_libs__5179491898158020782.zip
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-core_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-core_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-deep-learning_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-deep-learning_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-cognitive_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-cognitive_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-vw_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-vw_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-lightgbm_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-lightgbm_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.azure_synapseml-opencv_2.12-0.9.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.azure_synapseml-opencv_2.12-0.9.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.scalactic_scalactic_2.12-3.0.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.scalactic_scalactic_2.12-3.0.5.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.spray_spray-json_2.12-1.3.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.spray_spray-json_2.12-1.3.2.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.jcraft_jsch-0.1.54.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.jcraft_jsch-0.1.54.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.apache.httpcomponents_httpclient-4.5.6.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.apache.httpcomponents_httpclient-4.5.6.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.apache.httpcomponents_httpmime-4.5.6.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.apache.httpcomponents_httpmime-4.5.6.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.linkedin.isolation-forest_isolation-forest_3.2.0_2.12-2.0.8.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.linkedin.isolation-forest_isolation-forest_3.2.0_2.12-2.0.8.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.scala-lang_scala-reflect-2.12.4.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.scala-lang_scala-reflect-2.12.4.jar
22/02/01 13:31:17 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.apache.httpcomponents_httpcore-4.4.10.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.apache.httpcomponents_httpcore-4.4.10.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/commons-logging_commons-logging-1.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/commons-logging_commons-logging-1.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/commons-codec_commons-codec-1.10.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/commons-codec_commons-codec-1.10.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.chuusai_shapeless_2.12-2.3.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.chuusai_shapeless_2.12-2.3.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.apache.spark_spark-avro_2.12-3.2.0.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.apache.spark_spark-avro_2.12-3.2.0.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.testng_testng-6.8.8.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.testng_testng-6.8.8.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.typelevel_macro-compat_2.12-1.1.1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.typelevel_macro-compat_2.12-1.1.1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.tukaani_xz-1.8.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.tukaani_xz-1.8.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.spark-project.spark_unused-1.0.0.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.beanshell_bsh-2.0b4.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.beanshell_bsh-2.0b4.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.beust_jcommander-1.27.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.beust_jcommander-1.27.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.cntk_cntk-2.4.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.cntk_cntk-2.4.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.onnxruntime_onnxruntime_gpu-1.8.1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.onnxruntime_onnxruntime_gpu-1.8.1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.openpnp_opencv-3.2.0-1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.openpnp_opencv-3.2.0-1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.microsoft.cognitiveservices.speech_client-jar-sdk-1.14.0.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.microsoft.cognitiveservices.speech_client-jar-sdk-1.14.0.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-storage-blob-12.14.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-storage-blob-12.14.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-ai-textanalytics-5.1.4.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-ai-textanalytics-5.1.4.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-core-1.22.0.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-core-1.22.0.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-core-http-netty-1.11.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-core-http-netty-1.11.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-storage-common-12.14.1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-storage-common-12.14.1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.azure_azure-storage-internal-avro-12.1.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.azure_azure-storage-internal-avro-12.1.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.core_jackson-annotations-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.core_jackson-annotations-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.core_jackson-core-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.core_jackson-databind-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.core_jackson-databind-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.datatype_jackson-datatype-jsr310-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.datatype_jackson-datatype-jsr310-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.dataformat_jackson-dataformat-xml-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.dataformat_jackson-dataformat-xml-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.slf4j_slf4j-api-1.7.32.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.slf4j_slf4j-api-1.7.32.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.projectreactor_reactor-core-3.4.10.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.projectreactor_reactor-core-3.4.10.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-tcnative-boringssl-static-2.0.43.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-tcnative-boringssl-static-2.0.43.Final.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.jackson.module_jackson-module-jaxb-annotations-2.12.5.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.jackson.module_jackson-module-jaxb-annotations-2.12.5.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.codehaus.woodstox_stax2-api-4.2.1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.codehaus.woodstox_stax2-api-4.2.1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/com.fasterxml.woodstox_woodstox-core-6.2.4.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/com.fasterxml.woodstox_woodstox-core-6.2.4.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/jakarta.xml.bind_jakarta.xml.bind-api-2.3.2.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/jakarta.xml.bind_jakarta.xml.bind-api-2.3.2.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/jakarta.activation_jakarta.activation-api-1.2.1.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/jakarta.activation_jakarta.activation-api-1.2.1.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/org.reactivestreams_reactive-streams-1.0.3.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/org.reactivestreams_reactive-streams-1.0.3.jar
22/02/01 13:31:18 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-handler-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-handler-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-handler-proxy-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-handler-proxy-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-buffer-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-buffer-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-codec-http-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-codec-http-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-codec-http2-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-codec-http2-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-unix-common-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-transport-native-unix-common-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Uploading resource file:/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final.jar -> hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007/io.netty_netty-transport-native-epoll-4.1.68.Final.jar
22/02/01 13:31:19 INFO Client: Deleted staging directory hdfs://ip-10-3-32-74.eu-west-1.compute.internal:8020/user/ec2-user/.sparkStaging/application_1643705154665_0007
22/02/01 13:31:19 ERROR SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/home/ec2-user/.ivy2/jars/io.netty_netty-transport-native-epoll-4.1.68.Final.jar does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:639)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:865)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:629)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:456)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
    at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:384)
    at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:476)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10$$anonfun$apply$6.apply(Client.scala:598)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10$$anonfun$apply$6.apply(Client.scala:597)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:597)
    at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:596)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:596)
    at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:869)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:178)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:188)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:515)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:238)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
22/02/01 13:31:19 INFO SparkUI: Stopped Spark web UI at http://ip-1-2-3-4.eu-west-1.compute.internal:4040
22/02/01 13:31:19 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
22/02/01 13:31:19 INFO YarnClientSchedulerBackend: Stopped
22/02/01 13:31:19 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/02/01 13:31:19 INFO MemoryStore: MemoryStore cleared
22/02/01 13:31:19 INFO BlockManager: BlockManager stopped
22/02/01 13:31:19 INFO BlockManagerMaster: BlockManagerMaster stopped
22/02/01 13:31:19 WARN MetricsSystem: Stopping a MetricsSystem that is not running
22/02/01 13:31:19 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/02/01 13:31:19 INFO SparkContext: Successfully stopped SparkContext)
philmassie commented 2 years ago

hmm, this has happened with another unrelated library, making me think it might not have anything to do with SynapseML, no idea what it is though

jonathansp commented 2 years ago

@philmassie similar error here. Any workaround?

: java.io.FileNotFoundException: File file:/.ivy2/jars/io.netty_netty-tcnative-boringssl-static-2.0.43.Final-.jar does not exist
mhamilton723 commented 2 years ago

Hey thanks for reaching out! I see you are still on Spark 2.4. SynapseML 0.9.5 is for spark 3.2 and there are some known dependency challenges that will arise if these versions are mismatched. I would imagine this would be a cause of some of the challenges you are facing

philmassie commented 2 years ago

@jonathansp only thing that worked for me was to manually rename the jars in their directory. it seems like whatever's managing the download isn't reliably renaming them once theyre down. its a little painful but it got me past those initial errors at least :) If you do find a solution Id appreciate a heads up :)

@mhamilton723 good catch, I missed that. Will close this for sure. I saw the same errors importing other mvn packages so I dont think its actually SynapseML. Do you maybe know what the latest version of mmlspark/synapseml will work on 2.4? we're tied to EMR.

Thanks for the feedback

jonathansp commented 2 years ago

@philmassie thanks, indeed renaming the file worked, as something has to do with the gradle versionless build. However, it's hard to maintain.

@mhamilton723 absolutely, I downgraded SynapseML to 0.9.4 and it worked with spark 3.1.

Thank you guys.