khorn-infomedia
commented
2 years ago Ok another hour and worked it out: Here is the full POM file. I am now adding exclusions back in, and will publish final POM later and close. There is Assembly and Shade options, so
- mvn clean package -Passembly
- mvn clean package -Pshade
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.thing</groupId>
<artifactId>csv-to-delta</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<scala.version>2.12</scala.version>
<spark.version>3.2.0</spark.version>
<junit.version>4.13.1</junit.version>
<delta.version>1.1.0</delta.version>
<hadoop.version>3.2.0</hadoop.version>
<maven-compiler-plugin.version>3.8.0</maven-compiler-plugin.version>
<maven-assembly-plugin.version>3.3.0</maven-assembly-plugin.version>
<maven-shade-plugin.version>3.1.0</maven-shade-plugin.version>
<maven-scala-plugin.version>2.15.2</maven-scala-plugin.version>
</properties>
<dependencies>
<!-- Spark -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.version}</artifactId>
<version>${delta.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.10</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>shade</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>${maven-scala-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${maven-shade-plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>
META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
</resource>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>assembly</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>${maven-scala-plugin.version}</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>
zsxwing
scottsand-db
It would be a nice feature of DeltaLake if it can be an easy to use component within a submittable Uber.jar, using Java, and MVN, without having to become a world expert in MVN to use it. The class incompatibility issues with Spark, and DeltaLake seem to be very common. My blocking Issue is the code below produces this, when submit:
"Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: delta" issue. In the case:
Dataset df = ...... // valid data.
df.write().format("delta") .mode("overwrite") .save("/opt/spark-data/mydata");
The first symptom of a problem was that the "overwrite" was failing on AWS S3 for an existing folder, but now its the "delta" string issue. DeltaLake works fine in local mode, with non-uber jar. I made it a local directory to test removing S3, in above.
As one person commented this makes DeltaLake un-usable. Many people have tackled this but there is no definitive answer, threads get closed with no solution, with many magical comments that don't seem to work. See related threads below. I think an example and a POM file for this should be part of the DeltaLake documentation, given the complexity and common nature of this issue, as it occurs with other components like Kafka, Avro. There are no Java example in Delta source, maybe this would be a good example.
The "delta" string cannot be provided with full class definition, to resolve this. DeltaLake (and Spark) using untyped strings and not a FINAL String member is in itself poor programming practise.
So is there a 're-usable' MVN pom file (Not Gradle) that does a Uber.jar build of DeltaLake, with AWS libraries, for Java, that can be submitted to Spark and it actually works ? Not using the --packages and local --jars etc. If there is, I can help document this into the DeltaLAke documentation to make a contribution. It requires using mvn-shade to merge the META INF/services/org.apache.spark.sql.sources.DataSourceRegister of all the data sources,, not replace or first or whatever strategy you use. I have tried many things but nothing yet works ?
I have a full project for this, can provide more, but here are some snippets: