search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
921 stars 147 forks source link

[536] Exclude xtable-hudi-support-extensions from mvn deploy #570

Closed vinishjail97 closed 3 weeks ago

vinishjail97 commented 3 weeks ago

Important Read

What is the purpose of the pull request

We have non ASF compliant dependencies because of the bundling of dependencies (both direct and indirect) in xtable-hudi-support-extensions.

Both these bundled jars have been excluded from the release because there are indirect dependencies which are not ASF compliant in the class path, need to replace them with ASF compliant ones or remove the usage of maven-shade-plugin. 

find OpenSource/incubator-xtable -type f -name "*.jar" | grep bundled
/Users/vinishreddy/OpenSource/incubator-xtable/xtable-hudi-support/xtable-hudi-support-extensions/target/xtable-hudi-support-extensions_2.12-0.2.0-SNAPSHOT-bundled.jar
/Users/vinishreddy/OpenSource/incubator-xtable/xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar

Brief change log

(for example:)

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

the-other-tim-brown commented 3 weeks ago

This is mainly taking dependencies on Hudi, does that mean Hudi is not compliant?

vinishjail97 commented 3 weeks ago

This is mainly taking dependencies on Hudi, does that mean Hudi is not compliant?

I had assumed the same but there are few non ASF ones as well like org.openjdk.jol, javassist etc in the class path being added by maven-shade-plugin. 

[INFO] --- shade:3.6.0:shade (default) @ xtable-hudi-support-extensions_2.12 ---
[INFO] Including org.apache.xtable:xtable-hudi-support-utils:jar:0.2.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.logging.log4j:log4j-api:jar:2.22.0 in the shaded jar.
[INFO] Including org.apache.logging.log4j:log4j-1.2-api:jar:2.22.0 in the shaded jar.
[INFO] Including org.apache.xtable:xtable-core_2.12:jar:0.2.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.xtable:xtable-api:jar:0.2.0-SNAPSHOT in the shaded jar.
[INFO] Including com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.17.1 in the shaded jar.
[INFO] Including com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.17.1 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.8 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:32.1.3-jre in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including org.checkerframework:checker-qual:jar:3.37.0 in the shaded jar.
[INFO] Including com.google.errorprone:error_prone_annotations:jar:2.21.1 in the shaded jar.
[INFO] Including com.google.j2objc:j2objc-annotations:jar:2.8 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.12.20 in the shaded jar.
[INFO] Including org.apache.hudi:hudi-common:jar:0.14.0 in the shaded jar.
[INFO] Including org.openjdk.jol:jol-core:jar:0.16 in the shaded jar.
[INFO] Including org.apache.httpcomponents:fluent-hc:jar:4.4.1 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-client:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-protobuf:jar:3.5.1 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-common:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-logging:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-gson:jar:3.5.1 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-hadoop-compat:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-hadoop2-compat:jar:2.4.9 in the shaded jar.
[INFO] Including javax.activation:javax.activation-api:jar:1.2.0 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-protocol-shaded:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-protocol:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-miscellaneous:jar:3.5.1 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-netty:jar:3.5.1 in the shaded jar.
[INFO] Including org.apache.htrace:htrace-core4:jar:4.2.0-incubating in the shaded jar.
[INFO] Including org.jruby.jcodings:jcodings:jar:1.0.55 in the shaded jar.
[INFO] Including org.jruby.joni:joni:jar:2.1.31 in the shaded jar.
[INFO] Including org.apache.yetus:audience-annotations:jar:0.5.0 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-server:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-http:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-jetty:jar:3.5.1 in the shaded jar.
[INFO] Including org.apache.hbase.thirdparty:hbase-shaded-jersey:jar:3.5.1 in the shaded jar.
[INFO] Including javax.ws.rs:javax.ws.rs-api:jar:2.1.1 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-procedure:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-zookeeper:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-replication:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-metrics-api:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-metrics:jar:2.4.9 in the shaded jar.
[INFO] Including org.apache.hbase:hbase-asyncfs:jar:2.4.9 in the shaded jar.
[INFO] Including org.glassfish.web:javax.servlet.jsp:jar:2.3.2 in the shaded jar.
[INFO] Including org.glassfish:javax.el:jar:3.0.0 in the shaded jar.
[INFO] Including javax.servlet.jsp:javax.servlet.jsp-api:jar:2.3.1 in the shaded jar.
[INFO] Including org.jamon:jamon-runtime:jar:2.4.1 in the shaded jar.
[INFO] Including com.lmax:disruptor:jar:3.4.2 in the shaded jar.
[INFO] Including org.apache.iceberg:iceberg-api:jar:1.4.2 in the shaded jar.
[INFO] Including io.delta:delta-core_2.12:jar:2.4.0 in the shaded jar.
[INFO] Including io.delta:delta-storage:jar:2.4.0 in the shaded jar.
[INFO] Including org.antlr:antlr4-runtime:jar:4.9.3 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:2.0.9 in the shaded jar.
[INFO] Including org.apache.orc:orc-core:jar:1.6.0 in the shaded jar.
[INFO] Including org.apache.orc:orc-shims:jar:1.6.0 in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
[INFO] Including io.airlift:aircompressor:jar:0.15 in the shaded jar.
[INFO] Including javax.xml.bind:jaxb-api:jar:2.2.11 in the shaded jar.
[INFO] Including org.jetbrains:annotations:jar:17.0.0 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.1.1 in the shaded jar.
[INFO] Including org.apache.commons:commons-math3:jar:3.6.1 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.5.13 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.4.13 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.2 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.15 in the shaded jar.
[INFO] Including commons-io:commons-io:jar:2.14.0 in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.17.1 in the shaded jar.
[INFO] Including javax.annotation:javax.annotation-api:jar:1.3.2 in the shaded jar.
[INFO] Including org.apache.iceberg:iceberg-core:jar:1.4.2 in the shaded jar.
[INFO] Including org.apache.iceberg:iceberg-common:jar:1.4.2 in the shaded jar.
[INFO] Including org.apache.iceberg:iceberg-bundled-guava:jar:1.4.2 in the shaded jar.
[INFO] Including com.github.ben-manes.caffeine:caffeine:jar:2.9.3 in the shaded jar.
[INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.47 in the shaded jar.
[INFO] Including org.roaringbitmap:shims:jar:0.9.47 in the shaded jar.
[INFO] Including org.apache.iceberg:iceberg-parquet:jar:1.4.2 in the shaded jar.
[INFO] Including org.lz4:lz4-java:jar:1.8.0 in the shaded jar.
[INFO] Including jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6 in the shaded jar.
[INFO] Including org.glassfish.hk2.external:jakarta.inject:jar:2.6.1 in the shaded jar.
[INFO] Including jakarta.annotation:jakarta.annotation-api:jar:1.3.5 in the shaded jar.
[INFO] Including jakarta.validation:jakarta.validation-api:jar:2.0.2 in the shaded jar.
[INFO] Including org.javassist:javassist:jar:3.25.0-GA in the shaded jar.
[INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar.
[INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar.
[INFO] Including org.apache.hive:hive-storage-api:jar:2.8.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.25.5 in the shaded jar.
vinishjail97 commented 3 weeks ago

Closing this pull request, found a solution.