search

linkedin / transport

A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.
BSD 2-Clause "Simplified" License
297 stars 73 forks source link

Add Hadoop libraries as runtimeOnly dependeny in transportable-udfs-test-hive/generic/trino #154

Closed yiqiangin closed 1 year ago

yiqiangin commented 1 year ago

Code Changes As consumer UDFs generated by Transport Plugin use HiveTester defined in the module of transportable-udfs-hive-test to execute the unit tests on UDF generated against Hive at run time and HiveTester depends on some classes from the libraries of hadoop-common and hadoop-mapreduce-client-core, two libraries are required at run time and they are added as runtimeOnly dependencies in the module of transportable-udfs-hive-test. Also the test classes from transportable-udfs-test-generic and transportable-udfs-test-trino depends on hdfs library of Hadoop when the tests of UDF are executed at runtime. Therefore the runtimeOnly dependencies on two libraries are added into transportable-udfs-test-generic and transportable-udfs-test-trino.

Tests

cd transport
./gradlew clean build

cd transportable-udfs-examples
./gradlew clean build

Test UDF with a locally-built TransportPlugin with this code change.
weijiii commented 1 year ago

Should you remove the testImplementation for hadoop in transportable-udfs-example/transportable-udfs-example/build.gradle then run the tests? Is it needed for the testing on Trino? LG