linkedin / transport

A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.
BSD 2-Clause "Simplified" License
291 stars 72 forks source link

Publish *-trino.tar(.zip) as an artifact for UDFs generated #150

Closed yiqiangin closed 1 year ago

yiqiangin commented 1 year ago

Code Change

Test cd transport ./gradlew clean build

// in a consumer UDF MP ./gradlew clean build mint release

The resulted artifact list in the file of buildspec.json are as follows:


"artifacts": {
      "marketing-std-udf": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:hive": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:javadoc": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:sources": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:spark_2.11": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:spark_2.12": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:trino": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
     "marketing-std-udf:zip": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7",
      "marketing-std-udf:trino-thin": "ivy:/com.linkedin.standard-udfs-marketing/marketing-std-udf/1.0.7"
    },
aastha25 commented 1 year ago

@yiqiangin I thought PR https://github.com/linkedin/transport/pull/147 could generate the *-trino.tar artifact? Could you clarify the diff between PR #147 and PR#150? Were the jars not getting published on the artifactory prior to this PR?

yiqiangin commented 1 year ago

@yiqiangin I thought PR #147 could generate the *-trino.tar artifact? Could you clarify the diff between PR #147 and PR#150? Were the jars not getting published on the artifactory prior to this PR? @aastha25 There are 4 build outputs for Trino including *-trino.tar, *-trino.zip, *-trino-thin.jar and *-trino-dist-thin.jar. *-trino.tar and *-trino.zip are built by two distribution tasks in DistributionPackaging and placed under distribution folder; *-trino-thin.jar and *-trino-dist-thin.jar are built by two Jar tasks in ThinJarPackaging and DistributionPackaging respectively and placed under libs folder. PR #147 only adds the Jar tasks to build *-trino-thin.jar and *-trino-dist-thin.jar into project.getArtifacts() explicitly, therefore only *-trino-thin.jar and *-trino-dist-thin.jar are published as artifacts, not *-trino.tar and *-trino.zip. However, Trino server requires the artifacts of *-trino.tar, *-trino.zip and *-trino-thin.jar, not *-trino-dist-thin.jar . The difference between PR #150 and PR #147 is adding two distribution tasks in DistributionPackaging to build *-trino.tar and *-trino.zip into project.getArtifacts() and no longer adding the Jar task to build *-trino-dist-thin.jar into project.getArtifacts(). Yes, *-trino.tar and *-trino.zip are not published as artifacts before this PR.