search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.53k stars 1.29k forks source link

Weird shaded pom in pinot-plugins #5022

Closed haibow closed 4 years ago

haibow commented 4 years ago

After running maven build with mvn clean install -DskipTests -Pbin-dist, I'm seeing *shaded.pom files for several "parent" modules.

pinot-plugins/target/plugins/pinot-stream-ingestion/pinot-stream-ingestion/pinot-stream-ingestion-0.3.0-SNAPSHOT-shaded.pom
pinot-plugins/target/plugins/pinot-batch-ingestion/v0_deprecated/v0_deprecated-0.3.0-SNAPSHOT-shaded.pom
pinot-plugins/target/plugins/pinot-batch-ingestion/pinot-batch-ingestion/pinot-batch-ingestion-0.3.0-SNAPSHOT-shaded.pom
pinot-plugins/target/plugins/pinot-file-system/pinot-file-system/pinot-file-system-0.3.0-SNAPSHOT-shaded.pom
pinot-plugins/target/plugins/pinot-input-format/pinot-input-format/pinot-input-format-0.3.0-SNAPSHOT-shaded.pom
pinot-plugins/target/plugins/pinot-plugins/pinot-plugins-0.3.0-SNAPSHOT-shaded.pom

The build passed, with *shaded.jar generated in pinot-plugins/target/plugins/ for each plugin. Still, something seems off with the shading config.

mcvsubbu commented 4 years ago

Hi @fx19880617 can u take a look at this? thanks

xiangfu0 commented 4 years ago

The reason is we put the maven shaded plugin at pinot-plugins level pom file, so it will apply to all the nested leaf modules. *shaded.pom is generated because we set <packaging>pom</packaging> in those modules pom.xml file, so it's expected behavior. In pinot-distribution, we have this assembly file to package all the plugins together: https://github.com/apache/incubator-pinot/blob/master/pinot-distribution/pinot-assembly.xml#L91

haibow commented 4 years ago

The extra *shaded.pom files caused issues with our Jenkins build. It tried to upload the shaded pom and threw errors. I could resolve the issue by removing the shading from the parent pom and putting them back to each leaf module.

Ref: Our release command:

mvn -e -B release:clean release:prepare release:perform -Darguments="-P build-shaded-jar"

Full error log:

[INFO] [INFO] --- maven-install-plugin:2.5.2:install (default-install) @ pinot-plugins ---
[INFO] [INFO] Installing /jenkins/workspace/pinot-release/target/checkout/pinot-plugins/pom.xml to /jenkins/.m2/repository/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14.pom
[INFO] [INFO] Installing /jenkins/workspace/pinot-release/target/checkout/pinot-plugins/../pinot-plugins/target/plugins/pinot-plugins/pinot-plugins-0.3.14-shaded.pom to /jenkins/.m2/repository/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14-shaded.pom
[INFO] [INFO] 
[INFO] [INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ pinot-plugins ---
[INFO] [INFO] Uploading: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14.pom
[INFO] [INFO] Uploaded: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14.pom (5 KB at 55.4 KB/sec)
[INFO] [INFO] Downloading: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/maven-metadata.xml
[INFO] [INFO] Downloaded: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/maven-metadata.xml (559 B at 24.8 KB/sec)
[INFO] [INFO] Uploading: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/maven-metadata.xml
[INFO] [INFO] Uploaded: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/maven-metadata.xml (591 B at 10.7 KB/sec)
[INFO] [INFO] Uploading: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14-shaded.pom
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Reactor Summary:
[INFO] [INFO] 
[INFO] [INFO] Pinot .............................................. SUCCESS [  3.446 s]
[INFO] [INFO] Pinot Service Provider Interface ................... SUCCESS [  2.956 s]
[INFO] [INFO] Pinot Common ....................................... SUCCESS [ 14.740 s]
[INFO] [INFO] Pinot Plugins ...................................... FAILURE [  0.404 s]
[INFO] [INFO] Pinot Input Format ................................. SKIPPED
[INFO] [INFO] Pinot Avro ......................................... SKIPPED
[INFO] [INFO] Pinot Csv .......................................... SKIPPED
[INFO] [INFO] Pinot JSON ......................................... SKIPPED
[INFO] [INFO] Pinot Core ......................................... SKIPPED
[INFO] [INFO] Pinot Server ....................................... SKIPPED
[INFO] [INFO] Pinot Controller ................................... SKIPPED
[INFO] [INFO] Pinot Broker ....................................... SKIPPED
[INFO] [INFO] Pinot Clients ...................................... SKIPPED
[INFO] [INFO] Pinot Java Client .................................. SKIPPED
[INFO] [INFO] Pinot Minion ....................................... SKIPPED
[INFO] [INFO] Pinot ORC .......................................... SKIPPED
[INFO] [INFO] Pinot Parquet ...................................... SKIPPED
[INFO] [INFO] Pinot Thrift ....................................... SKIPPED
[INFO] [INFO] Pluggable Pinot file system ........................ SKIPPED
[INFO] [INFO] Pinot Azure Data Lake Storage ...................... SKIPPED
[INFO] [INFO] Pinot Hadoop Filesystem ............................ SKIPPED
[INFO] [INFO] Pinot Google Cloud Storage ......................... SKIPPED
[INFO] [INFO] Pinot Batch Ingestion .............................. SKIPPED
[INFO] [INFO] Pinot Batch Ingestion Common ....................... SKIPPED
[INFO] [INFO] Pinot Batch Ingestion for Spark .................... SKIPPED
[INFO] [INFO] Pinot Batch Ingestion for Hadoop ................... SKIPPED
[INFO] [INFO] Pinot Batch Ingestion Standalone ................... SKIPPED
[INFO] [INFO] Pinot Batch Ingestion .............................. SKIPPED
[INFO] [INFO] Pinot Ingestion Common ............................. SKIPPED
[INFO] [INFO] Pinot Stream Ingestion ............................. SKIPPED
[INFO] [INFO] Pinot Kafka Base ................................... SKIPPED
[INFO] [INFO] Pinot Kafka 2.0 .................................... SKIPPED
[INFO] [INFO] Pinot Tools ........................................ SKIPPED
[INFO] [INFO] Pinot Integration Tests ............................ SKIPPED
[INFO] [INFO] Pinot Hadoop ....................................... SKIPPED
[INFO] [INFO] Pinot Spark ........................................ SKIPPED
[INFO] [INFO] Pinot Kafka 0.9 .................................... SKIPPED
[INFO] [INFO] Pinot Perf ......................................... SKIPPED
[INFO] [INFO] Pinot Distribution ................................. SKIPPED
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] BUILD FAILURE
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Total time: 22.188 s
[INFO] [INFO] Finished at: 2020-01-29T01:53:21+00:00
[INFO] [INFO] Final Memory: 133M/3964M
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project pinot-plugins: Failed to deploy artifacts: Could not transfer artifact org.apache.pinot:pinot-plugins:pom:shaded:0.3.14 from/to central (http://artifactory.com/artifactory/libs-release-local/): Failed to transfer file: http://artifactory.com/artifactory/libs-release-local/org/apache/pinot/pinot-plugins/0.3.14/pinot-plugins-0.3.14-shaded.pom. Return code is: 409, ReasonPhrase: Conflict. -> [Help 1]
[INFO] [ERROR] 
[INFO] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[INFO] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[INFO] [ERROR] 
[INFO] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[INFO] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[INFO] [ERROR] 
[INFO] [ERROR] After correcting the problems, you can resume the build with the command
[INFO] [ERROR]   mvn <goals> -rf :pinot-plugins
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Pinot .............................................. FAILURE [04:12 min]
[INFO] Pinot Service Provider Interface ................... SKIPPED
[INFO] Pinot Common ....................................... SKIPPED
[INFO] Pinot Plugins ...................................... SKIPPED
[INFO] Pinot Input Format ................................. SKIPPED
[INFO] Pinot Avro ......................................... SKIPPED
[INFO] Pinot Csv .......................................... SKIPPED
[INFO] Pinot JSON ......................................... SKIPPED
[INFO] Pinot Core ......................................... SKIPPED
[INFO] Pinot Server ....................................... SKIPPED
[INFO] Pinot Controller ................................... SKIPPED
[INFO] Pinot Broker ....................................... SKIPPED
[INFO] Pinot Clients ...................................... SKIPPED
[INFO] Pinot Java Client .................................. SKIPPED
[INFO] Pinot Minion ....................................... SKIPPED
[INFO] Pinot ORC .......................................... SKIPPED
[INFO] Pinot Parquet ...................................... SKIPPED
[INFO] Pinot Thrift ....................................... SKIPPED
[INFO] Pluggable Pinot file system ........................ SKIPPED
[INFO] Pinot Azure Data Lake Storage ...................... SKIPPED
[INFO] Pinot Hadoop Filesystem ............................ SKIPPED
[INFO] Pinot Google Cloud Storage ......................... SKIPPED
[INFO] Pinot Batch Ingestion .............................. SKIPPED
[INFO] Pinot Batch Ingestion Common ....................... SKIPPED
[INFO] Pinot Batch Ingestion for Spark .................... SKIPPED
[INFO] Pinot Batch Ingestion for Hadoop ................... SKIPPED
[INFO] Pinot Batch Ingestion Standalone ................... SKIPPED
[INFO] Pinot Batch Ingestion .............................. SKIPPED
[INFO] Pinot Ingestion Common ............................. SKIPPED
[INFO] Pinot Stream Ingestion ............................. SKIPPED
[INFO] Pinot Kafka Base ................................... SKIPPED
[INFO] Pinot Kafka 2.0 .................................... SKIPPED
[INFO] Pinot Tools ........................................ SKIPPED
[INFO] Pinot Integration Tests ............................ SKIPPED
[INFO] Pinot Hadoop ....................................... SKIPPED
[INFO] Pinot Spark ........................................ SKIPPED
[INFO] Pinot Kafka 0.9 .................................... SKIPPED
[INFO] Pinot Perf ......................................... SKIPPED
[INFO] Pinot Distribution ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:12 min
[INFO] Finished at: 2020-01-29T01:53:22+00:00
[INFO] Final Memory: 49M/2475M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.3:perform (default-cli) on project pinot: Maven execution failed, exit code: '1' -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.5.3:perform (default-cli) on project pinot: Maven execution failed, exit code: '1'
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Maven execution failed, exit code: '1'
    at org.apache.maven.plugins.release.PerformReleaseMojo.execute(PerformReleaseMojo.java:204)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
    ... 20 more
Caused by: org.apache.maven.shared.release.ReleaseExecutionException: Maven execution failed, exit code: '1'
    at org.apache.maven.shared.release.phase.AbstractRunGoalsPhase.execute(AbstractRunGoalsPhase.java:91)
    at org.apache.maven.shared.release.phase.RunPerformGoalsPhase.runLogic(RunPerformGoalsPhase.java:135)
    at org.apache.maven.shared.release.phase.RunPerformGoalsPhase.execute(RunPerformGoalsPhase.java:46)
    at org.apache.maven.shared.release.DefaultReleaseManager.perform(DefaultReleaseManager.java:429)
    at org.apache.maven.shared.release.DefaultReleaseManager.perform(DefaultReleaseManager.java:381)
    at org.apache.maven.plugins.release.PerformReleaseMojo.execute(PerformReleaseMojo.java:200)
    ... 22 more
Caused by: org.apache.maven.shared.release.exec.MavenExecutorException: Maven execution failed, exit code: '1'
    at org.apache.maven.shared.release.exec.InvokerMavenExecutor.executeGoals(InvokerMavenExecutor.java:433)
    at org.apache.maven.shared.release.exec.AbstractMavenExecutor.executeGoals(AbstractMavenExecutor.java:114)
    at org.apache.maven.shared.release.phase.AbstractRunGoalsPhase.execute(AbstractRunGoalsPhase.java:85)
    ... 27 more
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Build step 'Execute shell' marked build as failure
xiangfu0 commented 4 years ago

@haibow can you check if https://github.com/apache/incubator-pinot/pull/5036 will fix your issue?

haibow commented 4 years ago

Thanks for the fix. Running clean at the end will remove the extra pom files now. Will test it with our deployment flow and let you know.