For Spark3 mvn install error for "maven-enforcer-plugin:3.0.0-M2:enforce (enforce-no-duplicate-dependencies) @ spark-yarn_2.12"

[rpignolet]

When using branch branch-3.2.4-build I tried to run mvn install after running ./dev/make-distribution.sh and it doesn't work.

Command used:

./build/mvn install -Phive -Phive-thriftserver -Pyarn -Phadoop-3.1 -Dscalastyle.skip=true -DskipTests --fail-at-end

I used -Dscalastyle.skip=true because scalastyle detect style errors.

Maven log:

[INFO] Reactor Summary for Spark Project Parent POM 3.2.2-TDP-0.1.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [ 10.390 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 10.180 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  6.883 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  7.815 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 12.130 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 16.623 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  8.414 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  7.814 s]
[INFO] Spark Project Core ................................. SUCCESS [02:55 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 57.197 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:13 min]
[INFO] Spark Project Streaming ............................ SUCCESS [01:55 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [05:46 min]
[INFO] Spark Project SQL .................................. SUCCESS [07:22 min]
[INFO] Spark Project ML Library ........................... SUCCESS [04:28 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 11.460 s]
[INFO] Spark Project Hive ................................. SUCCESS [02:25 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 46.587 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 18.614 s]
[INFO] Spark Project YARN ................................. FAILURE [01:03 min]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [01:29 min]
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 38.343 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 49.537 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [01:25 min]
[INFO] Spark Project Examples ............................. SUCCESS [01:26 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 14.892 s]
[INFO] Spark Avro ......................................... SUCCESS [01:16 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  38:09 min
[INFO] Finished at: 2022-07-30T09:50:59Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce (enforce-no-duplicate-dependencies) on project spark-yarn_2.12: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-yarn_2.12

Here the log for the plugin which give errors:

[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-no-duplicate-dependencies) @ spark-yarn_2.12 ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicatePomDependencyVersions failed with message:
Found 1 duplicate dependency declaration in this project:
 - dependencies.dependency[org.apache.hadoop:hadoop-client:jar] ( 2 times )

[gonzaloetjo]

I had bypassed this error by adding -Denforcer.skip=true, I'll check if there's an other solution to evade the duplication, or if it affects anything else.

[SteBaum]

It is related to the fact that profile hadoop-3.1 has been added with two artifact names pointing to the same artifact hadoop-client while profile hadoop-3.2 is always activated. Decision is to deactivate profile hadoop-3.2 in maven command and add profile hadoop-3.1 where it is missing as copy of the latter profile but by removing the dependency which comes twice and test this modification on a new branch branch-3.2.4-build-new.