Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.86k stars 2.94k forks source link

Build requires two runs in a clean-checked out Alluxio 1.8 branch #8592

Closed apc999 closed 5 years ago

apc999 commented 5 years ago

Alluxio Version: 1.8.1

Describe the bug source code of Alluxio 1.8 branch requires two builds with the first build failing but generating a preprocessed Java src (alluxio/tests/target/generated-test-sources/preprocessed)

To Reproduce

$ git clone https://github.com/Alluxio/alluxio.git & cd alluxio
$ git fetch origin branch-1.8
$ git checkout branch-1.8
$ mvn clean package install -Dhadoop.version=2.6.0 -Pyarn -Pspark -Dlicense.skip -DskipTests -Dfindbugs.skip -Dmaven.javadoc.skip -Dcheckstyle.skip
...
[ERROR] Failed to execute goal com.igormaznitsa:jcp:6.1.2:preprocessTests (preprocessSources) on project alluxio-tests: Can't find a source directory [/Users/binfan/tmp/alluxio/tests/target/generated-test-sources/preprocessed] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :alluxio-tests
$ mvn -rf :alluxio-tests
... it works now ...

Expected behavior It should work in one run

Additional context This issue is reported from the mailing list: https://groups.google.com/forum/#!topic/alluxio-users/ZbDpRYp3v00 and reproduced by me on my local MacBook. I suspect the problem is due to that the phase of generating preprocessed Java src was executed in a wrong order

Reidddddd commented 5 years ago

Hi Bin, Thanks for filing this issue. What you mentioned seems not working on centos6.

I tried

~/xu.wenchun/apache-maven-3.3.9/bin/mvn -rf :alluxio-tests

or

~/xu.wenchun/apache-maven-3.3.9/bin/mvn clean package install -Dhadoop.version=2.6.0 -Pyarn -Pspark -Dlicense.skip -DskipTests -Dfindbugs.skip -Dmaven.javadoc.skip -Dcheckstyle.skip -rf :alluxio-test

The former complains NoGoalSpecifiedException, while the latter is the same as e-mail mentioned.

FYI, I paste the latter build info:

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] Alluxio Tests [INFO] Alluxio Under File System - Tencent Cloud COS [INFO] Alluxio Under File System - GCS [INFO] Alluxio Under File System - HDFS [INFO] Alluxio Under File System - Aliyun OSS [INFO] Alluxio Under File System - S3A [INFO] Alluxio Under File System - Swift [INFO] Alluxio Under File System - Microsoft Azure Blob Storage [INFO] Alluxio Integration - YARN [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Alluxio Tests 1.8.1 [INFO] ------------------------------------------------------------------------ Downloading: http://repo.hortonworks.com/content/repositories/releases/org/glassfish/javax.el/maven-metadata.xml [WARNING] The POM for org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.glassfish:javax.el:jar:3.0.1-b07-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.glassfish:javax.el:jar:3.0.1-b08-SNAPSHOT is missing, no dependency information available [WARNING] The POM for org.glassfish:javax.el:jar:3.0.1-b11-SNAPSHOT is missing, no dependency information available [INFO] [INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ alluxio-tests --- [INFO] Deleting /home/hadoop/alluxio/tests/target [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-maven) @ alluxio-tests --- [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ alluxio-tests --- [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ alluxio-tests --- [INFO] [INFO] --- license-maven-plugin:2.9:check (default) @ alluxio-tests --- [INFO] Checking licenses... [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ alluxio-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/hadoop/alluxio/tests/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ alluxio-tests --- [INFO] No sources to compile [INFO] [INFO] >>> findbugs-maven-plugin:3.0.5:check (default) > :findbugs @ alluxio-tests >>> [INFO] [INFO] --- findbugs-maven-plugin:3.0.5:findbugs (findbugs) @ alluxio-tests --- [INFO] [INFO] <<< findbugs-maven-plugin:3.0.5:check (default) < :findbugs @ alluxio-tests <<< [INFO] [INFO] --- findbugs-maven-plugin:3.0.5:check (default) @ alluxio-tests --- [INFO] [INFO] --- jcp:6.1.2:preprocessTests (preprocessSources) @ alluxio-tests --- [INFO] Preprocess sources : /home/hadoop/alluxio/tests/src/test/java [INFO] Preprocess destination : /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed [INFO] ----------------------------------------------------------------- [INFO] Completed, preprocessed 151 files, copied 0 files, elapsed time 161ms [INFO] A Compile source root has been removed from the root list [/home/hadoop/alluxio/tests/src/test/java] [INFO] The New compile source root has been added into the list [/home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed] [INFO] [INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ alluxio-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.2:testCompile (default-testCompile) @ alluxio-tests --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 143 source files to /home/hadoop/alluxio/tests/target/test-classes [INFO] /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed/alluxio/client/rest/FileSystemMasterClientRestApiTest.java: Some input files use or override a deprecated API. [INFO] /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed/alluxio/client/rest/FileSystemMasterClientRestApiTest.java: Recompile with -Xlint:deprecation for details. [INFO] /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed/alluxio/server/ft/MasterFaultToleranceIntegrationTest.java: /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed/alluxio/server/ft/MasterFaultToleranceIntegrationTest.java uses unchecked or unsafe operations. [INFO] /home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed/alluxio/server/ft/MasterFaultToleranceIntegrationTest.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-surefire-plugin:2.19.1:test (default-test) @ alluxio-tests --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ alluxio-tests --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] Building jar: /home/hadoop/alluxio/tests/target/alluxio-tests-1.8.1.jar [INFO] [INFO] >>> maven-source-plugin:2.3:jar (attach-sources) > generate-sources @ alluxio-tests >>> [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-maven) @ alluxio-tests --- [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ alluxio-tests --- [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ alluxio-tests --- [INFO] [INFO] --- license-maven-plugin:2.9:check (default) @ alluxio-tests --- [INFO] Checking licenses... [INFO] [INFO] <<< maven-source-plugin:2.3:jar (attach-sources) < generate-sources @ alluxio-tests <<< [INFO] [INFO] --- maven-source-plugin:2.3:jar (attach-sources) @ alluxio-tests --- [INFO] No sources in project. Archive not created. [INFO] [INFO] --- maven-javadoc-plugin:2.9:jar (attach-javadoc) @ alluxio-tests --- [INFO] Skipping javadoc generation [INFO] [INFO] --- jcp:6.1.2:clear (clearGeneratedFolders) @ alluxio-tests --- [INFO] Cleaning has been started [INFO] Removing preprocessed source folder '/home/hadoop/alluxio/tests/target/generated-sources/preprocessed' [INFO] Preprocessed Source folder '/home/hadoop/alluxio/tests/target/generated-sources/preprocessed' doesn't exist [INFO] Removing preprocessed test source folder '/home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed' [INFO] Cleaning has been completed [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-maven) @ alluxio-tests --- [INFO] [INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ alluxio-tests --- [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ alluxio-tests --- [INFO] [INFO] --- license-maven-plugin:2.9:check (default) @ alluxio-tests --- [INFO] Checking licenses... [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ alluxio-tests --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/hadoop/alluxio/tests/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ alluxio-tests --- [INFO] No sources to compile [INFO] [INFO] >>> findbugs-maven-plugin:3.0.5:check (default) > :findbugs @ alluxio-tests >>> [INFO] [INFO] --- findbugs-maven-plugin:3.0.5:findbugs (findbugs) @ alluxio-tests --- [INFO] [INFO] <<< findbugs-maven-plugin:3.0.5:check (default) < :findbugs @ alluxio-tests <<< [INFO] [INFO] --- findbugs-maven-plugin:3.0.5:check (default) @ alluxio-tests --- [INFO] [INFO] --- jcp:6.1.2:preprocessTests (preprocessSources) @ alluxio-tests --- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Alluxio Tests ...................................... FAILURE [ 6.488 s] [INFO] Alluxio Under File System - Tencent Cloud COS ...... SKIPPED [INFO] Alluxio Under File System - GCS .................... SKIPPED [INFO] Alluxio Under File System - HDFS ................... SKIPPED [INFO] Alluxio Under File System - Aliyun OSS ............. SKIPPED [INFO] Alluxio Under File System - S3A .................... SKIPPED [INFO] Alluxio Under File System - Swift .................. SKIPPED [INFO] Alluxio Under File System - Microsoft Azure Blob Storage SKIPPED [INFO] Alluxio Integration - YARN ......................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 6.999 s [INFO] Finished at: 2019-03-19T11:04:47+08:00 [INFO] Final Memory: 78M/2192M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "spark" could not be activated because it does not exist. [ERROR] Failed to execute goal com.igormaznitsa:jcp:6.1.2:preprocessTests (preprocessSources) on project alluxio-tests: Can't find a source directory [/home/hadoop/alluxio/tests/target/generated-test-sources/preprocessed] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

ZacBlanco commented 5 years ago

@Reidddddd

You'll still need to specify a maven goal with that command in order to execute the build. -rf :<module> will only start the build from that maven module. To attempt to remedy you can try adding in a maven goal:

mvn -rf :alluxio-tests <intended maven target>

i.e


mvn -rf :alluxio-tests install
calvinjia commented 5 years ago

The problem is when multiple build phases like package and install are specified, the preprocessing plugin gets confused and tries to preprocess multiple times (preprocess, delete, 2nd preprocess will fail).

apc999 commented 5 years ago

Get it. close this issue now