search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.86k stars 2.94k forks source link

generate-tarballs build failure #9416

Closed phenixmzy closed 5 years ago

phenixmzy commented 5 years ago

Alluxio Version: v2.0.0

Describe the bug used mvn install -P -Dhadoop.version= -DskipTests can be compile success. i want to package tar and running ./dev/scripts/generate-tarballs release -ufs-modules=ufs-hadoop-2.7 -debug=true and ./dev/scripts/generate-tarballs single -ufs-modules=ufs-hadoop-2.7 -debug=true and ./dev/scripts/generate-tarballs release -ufs-modules=all -debug=true to compile and package tar are all failed.

the compile error log [INFO] ------------------< org.alluxio:alluxio-underfs-hdfs >------------------ [INFO] Building Alluxio Under File System - HDFS 2.0.0 [7/7] [INFO] --------------------------------[ jar ]--------------------------------- Downloading from HDPReleases: http://repo.hortonworks.com/content/repositories/releases/org/alluxio/alluxio-shaded-hadoop/2.7.3/alluxio-shaded-hadoop-2.7.3.pom [WARNING] The POM for org.alluxio:alluxio-shaded-hadoop:jar:2.7.3 is missing, no dependency information available Downloading from HDPReleases: http://repo.hortonworks.com/content/repositories/releases/org/alluxio/alluxio-shaded-hadoop/2.7.3/alluxio-shaded-hadoop-2.7.3.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Alluxio Parent 2.0.0: [INFO] [INFO] Alluxio Parent ..................................... SUCCESS [ 18.287 s] [INFO] Alluxio Core ....................................... SUCCESS [ 0.671 s] [INFO] Alluxio Core - Transport ........................... SUCCESS [ 11.587 s] [INFO] Alluxio Core - Base module ......................... SUCCESS [ 1.046 s] [INFO] Alluxio Core - Common Utilities .................... SUCCESS [ 3.168 s] [INFO] Alluxio Under File System .......................... SUCCESS [ 0.202 s] [INFO] Alluxio Under File System - HDFS ................... FAILURE [ 1.014 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 36.632 s (Wall Clock) [INFO] Finished at: 2019-07-04T14:05:52+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project alluxio-underfs-hdfs: Could not resolve dependencies for project org.alluxio:alluxio-underfs-hdfs:jar:2.0.0: Failure to find org.alluxio:alluxio-shaded-hadoop:jar:2.7.3 in https://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :alluxio-underfs-hdfs

exit status 1

I try to request
http://repo.hortonworks.com/content/repositories/releases/org/alluxio/alluxio-shaded-hadoop/2.7.3/alluxio-shaded-hadoop-2.7.3.pom get the response 404 - Path /org/alluxio/alluxio-shaded-hadoop/2.7.3/alluxio-shaded-hadoop-2.7.3.pom not found in local storage of repository "Releases" [id=releases]

apc999 commented 5 years ago

thanks for reporting this issue @phenixmzy . do you run the generate-tarball command on a clean checkout of the src repo, or have you modified any thing related to Hadoop version in the src repo?

phenixmzy commented 5 years ago

i am not modified any thing related to Hadoop version in the src repo.

i just was run several cmd：

when i can't find the about alluxio-2.0.0.tar.gz ，run：

still can't find the about alluxio-2.0.0.tar.gz , try to run:

  1. ./dev/scripts/generate-tarballs release -ufs-modules=ufs-hadoop-2.7 -debug=true
  2. ./dev/scripts/generate-tarballs single -ufs-modules=ufs-hadoop-2.7 -debug=true
  3. ./dev/scripts/generate-tarballs release -ufs-modules=all -debug=true

all show the error infomation: [ERROR] Failed to execute goal on project alluxio-underfs-hdfs: Could not resolve dependencies for project org.alluxio:alluxio-underfs-hdfs:jar:2.0.0: Failure to find org.alluxio:alluxio-shaded-hadoop:jar:2.7.3 in https://repo1.maven.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

apc999 commented 5 years ago

Neither mvn install -Phadoop-2 -Dhadoop.version=2.7.3 -DskipTests nor mvn clean package -Phadoop-2 -Dhadoop.version=2.7.3 -DskipTests is supposed to create alluxio-2.0.0.tar.gz. They are command to build local repo by maven.

./dev/scripts/generate-tarballs is supposed to achieve this goal. Were you able to find any file in your local repo like ${HOME}/.m2/repository/org/alluxio/alluxio-shaded-hadoop/2.7.3?

phenixmzy commented 5 years ago

@apc999

when compile compled use mvn install -Phadoop-2 -Dhadoop.version=2.7.3 -DskipTests or mvn clean package -Phadoop-2 -Dhadoop.version=2.7.3 -DskipTests

really cant't find any alluxio-*.tar.gz file in source code directories

Last compiled log: INFO] Reactor Summary: [INFO] [INFO] Alluxio Parent 2.0.0 ............................... SUCCESS [ 19.437 s] [INFO] Alluxio Core 2.0.0 ................................. SUCCESS [ 1.622 s] [INFO] Alluxio Core - Transport 2.0.0 ..................... SUCCESS [ 34.616 s] [INFO] Alluxio Core - Base module 2.0.0 ................... SUCCESS [ 11.497 s] [INFO] Alluxio Core - Common Utilities 2.0.0 .............. SUCCESS [ 29.883 s] [INFO] Alluxio Core - Client 2.0.0 ........................ SUCCESS [ 0.636 s] [INFO] Alluxio Core - Client - File System 2.0.0 .......... SUCCESS [ 11.547 s] [INFO] Alluxio Examples 2.0.0 ............................. SUCCESS [ 9.371 s] [INFO] Alluxio Job Service 2.0.0 .......................... SUCCESS [ 0.724 s] [INFO] Alluxio Job Service - Common Utilities 2.0.0 ....... SUCCESS [ 5.872 s] [INFO] Alluxio Job Service - Client 2.0.0 ................. SUCCESS [ 5.753 s] [INFO] Alluxio Shell 2.0.0 ................................ SUCCESS [ 11.258 s] [INFO] Alluxio Assembly 2.0.0 ............................. SUCCESS [ 0.617 s] [INFO] Alluxio Assembly - Client 2.0.0 .................... SUCCESS [ 7.228 s] [INFO] Alluxio Core - Server 2.0.0 ........................ SUCCESS [ 0.798 s] [INFO] Alluxio Core - Server - Common Utilities 2.0.0 ..... SUCCESS [ 15.768 s] [INFO] Alluxio Under File System 2.0.0 .................... SUCCESS [ 1.013 s] [INFO] Alluxio Under File System - Local FS 2.0.0 ......... SUCCESS [ 4.664 s] [INFO] Alluxio Core - Server - Master 2.0.0 ............... SUCCESS [ 24.051 s] [INFO] Alluxio Core - Server - Proxy 2.0.0 ................ SUCCESS [ 8.487 s] [INFO] Alluxio Core - Server - Worker 2.0.0 ............... SUCCESS [ 14.171 s] [INFO] Alluxio Job Service - Server 2.0.0 ................. SUCCESS [ 9.870 s] [INFO] Alluxio Log Server 2.0.0 ........................... SUCCESS [ 4.444 s] [INFO] Alluxio Assembly - Server 2.0.0 .................... SUCCESS [ 7.043 s] [INFO] Alluxio Core - Client - HDFS 2.0.0 ................. SUCCESS [ 6.379 s] [INFO] Alluxio Core - Client - Runtime 2.0.0 .............. SUCCESS [ 12.687 s] [INFO] Alluxio Integration 2.0.0 .......................... SUCCESS [ 0.874 s] [INFO] Alluxio Integration - Checker 2.0.0 ................ SUCCESS [ 26.726 s] [INFO] Alluxio Integration - FUSE 2.0.0 ................... SUCCESS [ 9.452 s] [INFO] Alluxio MiniCluster 2.0.0 .......................... SUCCESS [ 7.459 s] [INFO] Alluxio Shaded Libraries 2.0.0 ..................... SUCCESS [ 0.536 s] [INFO] Alluxio Shaded Libraries - Hadoop 2.2.0 ............ SUCCESS [ 10.058 s] [INFO] Alluxio Tests 2.0.0 ................................ SUCCESS [ 20.597 s] [INFO] Alluxio Under File System - Tencent Cloud COS 2.0.0 SUCCESS [ 5.645 s] [INFO] Alluxio Under File System - GCS 2.0.0 .............. SUCCESS [ 5.502 s] [INFO] Alluxio Under File System - HDFS 2.0.0 ............. SUCCESS [ 10.057 s] [INFO] Alluxio Under File System - Qiniu Kodo 2.0.0 ....... SUCCESS [ 5.114 s] [INFO] Alluxio Under File System - Aliyun OSS 2.0.0 ....... SUCCESS [ 5.710 s] [INFO] Alluxio Under File System - S3 2.0.0 ............... SUCCESS [ 6.372 s] [INFO] Alluxio Under File System - Swift 2.0.0 ............ SUCCESS [ 5.756 s] [INFO] Alluxio Under File System - Microsoft Azure Blob Storage 2.0.0 SUCCESS [ 10.865 s] [INFO] Alluxio Under File System - Web 2.0.0 .............. SUCCESS [ 4.569 s] [INFO] Alluxio UI 2.0.0 ................................... SUCCESS [01:45 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 08:21 min [INFO] Finished at: 2019-07-08T10:43:58+08:00 [INFO] ------------------------------------------------------------------------ [root@client-test alluxio]# find . -name 'alluxio-.tar.gz' [root@client-test alluxio]# ll -h total 112K drwxr-xr-x 5 root root 75 Jul 8 10:11 assembly drwxr-xr-x 2 root root 4.0K Jul 1 16:06 bin drwxr-xr-x 7 root root 79 Jul 1 16:06 build drwxr-xr-x 2 root root 37 Jul 1 16:48 client drwxr-xr-x 2 root root 4.0K Jul 1 16:06 conf drwxr-xr-x 8 root root 99 Jul 8 10:08 core drwxr-xr-x 4 root root 34 Jul 1 16:06 dev drwxr-xr-x 12 root root 4.0K Jul 1 16:17 docs drwxr-xr-x 4 root root 43 Jul 8 10:10 examples drwxr-xr-x 6 root root 54 Jul 1 16:47 generated drwxr-xr-x 11 root root 135 Jul 8 10:13 integration drwxr-xr-x 6 root root 72 Jul 8 10:10 job drwxr-xr-x 2 root root 4.0K Jul 8 10:42 lib drwxr-xr-x 2 root root 30 Jul 1 16:17 libexec -rw-r--r-- 1 root root 27K Jul 1 16:06 LICENSE drwxr-xr-x 2 root root 22 Jul 1 16:06 logs drwxr-xr-x 4 root root 43 Jul 8 10:12 logserver drwxr-xr-x 4 root root 43 Jul 8 10:13 minicluster -rw-r--r-- 1 root root 4.3K Jul 1 16:06 NOTICE -rw-r--r-- 1 root root 46K Jul 1 16:17 pom.xml -rw-r--r-- 1 root root 5.3K Jul 1 16:06 README.md drwxr-xr-x 4 root root 46 Jul 8 10:14 shaded drwxr-xr-x 4 root root 43 Jul 8 10:10 shell drwxr-xr-x 2 root root 124 Jul 8 10:08 target drwxr-xr-x 2 root root 70 Jul 1 16:06 templates drwxr-xr-x 4 root root 43 Jul 8 10:14 tests drwxr-xr-x 13 root root 140 Jul 8 10:11 underfs drwxr-xr-x 7 root root 4.0K Jul 8 10:42 webui [root@client-test alluxio]# find . -name 'alluxio-*.tar.gz' [root@client-test alluxio]# find . -name '\.tar.gz'

./dev/scripts/generate-tarballs yes , i can find the .m2/repository/org/alluxio/alluxio-shaded-hadoop/2.7.3 in /root which exists 2 file alluxio-shaded-hadoop-2.7.3.jar.lastUpdated alluxio-shaded-hadoop-2.7.3.pom.lastUpdated， in addition the MAVEN_HOME in /usr/local/

apc999 commented 5 years ago

I could repro this issue on a clean checkout on an EC2 machine with Linux AMI Here is what is off: when compiling alluxio-shaded-hadoop by running mvn -T 2C -am clean install -DskipTests -Dfindbugs.skip -Dmaven.javadoc.skip -Dcheckstyle.skip -Pmesos -Dhadoop.version=2.7.3 -Phadoop-2 -Pyarn -Pufs-hadoop-2 -Dufs.hadoop.version=2.7.3 -pl shaded/hadoop, the version is correctly set at my local MacOS to be 2.7.3 but mistakenly set to be 2.2.0 (default Hdfs) on my EC2 for some reason. Both are running maven 3.6.0.

[INFO] --- maven-shade-plugin:3.2.1:shade (shade-alluxio) @ alluxio-shaded-hadoop ---
[INFO] Including com.google.guava:guava:jar:14.0.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.7.3 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.7.3 in the shaded jar.
...
...
[INFO] Excluding org.objenesis:objenesis:jar:2.1 from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /home/ec2-user/alluxio/shaded/hadoop/target/alluxio-shaded-hadoop-2.2.0.jar with /home/ec2-user/alluxio/shaded/hadoop/target/alluxio-shaded-hadoop-2.2.0-shaded.jar
[INFO] Dependency-reduced POM written at: /home/ec2-user/alluxio/shaded/hadoop/dependency-reduced-pom.xml
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ alluxio-shaded-hadoop ---
[INFO] Installing /home/ec2-user/alluxio/shaded/hadoop/target/alluxio-shaded-hadoop-2.2.0.jar to /home/ec2-user/.m2/repository/org/alluxio/alluxio-shaded-hadoop/2.2.0/alluxio-shaded-hadoop-2.2.0.jar
[INFO] Installing /home/ec2-user/alluxio/shaded/hadoop/dependency-reduced-pom.xml to /home/ec2-user/.m2/repository/org/alluxio/alluxio-shaded-hadoop/2.2.0/alluxio-shaded-hadoop-2.2.0.pom

In addition, @calvinjia the current way to version alluxio-shaded-hadoop is completely on $ufs.hadoop.version which can be flawed: since the artifact version has nothing to do with Alluxio version, this may block the future publication of maven artifact as Alluxio 2.0.1 may collide with Alluxio 2.0.0 on artifact alluxio-shaded-hadoop/2.2.0/. We need to fix this (e.g., make the version a combo of Alluxio version and Hadoop version). Upgrading this issue to be high priority for 2.0.1

apc999 commented 5 years ago

@rastogiasr @calvinjia , this issue may potentially block publishing future Alluxio maven artifact. Change to priority-high and we should aim to address it before 2.0.1

asin929 commented 5 years ago

这个是的alluxio-shaded module的pom文件版本写死导致。

<name>Alluxio Shaded Libraries - Hadoop</name>
<version>2.2.1-SNAPSHOT</version>
<description>Shaded Hadoop Module</description>

应该改为，

  <name>Alluxio Shaded Libraries - Hadoop</name>
  <version>${ufs.hadoop.version}</version>
  <description>Shaded Hadoop Module</description>
apc999 commented 5 years ago

@asin929 where do you find this hard-coded version string? it is actually not found in https://github.com/Alluxio/alluxio/blob/master/shaded/hadoop/pom.xml or in the branch-2.0. This is what we use right now, identical to what you are proposing:

  <name>Alluxio Shaded Libraries - Hadoop</name>
  <version>${ufs.hadoop.version}</version>
  <description>Shaded Hadoop Module</description>
calvinjia commented 5 years ago

@apc999 I think this was a bug caused by the maven release plugin. It identified the version string as a module to release and updated it to a hardcoded value. In the future we should exclude this module from being updated.

apc999 commented 5 years ago

PR #9428 is the quick solution for this issue. Created another issue https://github.com/Alluxio/alluxio/issues/9432 as the clean solution for this bug. Close this PR now.