search

Alluxio / Community

New Contributor Tasks for Alluxio
20 stars 38 forks source link

Shade alluxio.underfs.wasb #590

Closed LuQQiu closed 3 years ago

LuQQiu commented 3 years ago

Prerequisite: Maven knowledge

Alluxio.underfs.wasb has dependencies related to hadoop like 

    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>${wasb.hadoop.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>${wasb.hadoop.version}</version>
    </dependency>

Alluxio default hadoop version is 3.3, while underfs like wasb using a hadoop version different than the default hadoop version, there is easy to have exceptions like 

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.util.VersionInfo

basically wasb cannot pick up the hadoop version it wants but using the default hadoop version. Incompatible problems occur.

One way to address this issue is using the maven-shaded-plugin. Examples are alluxio-underfs-cosn and alluxio-underfs-ozone

Step 1: Understand the compatibility issue brought by hadoop packages. Step 2: Take a look at the https://github.com/Alluxio/alluxio/blob/master/underfs/cosn/pom.xml https://github.com/Alluxio/alluxio/blob/master/shaded/cosn/pom.xml and maven shaded plugin related stuff Step 3: Create a shaded/wasb/pom.xml and modify the underfs/wasb/pom.xml

Expect time: 1 week

LuQQiu commented 3 years ago

Close for now since it may not be needed