search

aliyun / alibabacloud-jindodata

alibabacloud-jindodata
Apache License 2.0
179 stars 46 forks source link

希望提供hadoop-common v3.2.x 兼容的JindoData Maven 依赖 #111

Closed baisui1981 closed 1 year ago

baisui1981 commented 1 year ago

现象

当在项目工程中使用如下maven 依赖，由于将hadoop的版本依赖改成了3.2.1 导致（没有使用官方文档使用的2.8.5 https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/6.x/6.1.0/oss-maven.md）， 导致执行任务过程中报如下方法引用错误：

Caused by: java.lang.LinkageError: loader constraint violation: 
when resolving overridden method "com.aliyun.jindodata.common.JindoHadoopSystem.getStoragePolicy(Lorg/apache/hadoop/fs/Path;)Lorg/apache/hadoop/fs/BlockStoragePolicySpi;"
 the class loader (instance of com/qlangtech/tis/extension/impl/PluginFirstClassLoader) of 
the current class, com/aliyun/jindodata/common/JindoHadoopSystem
, and its superclass loader (instance of com/qlangtech/tis/extension/impl/PluginFirstClassLoader)
, have different Class objects for the type org/apache/hadoop/fs/BlockStoragePolicySpi used in the signature
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2575)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2540)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$HdfsUtils.getFileSystem(HdfsFileSystemFactory.java:322)

原因

我查看了hadoop-common 3.2.1 中有 也有类： BlockStoragePolicySpi.class 

[root@baisui-test-1 data]# jar tvf ./libs/plugins/tis-datax-hdfs-plugin_hadoop_3.2.1/WEB-INF/lib/hadoop-common-3.2.1.jar | grep BlockStoragePolicySpi
   817 Tue Sep 10 15:56:36 CST 2019 org/apache/hadoop/fs/BlockStoragePolicySpi.class

明显这个类 与 Jindo-sdk 中自带的BlockStoragePolicySpi类冲突了

期望

JindoData 是否能提供hadoop 3.x 兼容的maven依赖？

使用的Maven POM

    <properties>
        <hadoop.version>3.2.1</hadoop.version>
        <jindodata.version>6.1.0</jindodata.version>
    </properties>
 <dependencies>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <!-- add jindo-core -->
        <!--https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/6.x/6.0.1/oss-maven.md-->
        <dependency>
            <groupId>com.aliyun.jindodata</groupId>
            <artifactId>jindo-core</artifactId>
            <version>${jindodata.version}</version>
        </dependency>

        <dependency>
            <groupId>com.aliyun.jindodata</groupId>
            <artifactId>jindo-sdk</artifactId>
            <version>${jindodata.version}</version>
        </dependency>

    </dependencies>
adrian-wang commented 1 year ago

可以用sudo zip -d /opt/apps/JINDOSDK/jindosdk-current/lib/jindo-sdk-*.jar org/apache/hadoop/fs/BlockStoragePolicySpi.class 排除掉这个类

baisui1981 commented 1 year ago

可以用sudo zip -d /opt/apps/JINDOSDK/jindosdk-current/lib/jindo-sdk-*.jar org/apache/hadoop/fs/BlockStoragePolicySpi.class 排除掉这个类

ok ,谢谢指点，我用maven-shade-plugin 来删除