search

aliyun / alibabacloud-jindodata

alibabacloud-jindodata
Apache License 2.0
178 stars 46 forks source link

利用jindofs-sdk包初始化FileSystem对象实例失败 #108

Open baisui1981 opened 1 year ago

baisui1981 commented 1 year ago

按照 https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/3.x/hadoop/jindosdk_on_hadoop.md 文档所示，

  1. 下载jindofs-sdk，并且加入classpath中
  2. 在阿里云OSS中创建了基于OSS HDFS bucket，

  3. 本地依赖hadoop：3.2.1 版本相关的依赖包代码

    希望能够初始化org.apache.hadoop.fs.FileSystem 实例对象。

在执行过程中报告以下错误，不知道是哪儿出错了，本机是MacOS机器

Caused by: java.lang.RuntimeException: JbootNative loading failed: java.lang.UnsatisfiedLinkError: /private/tmp/libjboot-3397ed5b1f5602cb_20211216_104547.dylib: dlopen(/private/tmp/libjboot-3397ed5b1f5602cb_20211216_104547.dylib, 1): no suitable image found.  Did find:
    /private/tmp/libjboot-3397ed5b1f5602cb_20211216_104547.dylib: cannot load 'libjboot-3397ed5b1f5602cb_20211216_104547.dylib' (load command 0x80000034 is unknown)
    /private/tmp/libjboot-3397ed5b1f5602cb_20211216_104547.dylib: cannot load 'libjboot-3397ed5b1f5602cb_20211216_104547.dylib' (load command 0x80000034 is unknown)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at com.alibaba.jboot.NativeCodeLoader.loadLibrary(NativeCodeLoader.java:125)
    at com.alibaba.jboot.NativeCodeLoader.loadLibrary(NativeCodeLoader.java:82)
    at com.alibaba.jboot.NativeCodeLoader.load(NativeCodeLoader.java:44)
    at com.alibaba.jboot.JbootNative.<clinit>(JbootNative.java:19)
    at com.alibaba.jboot.JbootCache.<init>(JbootCache.java:11)
    at com.aliyun.emr.fs.internal.ossnative.OssFileletSystem.<init>(OssFileletSystem.java:36)
    at com.aliyun.emr.fs.internal.ossnative.OssNativeStore$Context.<init>(OssNativeStore.java:99)
    at com.aliyun.emr.fs.oss.JindoOssFileSystem.initNativeStoreContext(JindoOssFileSystem.java:140)
    at com.aliyun.emr.fs.oss.JindoOssFileSystem.initialize(JindoOssFileSystem.java:135)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$HdfsUtils.getFileSystem(HdfsFileSystemFactory.java:320)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$1.create(HdfsFileSystemFactory.java:173)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$1.visit(HdfsFileSystemFactory.java:149)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$1.visit(HdfsFileSystemFactory.java:145)
    at com.qlangtech.tis.config.authtoken.impl.OffUserToken.accept(OffUserToken.java:37)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory.createFS(HdfsFileSystemFactory.java:145)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory.getFileSystem(HdfsFileSystemFactory.java:113)
    at com.qlangtech.tis.hdfs.impl.TestAliayunJindoFSFactory.testListChild(TestAliayunJindoFSFactory.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

    at com.alibaba.jboot.JbootNative.checkLoaded(JbootNative.java:25)
    at com.alibaba.jboot.JbootNative.instance(JbootNative.java:30)
    at com.alibaba.jboot.JbootNative.initOnceNativeClass(JbootNative.java:74)
    at com.alibaba.jboot.JbootCache.<init>(JbootCache.java:11)
    at com.aliyun.emr.fs.internal.ossnative.OssFileletSystem.<init>(OssFileletSystem.java:36)
    at com.aliyun.emr.fs.internal.ossnative.OssNativeStore$Context.<init>(OssNativeStore.java:99)
    at com.aliyun.emr.fs.oss.JindoOssFileSystem.initNativeStoreContext(JindoOssFileSystem.java:140)
    at com.aliyun.emr.fs.oss.JindoOssFileSystem.initialize(JindoOssFileSystem.java:135)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
    at com.qlangtech.tis.hdfs.impl.HdfsFileSystemFactory$HdfsUtils.getFileSystem(HdfsFileSystemFactory.java:320)
    ... 32 more
yanbinyang commented 1 year ago

@baisui1981 你是不是 mac m1/m2 的机型？是的话，可能得用jindosdk-4.6.4-linux-el7-aarch64.tar.gz

baisui1981 commented 1 year ago

@baisui1981 你是不是 mac m1/m2 的机型？是的话，可能得用jindosdk-4.6.4-linux-el7-aarch64.tar.gz

谢谢如此快速的回复！我现在用的 jindofs-sdk包是否只能在centOS服务器上使用？我现在就按照你说的，试试

yanbinyang commented 1 year ago

@baisui1981 你是不是 mac m1/m2 的机型？是的话，可能得用jindosdk-4.6.4-linux-el7-aarch64.tar.gz

谢谢如此快速的回复！我现在用的 jindofs-sdk包是否只能在centOS服务器上使用？我现在就按照你说的，试试

不是，我们支持多平台使用的，4.x所有版本，基本上都支持macos x86、大部分linux发行版。 如果是macos arm版本，由于使用的人不多，目前只有4.6.4版本做了发布，以供demo使用。 下载链接参见jindodata_download.md。 安装说明参见install_dependeny_jindosdk.md

另外，你看的文档是我们的老版本3.x文档。目前最新4.x版本的文档参见jindosdk_ide_hadoop.md