search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.86k stars 2.94k forks source link

failed to load jar /opt/alluxio-2.9.3/lib/alluxio-underfs-hadoop-cosn-3.1.0-5.8.5-2.9.3.jar NoClassdDefFoundError :alluxio/underfs/hdfs/HdfsUnderFileSystemFactory #17555

Open morpheusyu opened 1 year ago

morpheusyu commented 1 year ago

my presto is deployed in docker individually,alluxio is deployed in another docker container individually, alluxio can start well,when presto query hive table ,I can get the query result ,I try to turn on the short circuit read but failed becaues web ui indicater BytesReadLocalThroughput is always 0 no matter how i query,and the alluxio worker log report below error 51685944317_ pic_hd I think my failing turning on short circuit read maybe have something with above error ,can u give me some advice

jiacheliu3 commented 1 year ago

@maobaolong This error looks relevant to COSN, do you mind taking a look? Thanks!

maobaolong commented 1 year ago

@jiacheliu3 This is encountered by the following commit. pr-link: Alluxio/alluxio#17024

Remove HdfsUnderFileSystemFactory from service loading in COSN UFS jar Haoning Sun* 2023/3/9, 00:36

@Haoning-Sun Would you like to take a look?

humengyu2012 commented 1 year ago

We recently discovered this issue as well, and we wanted to try modifying some code in HdfsUnderFileSystem. However, we didn't want to compile the entire Alluxio project, so we only compiled the HDFS UFS module to replace the ufs-hdfs.jar file and implement our changes. But we found that replacing the ufs-hdfs.jar file had no effect. In the end, we discovered that HdfsUnderFileSystem actually comes from the ufs-cosn.jar file, which left us confused.

menwenjun commented 9 months ago

We also encountered this problem (Using Tencent Cloud Object Storage as the backend, using the cosn protocol), is there any progress now? @maobaolong @humengyu2012 @morpheusyu ee59720d-10fe-46b3-9fc3-15d7accd3b33

humengyu2012 commented 9 months ago

@menwenjun what‘s your alluxio version, it seems that the cosn protocol is not implemented in this version of alluxio. Cos also supports AWS s3 protocol, could you try it?

menwenjun commented 9 months ago

@menwenjun what‘s your alluxio version, it seems that the cosn protocol is not implemented in this version of alluxio. Cos also supports AWS s3 protocol, could you try it?

Thank you for your reply. I am using alluxio version 2.9.3. I will try to access cos with the s3 protocol.

humengyu2012 commented 9 months ago

@menwenjun what‘s your alluxio version, it seems that the cosn protocol is not implemented in this version of alluxio. Cos also supports AWS s3 protocol, could you try it?

Thank you for your reply. I am using alluxio version 2.9.3. I will try to access cos with the s3 protocol.

We are using s3 protocol for BOS and KS3 in zhihu, it is very nice. 😄

menwenjun commented 9 months ago

@menwenjun what‘s your alluxio version, it seems that the cosn protocol is not implemented in this version of alluxio. Cos also supports AWS s3 protocol, could you try it?

Thank you for your reply. I am using alluxio version 2.9.3. I will try to access cos with the s3 protocol.

We are using s3 protocol for BOS and KS3 in zhihu, it is very nice. 😄

I tried to use s3 to access cos, but the master reported an error. image The properties configuration I used is as follows. image

github-actions[bot] commented 4 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.