Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.73k stars 2.92k forks source link

Support access more ufs in dora worker #17575

Open jffree opened 1 year ago

jffree commented 1 year ago

Alluxio version Dora(3.x)

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

  1. When the accessed path does not have schema information, add the prefix according to alluxio.dora.client.ufs.root.
    1. set alluxio.dora.client.ufs.root=/tmp
    2. access a/b/c/d in client
    3. add prefix in dora worker like /tmp/a/b/c/d
    4. use root ufs client access /tmp/a/b/c/d
  2. When the accessed path has schema information, build a new ufs client based on the schema information to access the path.
    1. set alluxio.dora.client.ufs.root=/tmp
    2. access hdfs://ns17/user/test/1.txx in client
    3. build hdfs ufs client in dora ufs (cache ufs client in dora worker)
    4. access hdfs://ns17/user/test/1.txx file by hdfs ufs client.
dbw9580 commented 1 year ago

Thanks for the feature request. This is also what we want for the Dora architecture.

maobaolong commented 1 year ago

@bzheng888 Please keep an eye on this issue, it looks a good feature.

github-actions[bot] commented 6 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.