treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.47k stars 359 forks source link

RouterFS: Add support for multiple default FileSystem implementations #3426

Closed Jonathan-Rosenberg closed 2 years ago

Jonathan-Rosenberg commented 2 years ago

For each configured mapping there will be a mandatory requirement to add a default FileSystem implementation mapping. For example: For the following mappings:

routerfs.mapping.s3a.1.replace=s3a://bucket/dir1/
routerfs.mapping.s3a.1.with=lakefs://router/main/
routerfs.mapping.blablabla.1.replace=blablabla://bucket/dir2/
routerfs.mapping.blablabla.1.with=lakefs://router/dev/path

We'll also need to configure default FileSystem implementation mappings:

routerfs.default.fs.s3a=org.apache.hadoop.fs.s3a.S3AFileSystem
routerfs.default.fs.blablabla=org.apache.hadoop.fs.s3native.NativeS3FileSystem
talSofer commented 2 years ago

@Jonathan-Rosenberg can we close this issue? I think it was already implemented

Jonathan-Rosenberg commented 2 years ago

Absolutely