Open nickalexander053 opened 2 months ago
@electrum do we plan to expose local file system factory?
Yes, we can do this, but we should have a required configuration parameter that specifies the local root for the file system, so that users don’t need to allow access to the entire machine. (assuming no symlinks pointing outside)
Many distributed file systems (WEKA, VAST, IBM Spectrum Scale) provide accelerated access to the distributed file system by using RDMA over ethernet or infiniband. These are mounted on the host using either a POSIX client or as an NFS mount. Although almost all of these solutions also provide an S3a interface, the RDMA based solution are substantially faster and use substantially less CPU resources. All of these solutions are available either on premise or in the cloud.
Currently Trino only supports access to object based storage. This is realised through implementations of
FileSystem
andFileSystemFactory
and binding the schema to an implementation ofFileSystemFactory
in the setup method ofio.trino.filesystem.manager.FileSystemModule.
Even though Trino does not officially support non-object storage, it does have an implementation of local file system through
LocalFileSystemFactory
andLocalFileSystem
. By adding the following code to theFileSystemModule
(and adding a config variable) setup method, local file system (a.k.a still a distributed file system) can be added: