Currently, only local file is supported as a source or sink, though it doesn't work well for distributed environment.
Though user should be able to specify any other way to handle files. For that purpose, first of all need to abstract away the File System Driver interface. File system Driver will be determined based on the protocol in the file URI, i.e. file:///path/file.txt in local file system, dropbox:///path/file.txt for accessing files on DropBox and so on.
The file system might be any like s3 or hdfs or gcs, and the registry may scan the classpath for available implementations on startup.
Currently, only local file is supported as a source or sink, though it doesn't work well for distributed environment.
Though user should be able to specify any other way to handle files. For that purpose, first of all need to abstract away the File System Driver interface. File system Driver will be determined based on the protocol in the file URI, i.e.
file:///path/file.txtin local file system,dropbox:///path/file.txtfor accessing files on DropBox and so on.The file system might be any like
s3orhdfsorgcs, and the registry may scan the classpath for available implementations on startup.Would be nice to reuse some of the existing implementations, i.e. from Flink https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/