I think the API is not correct to work with DataFusion:
Currently the AmazonS3FileSystem requires a bucket to be set. This means that one ObjectStore can only retrieve from one bucket. This does not fit with the DataFusion API which resolves ObjectStores by URI scheme (i.e. s3:// vs file://).
I will refactor this so we can dynamically request files correctly.
I think the API is not correct to work with DataFusion:
Currently the
AmazonS3FileSystemrequires a bucket to be set. This means that oneObjectStorecan only retrieve from one bucket. This does not fit with the DataFusion API which resolvesObjectStoresby URI scheme (i.e.s3://vsfile://).I will refactor this so we can dynamically request files correctly.