datafusion-contrib / datafusion-objectstore-s3

S3 as an ObjectStore for DataFusion
Apache License 2.0
59 stars 13 forks source link

Unable to use S3FileSystem with RuntimeEnv::register_object_store #70

Closed jwimberl closed 1 year ago

jwimberl commented 1 year ago

I am trying to add an S3 object store for a project using datafusion version 28. The README.md for this project notes that the S3FileSystem can be registered as an ObjectStore on an ExecutionContext, but that latter trait is not something that exists any longer in datafusion. Instead, the register_object_store function is now part of the RuntimeEnv trait. Given a SessionContext object ctx, I am trying to register the S3 store with

use datafusion_objectstore_s3::object_store::s3::S3FileSystem;
...
let s3_file_system = Arc::new(S3FileSystem::default().await);
let rv = ctx.runtime_env();
let s3url = Url::parse("s3").unwrap();
rv.register_object_store(&s3url, s3_file_system);

However, this produces the following error:

error[E0277]: the trait bound `S3FileSystem: object_store::ObjectStore` is not satisfied
   --> src/main.rs:127:42
    |
127 |         rv.register_object_store(&s3url, s3_file_system);
    |                                          ^^^^^^^^^^^^^^ the trait `object_store::ObjectStore` is not implemented for `S3FileSystem`
    |
    = help: the following other types implement trait `object_store::ObjectStore`:
              Box<(dyn object_store::ObjectStore + 'static)>
              object_store::chunked::ChunkedStore
              object_store::limit::LimitStore<T>
              object_store::local::LocalFileSystem
              object_store::memory::InMemory
              object_store::prefix::PrefixStore<T>
              object_store::throttle::ThrottledStore<T>
    = note: required for the cast from `S3FileSystem` to the object type `dyn object_store::ObjectStore`

For more information about this error, try `rustc --explain E0277`.
error: could not compile `shimsql` due to previous error

Is this crate incompatible with datfusion 28, or is there some using declaration I'm missing that's necessary for this trait implementation to be visible?

matthewmturner commented 1 year ago

hi @jwimberl - this project hasnt been maintained in a while. I would use https://docs.rs/object_store/latest/object_store/ if you are looking for object store functionality.

jwimberl commented 1 year ago

@matthewmturner Thank you for the pointer! I was able to use that crate successfully