datafusion-contrib / datafusion-objectstore-s3

S3 as an ObjectStore for DataFusion
Apache License 2.0
59 stars 13 forks source link

Implement `list_dir` #41

Open matthewmturner opened 2 years ago

seddonm1 commented 2 years ago

I think leave this until we can see what list_dir is actually for. Currently the implementation is todo: https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/datasource/object_store/local.rs#L45

matthewmturner commented 2 years ago

I was thinking of doing it based on the definition in the Trait:

/// Returns all the files in `prefix` if the `prefix` is already a leaf dir,
/// or all paths between the `prefix` and the first occurrence of the `delimiter` if it is provided.
async fn list_dir(
        &self,
        prefix: &str,
        delimiter: Option<String>,
) -> Result<ListEntryStream>;

But def ok holding off if you think more info needed.