Open a-agmon opened 10 months ago
Glob expressions are currently only supported for local files, the docs can be found here - https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTableUrl.html#paths-without-a-scheme
I can't remember the context on why though, I think it was to avoid needing to escape glob expression... It should be possible to add an API to allow creating a listing table with a prefix and a glob expression
Thanks @tustvold ,
I think its very useful, especially when we want to register a table on a certain partition path - e.g. s3://somebucket/data/source=*/env=prod/*.parquet
Describe the bug
Im trying to register a listing table using a certain glob string:
s3://somebucket/somepath/*44fad0765ac6-00001.parquet
and receive the errorhowever, I know the file is there because the action succeeds when I just using the full file name
s3://somebucket/somepath/00000-2367-2918fbc9-ea04-4927-a669-44fad0765ac6-00001.parquet
from the error, I have a feeling it's related to how the url and gloc char is being encoded, looks like its being encoded twice for some reason - from * -> %2A -> %252A
To Reproduce
Expected behavior
the files that correspond to the glob string will be registered as a table. In the code above if I just switch between the ListingTableUrl with the globe string to that of the schema the code executes successfully (so no permissions issue)/
Additional context
My Cargo.Toml