Open ZhenjieRuan opened 4 months ago
I've just run into what I think it the same bug with my fsspec adapter for csvbase.
Using fsspec for Parquet doesn't work in Polars
import polars as pl
df = pl.read_parquet("<some fsspec url>") # raises FileNotFoundError
But however it does work in Pandas
import pandas as pd
df = pd.read_parquet("<some fsspec url>") # works ok
and duckdb as well. So it would be great to get support for this in Polars.
Checks
Reproducible example
Log output
Issue description
Read parquet doesn't recognize oss url. However, the url works for read_csv, and the url + storage option works for
fsspec.open()
. Per the doc, If the cloud provider is not supported by Polars, the storage options are passed tofsspec.open()
. Clearly, it's not the case here. From the stack trace, looks likeread_parquet
somehow calledscan_parquet
, and per the doc, looks likescan_parquet
won't forward unsupported url tofsspec.open()
, this might be the issue here.Expected behavior
Should be able to read oss files directly
Installed versions