grofte
commented
3 months ago It's because Kedro opens the file and somehow doesn't recognize it as a bytes / io.BufferedIOBase / io.RawIOBase (or Polars doesn't) and therefore sends it to .scan_parquet which only takes paths as an argument.
If you do this is catalog.yml it will work:
load_args:
use_pyarrow: trueKedro sends it off to Polars here: https://github.com/kedro-org/kedro-plugins/blob/main/kedro-datasets/kedro_datasets/polars/eager_polars_dataset.py#L156
And here Polars checks if the file is already opened: https://github.com/pola-rs/polars/blob/py-0.20.13/py-polars/polars/io/parquet/functions.py#L156
Then Polars decides that it isn't opened and goes here: https://github.com/pola-rs/polars/blob/py-0.20.13/py-polars/polars/io/parquet/functions.py#L171
and it goes to .scan_parquet. And that only works with Paths or path strings (hence it doesn't find the path).
Either it is a Kedro bug or a Polars bug but I don't know which.
@astrojuanlu Apparently this has already been reported.
astrojuanlu
mark-druffel
noklam
Description
Trying to load a parquet w/ polars.EagerPolarsDataset with data catalog results in an error saying
no such file or directory:Context
I'm working on a new pipeline, bug is brand new to me. I'm trying to load parquets into my pipeline using polars. I realized lazy polars seems to work fine so this isn't preventing me from doing anything at the moment.
Steps to Reproduce
I reproduced the issue with the kedro starter pipeline in a conda environment.
kedro_new --name polars_issue --tools all --example ykedro runto populate data/.model_input_table@lazyPolars: type: polars.LazyPolarsDataset filepath: data/03_primary/model_input_table.parquet/*.parquet file_format: parquet
import polars as pl df_with_polars = pl.read_parquet(source = "data/03_primary/model_input_table.parquet/*.parquet") print(df_with_polars)
df_lazy = catalog.load("model_input_table@lazyPolars") print(df_lazy) print(df_lazy.collect())
df_eager = catalog.load("model_input_table@eagerPolars") print(df_eager)
df_eager = catalog.load("model_input_table@eagerPolars") print(df_eager)
02/29/24 21:21:22] INFO Loading data from model_input_table@eagerPolars data_catalog.py:483 (EagerPolarsDataset)...
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/kedro/io/core.py:193 in │ │ load │ │ │ │ 190 │ │ self._logger.debug("Loading %s", str(self)) │ │ 191 │ │ │ │ 192 │ │ try: │ │ ❱ 193 │ │ │ return self._load() │ │ 194 │ │ except DatasetError: │ │ 195 │ │ │ raise │ │ 196 │ │ except Exception as exc: │ │ │ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/kedro_datasets/polars/eag │ │ er_polars_dataset.py:156 in _load │ │ │ │ 153 │ │ │ │ " API" │ │ 154 │ │ │ │ " https://pola-rs.github.io/polars/py-polars/html/reference/io.html" │ │ 155 │ │ │ ) │ │ ❱ 156 │ │ with self._fs.open(load_path, self._fs_open_args_load) as fs_file: │ │ 157 │ │ │ return load_method(fs_file, self._load_args) │ │ 158 │ │ │ 159 │ def _save(self, data: pl.DataFrame) -> None: │ │ │ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/fsspec/spec.py:1295 in │ │ open │ │ │ │ 1292 │ │ │ ) │ │ 1293 │ │ else: │ │ 1294 │ │ │ ac = kwargs.pop("autocommit", not self._intrans) │ │ ❱ 1295 │ │ │ f = self._open( │ │ 1296 │ │ │ │ path, │ │ 1297 │ │ │ │ mode=mode, │ │ 1298 │ │ │ │ block_size=block_size, │ │ │ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/fsspec/implementations/lo │ │ cal.py:180 in _open │ │ │ │ 177 │ │ path = self._strip_protocol(path) │ │ 178 │ │ if self.auto_mkdir and "w" in mode: │ │ 179 │ │ │ self.makedirs(self._parent(path), exist_ok=True) │ │ ❱ 180 │ │ return LocalFileOpener(path, mode, fs=self, kwargs) │ │ 181 │ │ │ 182 │ def touch(self, path, truncate=True, kwargs): │ │ 183 │ │ path = self._strip_protocol(path) │ │ │ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/fsspec/implementations/lo │ │ cal.py:302 in init │ │ │ │ 299 │ │ self.autocommit = autocommit │ │ 300 │ │ self.compression = get_compression(path, compression) │ │ 301 │ │ self.blocksize = io.DEFAULT_BUFFER_SIZE │ │ ❱ 302 │ │ self._open() │ │ 303 │ │ │ 304 │ def _open(self): │ │ 305 │ │ if self.f is None or self.f.closed: │ │ │ │ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/fsspec/implementations/lo │ │ cal.py:307 in _open │ │ │ │ 304 │ def _open(self): │ │ 305 │ │ if self.f is None or self.f.closed: │ │ 306 │ │ │ if self.autocommit or "w" not in self.mode: │ │ ❱ 307 │ │ │ │ self.f = open(self.path, mode=self.mode) │ │ 308 │ │ │ │ if self.compression: │ │ 309 │ │ │ │ │ compress = compr[self.compression] │ │ 310 │ │ │ │ │ self.f = compress(self.f, mode=self.mode) │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ FileNotFoundError: [Errno 2] No such file or directory: '~/polars-bug/data/03_primary/model_input_table.parquet/*.parquet'
The above exception was the direct cause of the following exception:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ in:1 │
│ │
│ ❱ 1 df_eager = catalog.load("model_input_table@eagerPolars") │
│ 2 print(df_eager) │
│ 3 │
│ │
│ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/kedro/io/data_catalog.py: │
│ 490 in load │
│ │
│ 487 │ │ │ extra={"markup": True}, │
│ 488 │ │ ) │
│ 489 │ │ │
│ ❱ 490 │ │ result = dataset.load() │
│ 491 │ │ │
│ 492 │ │ return result │
│ 493 │
│ │
│ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/kedro/io/core.py:615 in │
│ load │
│ │
│ 612 │ │ return self._filepath / version / self._filepath.name │
│ 613 │ │
│ 614 │ def load(self) -> _DO: │
│ ❱ 615 │ │ return super().load() │
│ 616 │ │
│ 617 │ def save(self, data: _DI) -> None: │
│ 618 │ │ self._version_cache.clear() │
│ │
│ ~/miniconda3/envs/polars_bug/lib/python3.10/site-packages/kedro/io/core.py:202 in │
│ load │
│ │
│ 199 │ │ │ message = ( │
│ 200 │ │ │ │ f"Failed while loading data from data set {str(self)}.\n{str(exc)}" │
│ 201 │ │ │ ) │
│ ❱ 202 │ │ │ raise DatasetError(message) from exc │
│ 203 │ │
│ 204 │ def save(self, data: _DI) -> None: │
│ 205 │ │ """Saves data by delegation to the provided save method. │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DatasetError: Failed while loading data from data set EagerPolarsDataset(file_format=parquet,
filepath=~/polars-bug/data/03_primary/model_input_table.parquet/.parquet, load_args={},
protocol=file, save_args={}).
[Errno 2] No such file or directory:
'~/polars-bug/data/03_primary/model_input_table.parquet/.parquet'