spark_dataset: issue with functools partial - Githubissues

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.

Apache License 2.0

91 stars 84 forks source link

spark_dataset: issue with functools partial #737

Closed abocaz-elb-alcm closed 3 months ago

abocaz-elb-alcm commented 3 months ago

https://github.com/kedro-org/kedro-plugins/blob/105fcf217603749516961b00229795f6804d5f7e/kedro-datasets/kedro_datasets/spark/spark_dataset.py#L331

Using kedro==0.19.3

when calling versioned spark dataframes, I get an error in https://github.com/kedro-org/kedro/blob/ff133a5572903757b2cfe613d9b6fc848feca575/kedro/io/core.py#L553

Exception: S3FileSystem._find() got an unexpected keyword argument 'refresh'

I'm using fsspec==2023.12.2 and s3fs==2023.12.2

abocaz-elb-alcm commented 3 months ago

updating fsspec and s3fs fixed it