Spark dataframe - enable support for "s3://" protocol - Githubissues

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.

Apache License 2.0

91 stars 84 forks source link

Spark dataframe - enable support for "s3://" protocol #736

Open abocaz-elb-alcm opened 3 months ago

abocaz-elb-alcm commented 3 months ago

https://github.com/kedro-org/kedro-plugins/blob/105fcf217603749516961b00229795f6804d5f7e/kedro-datasets/kedro_datasets/spark/spark_dataset.py#L327

Basically only add fs_prefix in ("s3://", "s3a://")

this should do :D