Remove `setuptools` from `spaceflights-pyspark-viz` starter on `pyspark` release - Githubissues

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

https://kedro.org

Apache License 2.0

9.88k stars 893 forks source link

Remove `setuptools` from `spaceflights-pyspark-viz` starter on `pyspark` release #3836

Open ankatiyar opened 5 months ago

ankatiyar commented 5 months ago

Description

While adding python 3.12 support to kedro-starters https://github.com/kedro-org/kedro-starters/pull/219, this issue came up. Full discussion in the PR.

Python 3.12 got rid of distutils https://docs.python.org/3.12/whatsnew/3.12.html#distutils
pyspark uses distutils in their code which they have fixed but will be released in 4.0.0
Temp fix - add setuptools to requirements.txt in spaceflights-pyspark-viz starter for only `python_version>=3.12" https://github.com/kedro-org/kedro-starters/pull/220

When pyspark releases their 4.0.0 version

Remove kedro-datasets pyspark pin which is currently <4.0.0 I believe
Remove setuptools from the requirements.txt