Open raajaaDheeRaajaa opened 9 months ago
Hello, I can take up this issue and test brickflow with the updated python versions.
Hey team, intentially we are supporting 3.8 - 3.10 as those are the system versions supported by databricks runtimes. 14.3 LTS beta is provided with Python: 3.10.12 https://docs.databricks.com/en/release-notes/runtime/14.3lts.html#system-environment
The risk of supporting higher versions is it may encourage users to use newer 3.11, 3.12 and 3.13 features which may not work on 3.10. The user will not run into this without, actually running their workflow.
I am not opposed to this change, as the library should work in higher versions (since new versions of python contain all the previous features) but I do not know of a good way to indicate to the user to not use 3.11+ features. Let me know if you have any ideas around that.
Is your feature request related to a problem? Please describe. Currently, the compatible versions of python for brickflow are pinned to 3.8, 3.9 & 3.10 https://github.com/Nike-Inc/brickflow/blob/09a2af92a2d9730189ea71277b59e52878caaee6/pyproject.toml#L37
Requesting to add support for python versions 3.11 & 3.12, for users using pyspark version >=3.5
Cloud Information
Describe the solution you'd like A clear and concise description of what you want to happen.
Describe alternatives you've considered Updating the acceptable python version from
>=3.8,<3.11
to>=3.8,<=3.12
here: https://github.com/Nike-Inc/brickflow/blob/09a2af92a2d9730189ea71277b59e52878caaee6/pyproject.toml#L37Additional context This change will allow users running newer version of python with new features and bug fixes, able to use brickflow.