[FEATURE]: re-open 502 decouple python version for zenml core from integration dependencies

[strangemonad]

Contact Details [Optional]

No response

Describe the feature you'd like

This is a dupe of #502 (but I don't have permission to re-open it). I've left a comment there as well. I think it would be meaningful to decouple the python version required by zenml core from the zenml integration installed integrations. As the list of integrations grows, there will likely be many integrations that any one deployment of zenml doesn't use and there will likely be some integrations that only support specific python versions (e.g. when will tensorflow support pythhon 3.10, 3.11...)

Is your feature request related to a problem?

I don't want to be tied to a version of python required by an integration I don't plan on using. I don't think "wait for tensor flow to support 3.9" is a great answer because there will likely always be some integration that's lagging. I should only need to meet the requirements of zenml core + the integrations I plan on using for a project.

How do you solve your current problem with the current status-quo of ZenML?

I don't use zenml. My codebase requires python 3.9 (and very soon plan to adopt 3.11 only features). Reverting our code to support python 3.8 or waiting for tensorflow (or all other integrations) to support newer dependencies

Any other comments?

No response

[sotte]

I was just about to open a ticket in a similar vein. We also have a 3.9 codebase and I don't care about TF ;)

The problem with #502 is that tfx is a essential dependency of zenml AFAIK and that's why zenml is still <3.9.

[htahir1]

Yes indeed, @sotte. ml-pipelines-sdk (the really base version of tfx) is a core dependency @strangemonad. It is not a matter of zenml integration install.

The features ZenML is using of ml-pipelines-sdk are replaceable however, and that is the eventual goal of the core team in the next quarter or two. However, we are currently internally focusing on cooler features that add value on top rather than the internals. The only blowback of that is the limitations on Python versions. I would say the tfx team is working hard to get 3.9 out and 3.10 and 3.11 should not be far off.

In the meantimes, my estimation for you both is that the next 2-4 weeks the release of ZenML will support 3.9 (maybe even 3.9+ depending on what tfx releases soon)

[htahir1]

Let me know if there is more discussion to be had here, otherwise ill close the issue @strangemonad

[htahir1]

The 0.8.1 release now supports Python 3.9!

[strangemonad]

thanks @htahir1. Slowly getting familiar with the zenml codebase. Didn't realize how much of it was currently dependent on TFX / MLMD. Makes total sense.