[FEATURE]: No support for python 3.9 preventing deployment of zenml pipelines

[aimlnerd]

Contact Details [Optional]

No response

Describe the feature you'd like

Production/UAT/DEV systems are based on python 3.9 and cannot be changed. Developed zenml pipelines based on 0.6.2 on python 3.6 in local environment.

Now unable to deploy the zenml based model pipeline in production because zenml cannot be installed on python 3.9.

• When is support for python 3.9 expected to be released?
• Is there a work around till then to install zenml on python 3.9? we use docker for deploying python 3.9

Is your feature request related to a problem?

No response

How do you solve your current problem with the current status-quo of ZenML?

No response

Any other comments?

No response

[AlexejPenner]

Hi @deepakiim - as I understand it the only thing stopping us from offering python 3.9 support is our dependency on tfx. Once tfx offers python 3.9 support we will evaluate an upgrade from our side I would assume.

TFX seems to be close though with an approximate time of 1 month from now (see issue 4749 on the tfx repo)

[AlexejPenner]

Regarding a workaround

Not sure how your systems are setup and what your limitations are. Here are some initial thoughts,

Are you able to use docker-containers with python 3.8 installed? Alternatively would you be able to run inside a virtual environment with python 3.8?

Pyenv could be of help in that case to allow for multiple python versions within a system https://realpython.com/intro-to-pyenv/

If you have some more information on the limitations of your systems we can have a deeper look at potential workarounds

[strangemonad]

One suggestion, as the list of supported zenml integrations grows, the problem of version compatibilities will only get worse. Would it make sense to package some or all integrations separately? e.g. zenml vs zenml[tfx] or zenml[all-integrations]`?

[htahir1]

@strangemonad Its a good suggestion but we actually went through many iterations on managing these integrations and settled with the approach with zenml integration install. We do not package any integration currently (if you see the pyproject.toml you'll see the core dependencies). zenml integration install is a CLI utility we added to pip install other dependencies. This way we have more control over which versions of these are installed and supported outside of doing it via extras as you suggested.

If we did use the extras (i.e. doing zenml[kubeflow] , then we lose a bit of control over dependency management). But there might be a point where do consider going back if there are considerations today that we not making. Would be keen to get your thoughts

[aimlnerd]

In this case I don't need tfx related integrations of zenml but the core functionalities on python 3.9. But even then it cannot use zenml due to no support for 3.9 yet.

[htahir1]

@deepakiim Yes, I would say we can have Python 3.9 in about a month or so. Before that would you try the workaround @AlexejPenner has suggested?

[aimlnerd]

Thanks @AlexejPenner for th suggestions. It wont work for us. We decide to change the microservice to python 3.8 temporarily. So that should solve it.

[htahir1]

@deepakiim Then ill close this issue now :+1: Feel free to re-open it if needed

[strangemonad]

@htahir1 thanks for the response. It makes complete sense why you settled on zenml integration install. I'd like to echo what @deepakiim replied with. The workaround doesn't really work well for us either. More generally, what happens when I want python 3.10 or 3.11 and tfx or some other integration is still lagging.

I think what I'm suggesting is that the core zenml only be limited to the lowest python version of the integrations I actually need to use. If I'm using tfx then I'm already locked to the python version they support but if I don't I don't want to be held back and I wish the core of zenml wouldn't directly impose this restriction especially since integrations can be installed piece-meal for each project.

[htahir1]

@strangemonad The 0.8.1 release now supports Python 3.9!