search

elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.83k stars 343 forks source link

Lastest version of the system-owned pandas runtime image needs to build 'pyzmq' on demand #2585

Open ptitzler opened 2 years ago

ptitzler commented 2 years ago

Describe the issue The latest version of the Pandas runtime image can no longer access the pre-built pyzmq dependency that is required by Elyra to run Jupyter notebooks in KFP/AA clusters. As a result this package is built on-demand, resulting in a significant overhead whenever a notebook is processed.

To Reproduce Steps to reproduce the behavior:

  1. Create a dummy notebook
  2. Create a pipeline using this notebook and configure the node to use the Pandas 1.4.1 runtime image
  3. Run the pipeline on KFP or AA
  4. Inspect log
Building wheels for collected packages: pyzmq, tornado
  Building wheel for pyzmq (setup.py): started
  Building wheel for pyzmq (setup.py): still running...
  Building wheel for pyzmq (setup.py): still running...
  Building wheel for pyzmq (setup.py): finished with status 'done'
  Created wheel for pyzmq: filename=pyzmq-19.0.1-cp39-cp39-linux_x86_64.whl size=6964149 sha256=630b00ba92e15833a6dc777a8368711dbdb4f08158edec51fa5773759c088bf6

Expected behavior A clear and concise description of what you expected to happen.

Deployment information Describe what you've deployed and how:

kevin-bates commented 1 year ago

@ptitzler - I suspect this is the same issue as we found for older versions of pyzmq on Python 3.11. When the pandas 1.4.1 image was introduced, our requirements file referenced pyzmq == 19.0.1, but that image uses Python 3.9.10 which, at that time, would require a build since it wasn't "available" in wheel form in Python 3.9.