[KED-2540] Why are old versions pinned in requirements.txt?

[FlorianGD]

Hello,

First of all, thank you for the great tool, I discovered that recently and I really like it so far.

Introduction

In the requirements.txt file that ships when doing kedro new, some packages' versions are pinned, and sometimes to rather old versions. For example, jupyterlab==0.31.1 whereas the latest available version is 3.0.12

Background

I want to use Kedro with the latest version of jupyter lab, especially because the UI to add a tag to a cell is much better.

Problem

My question is: why are the versions pinned, and can I update them or is it going to break Kedro internals?

Below, I took the requirements.txt in the template. I added comments with the latest versions in PyPI if they are excluded by the requirements.

black==v19.10b0  # latest is 20.8b1
flake8>=3.7.9, <4.0
ipython==7.10  # latest is 7.21.0
isort>=4.3.21, <5.0  # latest 5.8.0
jupyter~=1.0
jupyter_client>=5.1, <7.0
jupyterlab==0.31.1  # latest is 3.0.12
kedro=={{ cookiecutter.kedro_version }}
nbstripout==0.3.3  # latest is 0.3.9
pytest-cov~=2.5 
pytest-mock>=1.7.1, <2.0  # latest is 3.5.1
pytest~=6.1.2
wheel==0.32.2  # latest is 0.36.2

[Minyus]

I'd like to use new versions of Python packages too.

[datajoely]

Hi @FlorianGD - this is a great point and I didn't realise lab in particular was so far behind I will add this to our backlog.

If you are okay upgrading Jupyter Lab manually it would be great to know if you spot any issues?

[WaylonWalker]

I've had a lot of grief with this ipython version and always update to latest, In particular there is a conflict with the latest jedi version that cause it to error or even crash when hitting tab.

[Minyus]

For the Kedro's requirements.txt issue, the workaround is --no-deps option.
You can install Kedro without dependency after installing the desired versions of dependencies.

pip install anyconfig>=0.10.0 cachetools>=4.1 click cookiecutter>=1.7.0 dynaconf>=3.1.2 fsspec>=0.5.1 gitpython>=3.0 jmespath>=0.9.5 jupyter_client>=5.1 pip-tools>=5.0 pluggy>=0.13.0 python-json-logger>=0.1.9 PyYAML>=4.2 setuptools>=38.0 toml>=0.10 toposort>=1.5
pip install --no-deps kedro

Hope we won't need this workaround soon.

[FlorianGD]

Hi @FlorianGD - this is a great point and I didn't realise lab in particular was so far behind I will add this to our backlog.

If you are okay upgrading Jupyter Lab manually it would be great to know if you spot any issues?

OK, I will. Thanks

[datajoely]

@WaylonWalker do you experience any issues working with the latest iPython version?

[WaylonWalker]

Looks like I need to checkout the latest (released last Friday).

I have had no issues up through 7.21.0, I live in tmux split between vim and ipython for much of my day.

I have had many issues trying to mix the pinned 7.10.0 with newer versions of jedi. Based on this experience, if you pin ipython you should also pin jedi with it.

[FlorianGD]

I have not tested all the features extensively, but the following requirements.txt seems to work. There is an issue with ipython that produces a warning (see this issue), but I prefer this over ipython crashing on <TAB> as @WaylonWalker pointed out:

black==v20.8b1
flake8>=3.7.9, <4.0
ipython~=7.21
isort>~=5.8
jupyter~=1.0
jupyter_client>=5.1, <7.0
jupyterlab~=3.0
kedro==0.17.2
nbstripout~=0.3.9
pytest-cov~=2.5
pytest-mock~=3.5
pytest~=6.1.2
wheel~=0.36.2

As for the warning in jupyter notebooks, the below snippet gets rid of it

from warnings import filterwarnings

filterwarnings("ignore", ".*`should_run_async`.*")

[datajoely]

Thank you @WaylonWalker and @FlorianGD I will communicate this back to the team

[ignacioparicio]

Thank you @FlorianGD for raising this point! We will upgrade the pinned dependencies in the 0.17.4 release. The table below summarizes the main changes we expect:

0.17.3	0.17.4
black==v19.10b0	black==21.5b1
isort>=4.3.21, <5.0	isort~=5.0
jupyterlab==0.31.1	jupyterlab~=3.0
nbstripout==0.3.3	nbstripout~=0.4
pytest~=6.1.2	pytest~=6.2
wheel==0.32.2	wheel>=0.35, <0.37

We are keeping ipython==7.10 for now to avoid the annoying notebook warnings. We will update this one as well once ipykernel master branch becomes 6.0 and is released.

I will close this ticket for now, but please feel free to reach out if you feel we should still discuss this.