Explicitly install `nose` to unbreak tests

[mbrukman]

The command nosetests appears to have disappeared from the GitHub Actions images for Python 3.8 and 3.9 on all versions of Ubuntu and macOS, so let's try installing it manually to fix the tests in the short-term.

Closes https://github.com/rossant/ipycache/issues/63

[mbrukman]

Strange, now we're getting yet a different error on both Python 3.8 and 3.9 (3.7 seems to be fine):

Run nosetests
E
======================================================================
ERROR: Failure: ModuleNotFoundError (No module named 'IPython.config')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/runner/work/ipycache/ipycache/test_ipycache.py", line 20, in <module>
    from ipycache import (save_vars, load_vars, clean_var, clean_vars, do_save,
  File "/home/runner/work/ipycache/ipycache/ipycache.py", line 14, in <module>
    from IPython.config.configurable import Configurable
ModuleNotFoundError: No module named 'IPython.config'

----------------------------------------------------------------------
Ran 1 test in 0.952s

FAILED (errors=1)
Error: Process completed with exit code 1.

[mbrukman]

Looks like this is an issue with recently-released IPython 8.0.0 (https://pypi.org/project/ipython/#history shows it was released on Jan 12, 2022) — since we specify ipython[all] in requirements.txt, we're getting the latest-available and compatible IPython library.

Looks like IPython 8.0.0 is not compatible with Python 3.7, so there we get an earlier version:

Run pip install -r requirements.txt
Collecting ipython[all]
  Downloading ipython-7.31.0-py3-none-any.whl (792 kB)
[...]

whereas for Python 3.8 and 3.9, we're getting IPython 8.0:

Run pip install -r requirements.txt
Collecting ipython[all]
  Downloading ipython-8.0.0-py3-none-any.whl (747 kB)
[...]

Thus, the solution here is to update requirements.txt and explicitly specify that ipycache is compatible with IPython prior to 8.0.0.

[mbrukman]

The new error is:

Run nosetests
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated since IPython 4.0. You should import from traitlets.config instead.
  "You should import from traitlets.config instead.", ShimWarning)
/home/runner/work/ipycache/ipycache/ipycache.py:17: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import Unicode
..........
----------------------------------------------------------------------
Ran 10 tests in 0.617s

OK
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.1.5dev' to '0.1.5.dev0'
running check
  warnings.warn(tmpl.format(**locals()))
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/nbformat.py:13: ShimWarning: The `IPython.nbformat` package has been deprecated since IPython 4.0. You should import from nbformat instead.
  "You should import from nbformat instead.", ShimWarning)
Traceback (most recent call last):
  File "ipynb_runner.py", line 5, in <module>
    from IPython.nbformat.current import read
ModuleNotFoundError: No module named 'IPython.nbformat.current'
Error: Process completed with exit code 1.

An earlier test run included the following helpful guidance:

- use nbformat for read/write/validate public API
- use nbformat.vX directly to composing notebooks of a particular version

and https://nbformat.readthedocs.io/en/latest/api.html shows that there should be a nbformat.read() method, so I think we can update ipynb_runner.py as follows:

-from IPython.nbformat.current import read
+from IPython.nbformat import read

to address this issue.

[mbrukman]

Nope, that didn't work either:

Run nosetests
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated since IPython 4.0. You should import from traitlets.config instead.
  "You should import from traitlets.config instead.", ShimWarning)
/home/runner/work/ipycache/ipycache/ipycache.py:17: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import Unicode
..........
----------------------------------------------------------------------
Ran 10 tests in 0.790s

OK
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.1.5dev' to '0.1.5.dev0'
running check
  warnings.warn(tmpl.format(**locals()))
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/nbformat.py:13: ShimWarning: The `IPython.nbformat` package has been deprecated since IPython 4.0. You should import from nbformat instead.
  "You should import from nbformat instead.", ShimWarning)
Traceback (most recent call last):
  File "ipynb_runner.py", line 5, in <module>
    from IPython.nbformat import read
ModuleNotFoundError: No module named 'nbformat'
Error: Process completed with exit code 1.

Next approach to try is:

-from IPython.nbformat import read
+from nbformat import read

and manually install nbformat in test_requirements.txt to make sure it's present in the environment.

[mbrukman]

We're making progress, but it still doesn't work:

Run nosetests
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated since IPython 4.0. You should import from traitlets.config instead.
  "You should import from traitlets.config instead.", ShimWarning)
/home/runner/work/ipycache/ipycache/ipycache.py:17: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import Unicode
..........
----------------------------------------------------------------------
Ran 10 tests in 0.610s

OK
running check
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.1.5dev' to '0.1.5.dev0'
  warnings.warn(tmpl.format(**locals()))
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
Traceback (most recent call last):
  File "ipynb_runner.py", line 6, in <module>
    from IPython.kernel import KernelManager
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/kernel/__init__.py", line 31, in <module>
    from ipykernel import comm, inprocess
ModuleNotFoundError: No module named 'ipykernel'
Error: Process completed with exit code 1.

We will next attempt the following change:

-from IPython.kernel import KernelManager
+from jupyter_client.manager import KernelManager

and install jupyter-client.

[mbrukman]

Looks like the new problem: is:

Run nosetests
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated since IPython 4.0. You should import from traitlets.config instead.
  "You should import from traitlets.config instead.", ShimWarning)
/home/runner/work/ipycache/ipycache/ipycache.py:17: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import Unicode
..........
----------------------------------------------------------------------
Ran 10 tests in 1.003s

OK
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.1.5dev' to '0.1.5.dev0'
running check
  warnings.warn(tmpl.format(**locals()))
Checking: examples/example.ipynb
Traceback (most recent call last):
  File "ipynb_runner.py", line 41, in <module>
    nb = read(open(notebook), 'json')
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nbformat/__init__.py", line 143, in read
    return reads(buf, as_version, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nbformat/__init__.py", line 75, in reads
    nb = convert(nb, as_version)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nbformat/converter.py", line 54, in convert
    "version doesn't exist" % (to_version))
TypeError: %d format: a number is required, not str
Error: Process completed with exit code 1.

Specifically, the following line of code is problematic with this version of nbformat:

nb = read(open(notebook), 'json')

because nbformat.read() takes as a second parameter the notebook version (int) rather than a string such as 'json'.

Let's see the versions of our notebooks:

$ grep nbformat examples/*
examples/capture_output.ipynb: "nbformat": 3,
examples/capture_output.ipynb: "nbformat_minor": 0,
examples/example.ipynb: "nbformat": 3,
examples/example.ipynb: "nbformat_minor": 0,
examples/example_outputs.ipynb: "nbformat": 3,
examples/example_outputs.ipynb: "nbformat_minor": 0,

Looks like they're all version 3, so we can make the following change:

-nb = read(open(notebook), 'json')
+nb = read(open(notebook), as_version=3)

[mbrukman]

Failing again. Python 2.7 tests failed with:

ERROR: Could not find a version that satisfies the requirement nbformat~=5.1.3 (from -r test_requirements.txt (line 3)) (from versions: 4.0.0, 4.0.1, 4.1.0, 4.2.0, 4.3.0, 4.4.0)
ERROR: No matching distribution found for nbformat~=5.1.3 (from -r test_requirements.txt (line 3))
Error: Process completed with exit code 1.

So we could go with 4.4.0 (which may resolve other issues as well).

Python 3.7 failed with:

Run nosetests
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated since IPython 4.0. You should import from traitlets.config instead.
  "You should import from traitlets.config instead.", ShimWarning)
/home/runner/work/ipycache/ipycache/ipycache.py:17: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  from IPython.utils.traitlets import Unicode
..........
----------------------------------------------------------------------
Ran 10 tests in 0.809s

OK
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.1.5dev' to '0.1.5.dev0'
running check
  warnings.warn(tmpl.format(**locals()))
Checking: examples/example.ipynb
Traceback (most recent call last):
  File "ipynb_runner.py", line 45, in <module>
    km.start_kernel()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/asyncio/futures.py", line 181, in result
    raise self._exception
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/asyncio/tasks.py", line 249, in __step
    result = coro.send(None)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/manager.py", line 79, in wrapper
    raise e
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/manager.py", line 71, in wrapper
    out = await method(self, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/manager.py", line 376, in _async_start_kernel
    kernel_cmd, kw = await ensure_async(self.pre_start_kernel(**kw))
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/asyncio/futures.py", line 181, in result
    raise self._exception
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/asyncio/tasks.py", line 249, in __step
    result = coro.send(None)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/manager.py", line 339, in _async_pre_start_kernel
    self.kernel_spec,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/manager.py", line 169, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 292, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3
Error: Process completed with exit code 1.

which is curious, as we don't specify python3 anywhere, so looks like that's the default.

Per this SO answer, it seems that we need to manually install the ipykernel module, which makes sense, so let's try that next. We'll also run jupyter kernelspec list before running the notebook tests to see what's actually installed, and split up the tests into separate phases to get shorter error messages per phase.

[mbrukman]

We're making some progress! Tests now pass on Python 3.7 and 3.8.

Python 2.7 fails with:

ERROR: Could not find a version that satisfies the requirement jupyter-client~=7.1.1 (from -r test_requirements.txt (line 5)) (from versions: 4.0.0, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.2.2, 4.3.0, 4.4.0, 5.0.0, 5.0.1, 5.1.0, 5.2.0, 5.2.1, 5.2.2, 5.2.3, 5.2.4, 5.3.0, 5.3.1, 5.3.2, 5.3.3, 5.3.4, 5.3.5)
ERROR: No matching distribution found for jupyter-client~=7.1.1 (from -r test_requirements.txt (line 5))

We could use jupyter-client ~= 5.3.5 for now, until we drop support for Python 2.

Python 3.9 fails with:

Run jupyter kernelspec list
Available kernels:
  python3    /opt/hostedtoolcache/Python/3.9.9/x64/share/jupyter/kernels/python3
Traceback (most recent call last):
  File "/home/runner/work/ipycache/ipycache/ipynb_runner.py", line 5, in <module>
    from nbformat import read
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/__init__.py", line 14, in <module>
    from . import v1
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/v1/__init__.py", line 19, in <module>
    from .nbjson import reads as reads_json, writes as writes_json
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/v1/nbjson.py", line 19, in <module>
    from base64 import encodestring
ImportError: cannot import name 'encodestring' from 'base64' (/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/base64.py)
Error: Process completed with exit code 1.

Which seems to be a failure purely due to jupyter kernelspec list, which we don't need anymore, now that we know ipykernel provided us with a working python3 kernel, so let's just drop it and see if that fixes things for us.

[mbrukman]

Python 2.7 now fails with:

ERROR: Could not find a version that satisfies the requirement ipykernel~=6.7.0 (from -r test_requirements.txt (line 6)) (from versions: 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.2.2, 4.3.0, 4.3.1, 4.4.0, 4.4.1, 4.5.0, 4.5.1, 4.5.2, 4.6.0, 4.6.1, 4.7.0, 4.8.0, 4.8.1, 4.8.2, 4.9.0, 4.10.0, 4.10.1)
ERROR: No matching distribution found for ipykernel~=6.7.0 (from -r test_requirements.txt (line 6))

so let's try ipykernel ~= 4.10.1.

Python 3.9 is failing with:

Run python ipynb_runner.py -v -s examples/example.ipynb
Traceback (most recent call last):
  File "/home/runner/work/ipycache/ipycache/ipynb_runner.py", line 5, in <module>
    from nbformat import read
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/__init__.py", line 14, in <module>
    from . import v1
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/v1/__init__.py", line 19, in <module>
    from .nbjson import reads as reads_json, writes as writes_json
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/site-packages/nbformat/v1/nbjson.py", line 19, in <module>
    from base64 import encodestring
ImportError: cannot import name 'encodestring' from 'base64' (/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/base64.py)
Error: Process completed with exit code 1.

which is the same error we saw previously, so it's not due to jupyter kernelspec list.

Per https://github.com/jupyter/nbformat/issues/215, this is due to our use of nbformat ~= 4.4.0; we need to use nbformat 5.1.3 for Python 3.9 support, but that means we won't be able to support Python 2.7. It turns out that the reason our tests were passing before is because we disabled Python 2.7 in https://github.com/rossant/ipycache/pull/55 when we fixed the CI the first time (after it hasn't run for quite some time).

Thus, we can either drop support for Python 2.7 immediately or we can create a separate requirements.txt such that we can run Python 2.7 tests in a separate environment with different versions of dependencies, until such time that we delete support for Python 2.7 entirely.

Let's try creating a separate test_requirements_python27.txt with downgraded dependencies, and upgrade our dependency versions in test_requirements.txt for the latest versions to support the newest Python 3 versions.

[mbrukman]

Looks like Python 3.7, 3.8, and 3.9 are passing! But both Python 2.7 and 3.x are being installed in all cases, so Python 2.7 is failing. We'll need to fix the YAML syntax of GitHub Actions if statement to make them mutually exclusive.

[mbrukman]

Success!! 🎉

All tests finally pass across the full matrix of Python and OS versions.