search

arogozhnikov / einops

Flexible and powerful tensor operations for readable and reliable code (for pytorch, jax, TF and others)
https://einops.rocks
MIT License
8.48k stars 352 forks source link

Tests failing on FreeBSD #313

Closed yurivict closed 6 months ago

yurivict commented 7 months ago

Describe the bug PyTest prints these failures:

========================================================================================= FAILURES ==========================================================================================
______________________________________________________________________________________ test_notebook_1 ______________________________________________________________________________________

    def test_notebook_1():
        [notebook] = Path(__file__).parent.with_name("docs").glob("1-*.ipynb")
>       render_notebook(notebook, replacements={})

tests/test_notebooks.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x13a48dbee3a0>
cell = {'cell_type': 'code', 'execution_count': 5, 'metadata': {'pycharm': {'name': '#%%\n'}, 'execution': {'iopub.status.bus...: No module named 'einops'"]}], 'source': "# we'll use three operations\nfrom einops import rearrange, reduce, repeat"}
cell_index = 7
exec_reply = {'buffers': [], 'content': {'ename': 'ModuleNotFoundError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'c866a4eb-...e, 'engine': 'c866a4eb-3178-416f-ae4c-a466cecc1ff8', 'started': '2024-04-03T16:25:21.534254Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           # we'll use three operations
E           from einops import rearrange, reduce, repeat
E           ------------------
E           
E           
E           ---------------------------------------------------------------------------
E           ModuleNotFoundError                       Traceback (most recent call last)
E           Cell In[5], line 2
E                 1 # we'll use three operations
E           ----> 2 from einops import rearrange, reduce, repeat
E           
E           ModuleNotFoundError: No module named 'einops'

/usr/local/lib/python3.9/site-packages/nbclient/client.py:918: CellExecutionError
______________________________________________________________________________________ test_notebook_3 ______________________________________________________________________________________

    def test_notebook_3():
        [notebook] = Path(__file__).parent.with_name("docs").glob("3-*.ipynb")
        if not is_backend_tested("torch"):
            pytest.skip()
>       render_notebook(notebook, replacements={})

tests/test_notebooks.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x13a48ff801c0>
cell = {'cell_type': 'code', 'execution_count': 1, 'id': '5f24a8f2-6680-4298-9736-081719c53f88', 'metadata': {'execution': {'...31mModuleNotFoundError\x1b[0m: No module named 'einops'"]}], 'source': 'from einops.layers.torch import EinMix as Mix'}
cell_index = 2
exec_reply = {'buffers': [], 'content': {'ename': 'ModuleNotFoundError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'c70533f8-...e, 'engine': 'c70533f8-d151-4fc1-a533-b3630f329c72', 'started': '2024-04-03T16:25:24.742754Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           from einops.layers.torch import EinMix as Mix
E           ------------------
E           
E           
E           ---------------------------------------------------------------------------
E           ModuleNotFoundError                       Traceback (most recent call last)
E           Cell In[1], line 1
E           ----> 1 from einops.layers.torch import EinMix as Mix
E           
E           ModuleNotFoundError: No module named 'einops'

/usr/local/lib/python3.9/site-packages/nbclient/client.py:918: CellExecutionError
______________________________________________________________________________________ test_notebook_4 ______________________________________________________________________________________

    def test_notebook_4():
        [notebook] = Path(__file__).parent.with_name("docs").glob("4-*.ipynb")
        if not is_backend_tested("torch"):
            pytest.skip()
>       render_notebook(notebook, replacements={})

tests/test_notebooks.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x13a48e477730>
cell = {'cell_type': 'code', 'execution_count': 2, 'metadata': {'collapsed': False, 'jupyter': {'outputs_hidden': False}, 'ex...e_depth = np.random.random([h, w])\n# but we can stack them\nimage_rgbd, ps = pack([image_rgb, image_depth], 'h w *')"}
cell_index = 3
exec_reply = {'buffers': [], 'content': {'ename': 'ModuleNotFoundError', 'engine_info': {'engine_id': -1, 'engine_uuid': '6e46e1d8-...e, 'engine': '6e46e1d8-b8f4-4af2-ad9d-6d31a6f98e2f', 'started': '2024-04-03T16:25:27.923862Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           from einops import pack, unpack
E           
E           h, w = 100, 200
E           # image_rgb is 3-dimensional (h, w, 3) and depth is 2-dimensional (h, w)
E           image_rgb = np.random.random([h, w, 3])
E           image_depth = np.random.random([h, w])
E           # but we can stack them
E           image_rgbd, ps = pack([image_rgb, image_depth], 'h w *')
E           ------------------
E           
E           
E           ---------------------------------------------------------------------------
E           ModuleNotFoundError                       Traceback (most recent call last)
E           Cell In[2], line 1
E           ----> 1 from einops import pack, unpack
E                 3 h, w = 100, 200
E                 4 # image_rgb is 3-dimensional (h, w, 3) and depth is 2-dimensional (h, w)
E           
E           ModuleNotFoundError: No module named 'einops'

/usr/local/lib/python3.9/site-packages/nbclient/client.py:918: CellExecutionError
_____________________________________________________________________________ test_notebook_2_with_all_backends _____________________________________________________________________________

    def test_notebook_2_with_all_backends():
        [notebook] = Path(__file__).parent.with_name("docs").glob("2-*.ipynb")
        backends = []
        if is_backend_tested("torch"):
            # notebook uses name pytorch
            backends.append("pytorch")
        if is_backend_tested("tensorflow"):
            backends.append("tensorflow")
        if is_backend_tested("chainer"):
            backends.append("chainer")

        if len(backends) == 0:
            pytest.skip()

        for backend in backends:
            print("Testing {} with backend {}".format(notebook, backend))
            replacements = {"flavour = 'pytorch'": "flavour = '{}'".format(backend)}
            expected_string = "selected {} backend".format(backend)
>           result = render_notebook(notebook, replacements=replacements)

tests/test_notebooks.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x13a48ff87bb0>
cell = {'cell_type': 'code', 'execution_count': 1, 'metadata': {'execution': {'iopub.status.busy': '2024-04-03T16:25:30.67020... "\x1b[0;31mModuleNotFoundError\x1b[0m: No module named 'einops'"]}], 'source': 'from einops import rearrange, reduce'}
cell_index = 1
exec_reply = {'buffers': [], 'content': {'ename': 'ModuleNotFoundError', 'engine_info': {'engine_id': -1, 'engine_uuid': '22768edd-...e, 'engine': '22768edd-8f6f-4a5b-906e-3ecd7ff2758a', 'started': '2024-04-03T16:25:30.671056Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None

        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None

        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           from einops import rearrange, reduce
E           ------------------
E           
E           
E           ---------------------------------------------------------------------------
E           ModuleNotFoundError                       Traceback (most recent call last)
E           Cell In[1], line 1
E           ----> 1 from einops import rearrange, reduce
E           
E           ModuleNotFoundError: No module named 'einops'

/usr/local/lib/python3.9/site-packages/nbclient/client.py:918: CellExecutionError
----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------
Testing /usr/ports/misc/py-einops/work-py39/einops-0.7.0/docs/2-einops-for-deep-learning.ipynb with backend pytorch
_____________________________________________________________________________________ test_torch_layer ______________________________________________________________________________________

    def test_torch_layer():
        if not is_backend_tested("torch"):
            pytest.skip()
        else:
            # checked that torch present
            import torch
            import torch.jit

            model1 = create_torch_model(use_reduce=True)
            model2 = create_torch_model(use_reduce=False)
            input = torch.randn([10, 3, 32, 32])
            # random models have different predictions
            assert not torch.allclose(model1(input), model2(input))
            model2.load_state_dict(pickle.loads(pickle.dumps(model1.state_dict())))
            assert torch.allclose(model1(input), model2(input))

            # tracing (freezing)
>           model3 = torch.jit.trace(model2, example_inputs=input)

tests/test_layers.py:219: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:489: in _fn
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/_dynamo/external_utils.py:17: in inner
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:806: in trace
    return trace_module(
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:1102: in trace_module
    _check_trace(
/usr/local/lib/python3.9/site-packages/torch/utils/_contextlib.py:115: in decorate_context
    return func(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:575: in _check_trace
    diag_info = graph_diagnostic_info()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def graph_diagnostic_info():
        mod_canonicalized = torch._C._jit_pass_canonicalize(traced_func.graph)
        torch._C._jit_pass_inline(mod_canonicalized)
        torch._C._jit_pass_erase_shape_information(mod_canonicalized)
        mod_str = str(mod_canonicalized)
        mod_str = re.sub(r"___torch_mangle_[0-9]+\.", "", mod_str)
        check_canonicalized = torch._C._jit_pass_canonicalize(check_mod_func.graph)
        torch._C._jit_pass_inline(check_canonicalized)
        torch._C._jit_pass_erase_shape_information(check_canonicalized)
        check_str = str(check_canonicalized)
        check_str = re.sub(r"___torch_mangle_[0-9]+\.", "", check_str)

        graph_diff_errors = None
        if mod_str != check_str:
            import difflib

            graph_diff = difflib.ndiff(
                mod_str.splitlines(True), check_str.splitlines(True)
            )
            graph_diff_errors = "Graph diff:\n" + indent("".join(graph_diff)) + "\n"

            for n_mod, n_check in zip(
                mod_canonicalized.nodes(), check_canonicalized.nodes()
            ):
                if str(n_mod) != str(n_check):
                    graph_diff_errors += "First diverging operator:\n"
                    node_diff = difflib.ndiff(
                        str(n_mod).splitlines(True), str(n_check).splitlines(True)
                    )
                    source_printout = (
                        "Node diff:\n" + indent("".join(node_diff)) + "\n"
                    )
                    mod_stack = n_mod.sourceRange()
                    if mod_stack:
                        source_printout += (
                            "Trace source location:\n" + indent(mod_stack) + "\n"
                        )
                    check_stack = n_check.sourceRange()
                    if check_stack:
                        source_printout += (
                            "Check source location:\n" + indent(check_stack) + "\n"
                        )
                    graph_diff_errors += source_printout

                    break  # For now, only print out the first pair of nodes that diverges

        tensor_compare_errors = None
        # Check Tensor-valued constant nodes
        for n_mod, n_check in zip(
            mod_canonicalized.nodes(), check_canonicalized.nodes()
        ):
            if n_mod.kind() != n_check.kind():
                break  # Graphs have already diverged

            if n_mod.kind() == "prim::Constant" and not (
                n_mod.mustBeNone() or n_check.mustBeNone()
            ):
                if not n_mod.hasAttribute("value"):
                    continue
                if n_mod.kindOf("value") != "t" or n_check.kindOf("value") != "t":
                    continue

>               mod_tensor_val = n_mod.t("value")
E               RuntimeError: required keyword attribute 'value' has the wrong type

/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:444: RuntimeError
===================================================================================== warnings summary ======================================================================================
../../../../../local/lib/python3.9/site-packages/jupyter_client/connect.py:22
  /usr/local/lib/python3.9/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

tests/test_packing.py::test_pack_unpack_array_api
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/tests/test_packing.py:275: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
    import numpy.array_api as xp

tests/test_other.py::testmod
  /usr/local/lib/python3.9/site-packages/_pytest/python.py:198: PytestReturnNotNoneWarning: Expected None, but tests/test_other.py::testmod returned TestResults(failed=0, attempted=0), which will be an error in a future version of pytest.  Did you mean to use `assert` instead of `return`?
    warnings.warn(

tests/test_einsum.py::test_layer
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/einops/layers/_einmix.py:112: UserWarning: EinMix: weight has no dimensions (means multiplication by a number)
    warnings.warn('EinMix: weight has no dimensions (means multiplication by a number)')

tests/test_layers.py::test_torch_layers_scripting
  <unknown>:827: DeprecationWarning: invalid escape sequence \s

tests/test_layers.py::test_reduce_imperative
  /usr/local/lib/python3.9/site-packages/numpy/core/_methods.py:53: RuntimeWarning: overflow encountered in reduce
    return umr_prod(a, axis, dtype, out, keepdims, initial, where)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
SKIPPED [1] tests/test_layers.py:340: Skipped
SKIPPED [1] tests/test_layers.py:244: Skipped
=================================================================== 5 failed, 98 passed, 2 skipped, 6 warnings in 33.99s ====================================================================
*** Error code 1

Reproduction steps

cd /usr/ports/misc/py-einops/work-py39/einops-0.7.0 && /usr/bin/env -i HOME=/usr/ports/misc/py-einops/work-py39  PWD="${PWD}"  __MAKE_CONF=/nonexistent OSVERSION=1400509 PATH=/usr/local/libexec/ccache:/usr/ports/misc/py-einops/work-py39/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin TERM=xterm-256color XDG_DATA_HOME=/usr/ports/misc/py-einops/work-py39  XDG_CONFIG_HOME=/usr/ports/misc/py-einops/work-py39  XDG_CACHE_HOME=/usr/ports/misc/py-einops/work-py39/.cache  HOME=/usr/ports/misc/py-einops/work-py39 PATH=/usr/local/libexec/ccache:/usr/ports/misc/py-einops/work-py39/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/usr/ports/misc/py-einops/work-py39/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS=""  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fstack-protector-strong -fno-strict-aliasing  " CCACHE_DIR="/tmp/.ccache" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" EINOPS_TEST_BACKENDS="numpy,torch" /usr/local/bin/python3.9 -m pytest -k '' -rs -v -o addopts= 
==================================================================================== test session starts ====================================================================================

Expected behavior n/a

Your platform FreeBSD 14

Version: 0.7.0 Python-3.9 pytorch-2.2.1 (the latest release)

arogozhnikov commented 7 months ago

that's kinda very interesting repro steps.

testing in repo is performed by test.py script, not pytest.

Your setup seems to lack einops (4 out of 5 errors). Torch.jit error is maybe intersting for torch, but they can't be caused by einops itself - only if smth is wrong with torch+BSD.

Can you explain your test setup and what are you trying to check/confirm? Einops is pure-python and OS-agnostic. If it works on linux, it would work in BSD too unless underlying library (numpy/torch/etc) has issues with either of these.

yurivict commented 7 months ago

It runs the standard PyTest-based test command provided by the FreeBSD ports framework. It is used by thousands of other Python-based ports.

This approach uses the same dependency packages that will be used when this package will be run by users.

There is another target that is defined in the port for testing: python test.py numpy pytorch.

It downloads a lot of dependencies for some reason, even though they are supposed to be already installed, and still has 2 failures:

Successfully installed einops-0.7.0
==================================================================================== test session starts ====================================================================================
platform freebsd14 -- Python 3.9.18, pytest-7.4.4, pluggy-1.4.0
Using --randomly-seed=3005111187
rootdir: /usr/ports/misc/py-einops/work-py39/einops-0.7.0
configfile: pyproject.toml
plugins: anyio-4.3.0, hypothesis-6.98.18, cov-4.1.0, datadir-1.5.0, randomly-3.12.0, timeout-2.1.0, time-machine-2.11.0, rerunfailures-11.1.2, flaky-3.7.0, forked-1.6.0, aspectlib-2.0.0, xdist-3.5.0, env-0.6.2, mock-3.10.0, typeguard-4.1.5
collected 105 items                                                                                                                                                                         

tests/test_examples.py ..                                                                                                                                                             [  1%]
tests/test_ops.py ........................                                                                                                                                            [ 24%]
tests/test_parsing.py ....                                                                                                                                                            [ 28%]
tests/test_notebooks.py .F..                                                                                                                                                          [ 32%]
tests/test_other.py ......................................................                                                                                                            [ 83%]
tests/test_packing.py ....                                                                                                                                                            [ 87%]
tests/test_einsum.py ....                                                                                                                                                             [ 91%]
tests/test_layers.py .F.s..s..                                                                                                                                                        [100%]

========================================================================================= FAILURES ==========================================================================================
______________________________________________________________________________________ test_notebook_3 ______________________________________________________________________________________

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x2f1de08838b0>, msg_id = '81dab144-2fafa360cbc487d82513b3d4_49971_9'
cell = {'cell_type': 'code', 'execution_count': 6, 'id': '3987bbbc-ccc2-4374-8979-fd3fb29c4910', 'metadata': {'execution': {'...it=1.)),\n    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),\n]:\n    %timeit -n 10 y = layer(x)'}
timeout = 60
task_poll_output_msg = <Task pending name='Task-256' coro=<NotebookClient._async_poll_output_msg() running at /usr/local/lib/python3.9/site-packages/nbclient/client.py:813> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x2f1ddef25e20>()]>>
task_poll_kernel_alive = <Task cancelled name='Task-255' coro=<NotebookClient._async_poll_kernel_alive() done, defined at /usr/local/lib/python3.9/site-packages/nbclient/client.py:821>>

    async def _async_poll_for_reply(
        self,
        msg_id: str,
        cell: NotebookNode,
        timeout: int | None,
        task_poll_output_msg: asyncio.Future[t.Any],
        task_poll_kernel_alive: asyncio.Future[t.Any],
    ) -> dict[str, t.Any]:
        msg: dict[str, t.Any]
        assert self.kc is not None
        new_timeout: float | None = None
        if timeout is not None:
            deadline = monotonic() + timeout
            new_timeout = float(timeout)
        error_on_timeout_execute_reply = None
        while True:
            try:
                if error_on_timeout_execute_reply:
                    msg = error_on_timeout_execute_reply  # type:ignore[unreachable]
                    msg["parent_header"] = {"msg_id": msg_id}
                else:
>                   msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))

/usr/local/lib/python3.9/site-packages/nbclient/client.py:782: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:198: in ensure_async
    result = await obj
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <jupyter_client.channels.AsyncZMQSocketChannel object at 0x2f1de088b0d0>, timeout = 60000.0

    async def get_msg(  # type:ignore[override]
        self, timeout: t.Optional[float] = None
    ) -> t.Dict[str, t.Any]:
        """Gets a message if there is one that is ready."""
        assert self.socket is not None
        if timeout is not None:
            timeout *= 1000  # seconds to ms
        ready = await self.socket.poll(timeout)
        if ready:
            res = await self._recv()
            return res
        else:
>           raise Empty
E           _queue.Empty

/usr/local/lib/python3.9/site-packages/jupyter_client/channels.py:315: Empty

During handling of the above exception, another exception occurred:

    def test_notebook_3():
        [notebook] = Path(__file__).parent.with_name("docs").glob("3-*.ipynb")
        if not is_backend_tested("torch"):
            pytest.skip()
>       render_notebook(notebook, replacements={})

tests/test_notebooks.py:66: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_notebooks.py:28: in render_notebook
    ep.preprocess(nb, {"metadata": {"path": str(filename.parent.absolute())}})
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:102: in preprocess
    self.preprocess_cell(cell, resources, index)
/usr/local/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py:123: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/usr/local/lib/python3.9/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/usr/local/lib/python3.9/asyncio/base_events.py:647: in run_until_complete
    return future.result()
/usr/local/lib/python3.9/site-packages/nbclient/client.py:1005: in async_execute_cell
    exec_reply = await self.task_poll_for_reply
/usr/local/lib/python3.9/site-packages/nbclient/client.py:806: in _async_poll_for_reply
    error_on_timeout_execute_reply = await self._async_handle_timeout(timeout, cell)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x2f1de08838b0>, timeout = 60
cell = {'cell_type': 'code', 'execution_count': 6, 'id': '3987bbbc-ccc2-4374-8979-fd3fb29c4910', 'metadata': {'execution': {'...it=1.)),\n    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),\n]:\n    %timeit -n 10 y = layer(x)'}

    async def _async_handle_timeout(
        self, timeout: int, cell: NotebookNode | None = None
    ) -> None | dict[str, t.Any]:
        self.log.error("Timeout waiting for execute reply (%is)." % timeout)
        if self.interrupt_on_timeout:
            self.log.error("Interrupting kernel")
            assert self.km is not None
            await ensure_async(self.km.interrupt_kernel())
            if self.error_on_timeout:
                execute_reply = {"content": {**self.error_on_timeout, "status": "error"}}
                return execute_reply
            return None
        else:
            assert cell is not None
>           raise CellTimeoutError.error_from_timeout_and_cell(
                "Cell execution timed out", timeout, cell
            )
E           nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 60 seconds.
E           The message was: Cell execution timed out.
E           Here is a preview of the cell contents:
E           -------------------
E           ['x = torch.zeros([32, 128, 128])', 'for layer in [', '    ResMLP_Blocks(128, dim=128, layerscale_init=1.),', '    ResMLP_Blocks2(128, dim=128, layerscale_init=1.),', '    ResMLP_Blocks3(128, dim=128, layerscale_init=1.),']
E           ...
E           ['    torch.jit.script(ResMLP_Blocks(128, dim=128, layerscale_init=1.)),', '    torch.jit.script(ResMLP_Blocks2(128, dim=128, layerscale_init=1.)),', '    torch.jit.script(ResMLP_Blocks3(128, dim=128, layerscale_init=1.)),', ']:', '    %timeit -n 10 y = layer(x)']
E           -------------------

/usr/local/lib/python3.9/site-packages/nbclient/client.py:856: CellTimeoutError
------------------------------------------------------------------------------------- Captured log call -------------------------------------------------------------------------------------
ERROR    traitlets:client.py:845 Timeout waiting for execute reply (60s).
_____________________________________________________________________________________ test_torch_layer ______________________________________________________________________________________

    def test_torch_layer():
        if not is_backend_tested("torch"):
            pytest.skip()
        else:
            # checked that torch present
            import torch
            import torch.jit

            model1 = create_torch_model(use_reduce=True)
            model2 = create_torch_model(use_reduce=False)
            input = torch.randn([10, 3, 32, 32])
            # random models have different predictions
            assert not torch.allclose(model1(input), model2(input))
            model2.load_state_dict(pickle.loads(pickle.dumps(model1.state_dict())))
            assert torch.allclose(model1(input), model2(input))

            # tracing (freezing)
>           model3 = torch.jit.trace(model2, example_inputs=input)

tests/test_layers.py:219: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:489: in _fn
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/_dynamo/external_utils.py:17: in inner
    return fn(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:806: in trace
    return trace_module(
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:1102: in trace_module
    _check_trace(
/usr/local/lib/python3.9/site-packages/torch/utils/_contextlib.py:115: in decorate_context
    return func(*args, **kwargs)
/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:575: in _check_trace
    diag_info = graph_diagnostic_info()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def graph_diagnostic_info():
        mod_canonicalized = torch._C._jit_pass_canonicalize(traced_func.graph)
        torch._C._jit_pass_inline(mod_canonicalized)
        torch._C._jit_pass_erase_shape_information(mod_canonicalized)
        mod_str = str(mod_canonicalized)
        mod_str = re.sub(r"___torch_mangle_[0-9]+\.", "", mod_str)
        check_canonicalized = torch._C._jit_pass_canonicalize(check_mod_func.graph)
        torch._C._jit_pass_inline(check_canonicalized)
        torch._C._jit_pass_erase_shape_information(check_canonicalized)
        check_str = str(check_canonicalized)
        check_str = re.sub(r"___torch_mangle_[0-9]+\.", "", check_str)

        graph_diff_errors = None
        if mod_str != check_str:
            import difflib

            graph_diff = difflib.ndiff(
                mod_str.splitlines(True), check_str.splitlines(True)
            )
            graph_diff_errors = "Graph diff:\n" + indent("".join(graph_diff)) + "\n"

            for n_mod, n_check in zip(
                mod_canonicalized.nodes(), check_canonicalized.nodes()
            ):
                if str(n_mod) != str(n_check):
                    graph_diff_errors += "First diverging operator:\n"
                    node_diff = difflib.ndiff(
                        str(n_mod).splitlines(True), str(n_check).splitlines(True)
                    )
                    source_printout = (
                        "Node diff:\n" + indent("".join(node_diff)) + "\n"
                    )
                    mod_stack = n_mod.sourceRange()
                    if mod_stack:
                        source_printout += (
                            "Trace source location:\n" + indent(mod_stack) + "\n"
                        )
                    check_stack = n_check.sourceRange()
                    if check_stack:
                        source_printout += (
                            "Check source location:\n" + indent(check_stack) + "\n"
                        )
                    graph_diff_errors += source_printout

                    break  # For now, only print out the first pair of nodes that diverges

        tensor_compare_errors = None
        # Check Tensor-valued constant nodes
        for n_mod, n_check in zip(
            mod_canonicalized.nodes(), check_canonicalized.nodes()
        ):
            if n_mod.kind() != n_check.kind():
                break  # Graphs have already diverged

            if n_mod.kind() == "prim::Constant" and not (
                n_mod.mustBeNone() or n_check.mustBeNone()
            ):
                if not n_mod.hasAttribute("value"):
                    continue
                if n_mod.kindOf("value") != "t" or n_check.kindOf("value") != "t":
                    continue

>               mod_tensor_val = n_mod.t("value")
E               RuntimeError: required keyword attribute 'value' has the wrong type

/usr/local/lib/python3.9/site-packages/torch/jit/_trace.py:444: RuntimeError
===================================================================================== warnings summary ======================================================================================
../../../../../local/lib/python3.9/site-packages/jupyter_client/connect.py:22
  /usr/local/lib/python3.9/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

tests/test_ops.py::test_repeat_array_api
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/tests/test_ops.py:540: UserWarning: The numpy.array_api submodule is still experimental. See NEP 47.
    import numpy.array_api as xp

tests/test_other.py::testmod
  /usr/local/lib/python3.9/site-packages/_pytest/python.py:198: PytestReturnNotNoneWarning: Expected None, but tests/test_other.py::testmod returned TestResults(failed=0, attempted=0), which will be an error in a future version of pytest.  Did you mean to use `assert` instead of `return`?
    warnings.warn(

tests/test_einsum.py::test_layer
  /usr/ports/misc/py-einops/work-py39/einops-0.7.0/einops/layers/_einmix.py:112: UserWarning: EinMix: weight has no dimensions (means multiplication by a number)
    warnings.warn('EinMix: weight has no dimensions (means multiplication by a number)')

tests/test_layers.py::test_reduce_imperative
  /usr/local/lib/python3.9/site-packages/numpy/core/_methods.py:53: RuntimeWarning: overflow encountered in reduce
    return umr_prod(a, axis, dtype, out, keepdims, initial, where)

tests/test_layers.py::test_torch_layers_scripting
  <unknown>:827: DeprecationWarning: invalid escape sequence \s

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
FAILED tests/test_notebooks.py::test_notebook_3 - nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 60 seconds.
FAILED tests/test_layers.py::test_torch_layer - RuntimeError: required keyword attribute 'value' has the wrong type
============================================================= 2 failed, 101 passed, 2 skipped, 6 warnings in 178.36s (0:02:58) ==============================================================
Traceback (most recent call last):
  File "/usr/ports/misc/py-einops/work-py39/einops-0.7.0/test.py", line 100, in <module>
    main()
  File "/usr/ports/misc/py-einops/work-py39/einops-0.7.0/test.py", line 96, in main
    assert return_code == 0
AssertionError
*** Error code 1

Difference probably means that test dependency version specifications aren't accurate or not set in pyproject.toml

arogozhnikov commented 7 months ago

Difference probably means that test dependency version specifications aren't accurate or not set in pyproject.toml

einops has no dependencies. It works atop whatever framework user chose, but does not require either of those, so they will not appear in requirements/pyproject.

Einops' tests are meant to be run in CI: they confirm documentation is not outdated/broken (hence notebook testing) and that interaction with all frameworks is still correct. These tests may demonstrate some errors in user setup or upstream frameworks - but unlikely to find any errors in einops - as einops is very platform-agnostic.

Exact issues you post above were already opened: #268 and #269 and those are not related to einops.