search

python / mypy

Optional static typing for Python
https://www.mypy-lang.org/
Other
18.19k stars 2.78k forks source link

Missing unit test dependencies on Linux? #12903

Open edreamleo opened 2 years ago

edreamleo commented 2 years ago

Bug Report

It appears that there are several missing dependencies in test-requirements.txt, including:

  1. gcc and make.
  2. Some other dependency, whose symptom is that #include <Python.h> fails. For example:
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/edreamleo/mypy/mypyc/lib-rt -I/home/edreamleo/mypy/venv/include -I/usr/include/python3.8 -c build/__native.c -o build/temp.linux-x86_64-3.8/build/__native.o -O0 -g1 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable
---------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------
In file included from build/__native.c:1:
/home/edreamleo/mypy/mypyc/lib-rt/init.c:1:10: fatal error: Python.h: No such file or directory
    1 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

To Reproduce

Rather than including multiple (huge) console logs, I'll summarize. I'll be happy to run any tests that you might want.

apt-get remove gcc
apt-get remove make
<< install mypy >>
source venv/bin/activate
python3 -m pytest

Hundreds of tests will fail, complaining either that x86_64-linux-gnu-gcc does not exist or that make does not exist.

After:

apt-get install gcc
apt-get install make

I still get many failures: 346 failed, 9620 passed, 370 skipped, 9 xfailed, with many tests failing because #include <Python.h> fails, as shown above.

Expected Behavior

All unit tests pass (in the venv) after installing mypy.

Actual Behavior

See above.

Your Environment

mypy installation:

```console (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ python3 -m pip install -r test-requirements.txt Ignoring typed-ast: markers 'python_version < "3.8"' don't match your environment Ignoring typed-ast: markers 'python_version < "3.8"' don't match your environment Ignoring filelock: markers 'python_version < "3.7"' don't match your environment Collecting typing_extensions>=3.10 Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB) Collecting mypy_extensions>=0.4.3 Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB) Collecting tomli>=1.1.0 Using cached tomli-2.0.1-py3-none-any.whl (12 kB) Collecting types-typed-ast<1.6.0,>=1.5.0 Using cached types_typed_ast-1.5.6-py3-none-any.whl (5.9 kB) Collecting attrs>=18.0 Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB) Collecting filelock>=3.0.0 Using cached filelock-3.7.0-py3-none-any.whl (10 kB) Collecting flake8==3.9.2 Using cached flake8-3.9.2-py2.py3-none-any.whl (73 kB) Collecting flake8-bugbear==22.3.20 Using cached flake8_bugbear-22.3.20-py3-none-any.whl (19 kB) Collecting flake8-pyi>=20.5 Using cached flake8_pyi-22.5.1-py37-none-any.whl (20 kB) Collecting lxml>=4.4.0 Using cached lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.9 MB) Collecting psutil>=4.0 Using cached psutil-5.9.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (284 kB) Collecting pytest>=6.2.4 Using cached pytest-7.1.2-py3-none-any.whl (297 kB) Collecting pytest-xdist>=1.34.0 Using cached pytest_xdist-2.5.0-py3-none-any.whl (41 kB) Collecting pytest-forked<2.0.0,>=1.3.0 Using cached pytest_forked-1.4.0-py3-none-any.whl (4.9 kB) Collecting pytest-cov<3.0.0,>=2.10.0 Using cached pytest_cov-2.12.1-py2.py3-none-any.whl (20 kB) Collecting py>=1.5.2 Using cached py-1.11.0-py2.py3-none-any.whl (98 kB) Collecting typed_ast<2,>=1.5.4 Using cached typed_ast-1.5.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (897 kB) Collecting virtualenv>=20.6.0 Using cached virtualenv-20.14.1-py2.py3-none-any.whl (8.8 MB) Requirement already satisfied: setuptools!=50 in ./venv/lib/python3.8/site-packages (from -r test-requirements.txt (line 19)) (44.0.0) Collecting importlib-metadata<5.0.0,>=4.6.1 Using cached importlib_metadata-4.11.4-py3-none-any.whl (18 kB) Collecting pyflakes<2.4.0,>=2.3.0 Using cached pyflakes-2.3.1-py2.py3-none-any.whl (68 kB) Collecting pycodestyle<2.8.0,>=2.7.0 Using cached pycodestyle-2.7.0-py2.py3-none-any.whl (41 kB) Collecting mccabe<0.7.0,>=0.6.0 Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB) Collecting ast-decompiler<0.5.0,>=0.4.0; python_version < "3.9" Using cached ast_decompiler-0.4.0-py2.py3-none-any.whl (12 kB) Collecting packaging Using cached packaging-21.3-py3-none-any.whl (40 kB) Collecting pluggy<2.0,>=0.12 Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB) Collecting iniconfig Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB) Collecting execnet>=1.1 Using cached execnet-1.9.0-py2.py3-none-any.whl (39 kB) Collecting toml Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting coverage>=5.2.1 Using cached coverage-6.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB) Collecting platformdirs<3,>=2 Using cached platformdirs-2.5.2-py3-none-any.whl (14 kB) Collecting six<2,>=1.9.0 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting distlib<1,>=0.3.1 Using cached distlib-0.3.4-py2.py3-none-any.whl (461 kB) Collecting zipp>=0.5 Using cached zipp-3.8.0-py3-none-any.whl (5.4 kB) Collecting pyparsing!=3.0.5,>=2.0.2 Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) Installing collected packages: typing-extensions, mypy-extensions, tomli, types-typed-ast, attrs, filelock, pyflakes, pycodestyle, mccabe, flake8, flake8-bugbear, ast-decompiler, flake8-pyi, lxml, psutil, pyparsing, packaging, py, pluggy, iniconfig, pytest, execnet, pytest-forked, pytest-xdist, toml, coverage, pytest-cov, typed-ast, platformdirs, six, distlib, virtualenv, zipp, importlib-metadata Successfully installed ast-decompiler-0.4.0 attrs-21.4.0 coverage-6.4 distlib-0.3.4 execnet-1.9.0 filelock-3.7.0 flake8-3.9.2 flake8-bugbear-22.3.20 flake8-pyi-22.5.1 importlib-metadata-4.11.4 iniconfig-1.1.1 lxml-4.8.0 mccabe-0.6.1 mypy-extensions-0.4.3 packaging-21.3 platformdirs-2.5.2 pluggy-1.0.0 psutil-5.9.1 py-1.11.0 pycodestyle-2.7.0 pyflakes-2.3.1 pyparsing-3.0.9 pytest-7.1.2 pytest-cov-2.12.1 pytest-forked-1.4.0 pytest-xdist-2.5.0 six-1.16.0 toml-0.10.2 tomli-2.0.1 typed-ast-1.5.4 types-typed-ast-1.5.6 typing-extensions-4.2.0 virtualenv-20.14.1 zipp-3.8.0 (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ python3 -m pip install -e . Obtaining file:///home/edreamleo/mypy Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Requirement already satisfied: typing-extensions>=3.10 in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (4.2.0) Requirement already satisfied: tomli>=1.1.0; python_version < "3.11" in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (2.0.1) Requirement already satisfied: mypy-extensions>=0.4.3 in ./venv/lib/python3.8/site-packages (from mypy==0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879) (0.4.3) Installing collected packages: mypy Attempting uninstall: mypy Found existing installation: mypy 0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879 Not uninstalling mypy at /home/edreamleo/mypy, outside environment /home/edreamleo/mypy/venv Can't uninstall 'mypy'. No files were found to uninstall. Running setup.py develop for mypy Successfully installed mypy (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ hash -r ```

Environment:

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux

pip list (outer):

```console Package Version Editable project location ---------------------- -------------------- -------------------------- appdirs 1.4.3 apturl 0.5.2 astroid 2.11.5 attrs 21.4.0 bcrypt 3.1.7 blinker 1.4 Brlapi 0.7.0 certifi 2019.11.28 chardet 3.0.4 Click 7.0 colorama 0.4.3 command-not-found 0.3 cryptography 2.8 cupshelpers 1.0 dbus-python 1.2.16 defer 1.0.6 dill 0.3.4 distlib 0.3.0 distro 1.4.0 distro-info 0.23ubuntu1 docutils 0.18.1 duplicity 0.8.12.0 entrypoints 0.3 fasteners 0.14.1 filelock 3.0.12 future 0.18.2 gitdb 4.0.9 GitPython 3.1.27 httplib2 0.14.0 idna 2.8 importlib-metadata 1.5.0 iniconfig 1.1.1 isort 5.10.1 keyring 18.0.1 language-selector 0.1 launchpadlib 1.10.13 lazr.restfulclient 0.14.2 lazr.uri 1.0.3 lazy-object-proxy 1.7.1 lockfile 0.12.2 louis 3.12.0 macaroonbakery 1.3.1 Mako 1.1.0 MarkupSafe 1.1.0 mccabe 0.7.0 monotonic 1.5 more-itertools 4.2.0 netifaces 0.10.4 oauthlib 3.1.0 olefile 0.46 packaging 21.3 paramiko 2.6.0 pexpect 4.6.0 Pillow 7.0.0 pip 22.0.4 platformdirs 2.5.2 pluggy 1.0.0 protobuf 3.6.1 py 1.11.0 py-cpuinfo 8.0.0 pycairo 1.16.2 pycups 1.9.73 pyenchant 3.2.2 PyGObject 3.36.0 PyJWT 1.7.1 pylint 2.14.0b1 /home/edreamleo/ekr-pylint pymacaroons 0.13.0 PyNaCl 1.3.0 pyparsing 3.0.8 PyQt6 6.3.0 PyQt6-Qt6 6.3.0 PyQt6-sip 13.3.1 PyQt6-WebEngine 6.3.0 PyQt6-WebEngine-Qt6 6.3.0 pyRFC3339 1.1 pytest 7.1.2 pytest-benchmark 3.4.1 pytest-timeout 2.1.0 python-apt 2.0.0+ubuntu0.20.4.7 python-dateutil 2.7.3 python-debian 0.1.36ubuntu1 pytz 2019.3 pyxdg 0.26 PyYAML 5.3.1 reportlab 3.5.34 requests 2.22.0 requests-unixsocket 0.2.0 SecretStorage 2.3.1 setuptools 62.1.0 simplejson 3.16.0 six 1.14.0 smmap 5.0.0 systemd-python 234 toml 0.10.0 tomli 2.0.1 tomlkit 0.10.2 tox 3.13.2 typing_extensions 4.2.0 ubuntu-advantage-tools 27.8 ubuntu-drivers-common 0.0.0 ufw 0.36 unattended-upgrades 0.1 urllib3 1.25.8 usb-creator 0.3.7 virtualenv 20.0.17 wadllib 1.3.3 wheel 0.37.1 wrapt 1.14.1 xkit 0.0.0 zipp 1.0.0 ```

(venv) pip list:

```console edreamleo@edreamleo-MS-7D09:~$ cd mypy edreamleo@edreamleo-MS-7D09:~/mypy$ source venv/bin/activate (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ pip list Package Version Location ------------------ -------------------------------------------------- -------------------- ast-decompiler 0.4.0 attrs 21.4.0 coverage 6.4 distlib 0.3.4 execnet 1.9.0 filelock 3.7.0 flake8 3.9.2 flake8-bugbear 22.3.20 flake8-pyi 22.5.1 importlib-metadata 4.11.4 iniconfig 1.1.1 lxml 4.8.0 mccabe 0.6.1 mypy 0.970+dev.568b98a6f08205cd364a2511f8c8873f67e8b879 /home/edreamleo/mypy mypy-extensions 0.4.3 packaging 21.3 pip 20.0.2 pkg-resources 0.0.0 platformdirs 2.5.2 pluggy 1.0.0 psutil 5.9.1 py 1.11.0 pycodestyle 2.7.0 pyflakes 2.3.1 pyparsing 3.0.9 pytest 7.1.2 pytest-cov 2.12.1 pytest-forked 1.4.0 pytest-xdist 2.5.0 setuptools 44.0.0 six 1.16.0 toml 0.10.2 tomli 2.0.1 typed-ast 1.5.4 types-typed-ast 1.5.6 typing-extensions 4.2.0 virtualenv 20.14.1 zipp 3.8.0 (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ ```

I have spent several hours trying various permutations. This report is far from the last word. To repeat, I'll be happy to run any tests that you might want.

JelleZijlstra commented 2 years ago

Thanks for reporting! You probably need to install something like python3-dev for Python.h to be available. Agree that we should make it explicit that gcc and make are required.

edreamleo commented 2 years ago

@JelleZijlstra Thanks. I'll try python3-dev later today.

edreamleo commented 2 years ago

@JelleZijlstra apt-get install python3-dev fixed all but one of the test failures:

```console (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ python3 -m pytest ======================================= test session starts ======================================== platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/edreamleo/mypy, configfile: pytest.ini, testpaths: mypy/test, mypyc/test plugins: xdist-2.5.0, cov-2.12.1, forked-1.4.0 gw0 [10345] / gw1 [10345] / gw2 [10345] / gw3 [10345] / gw4 [10345] / gw5 [10345] / gw6 [10345] / gwgw0 [10345] / gw1 [10345] / gw2 [10345] / gw3 [10345] / gw4 [10345] / gw5 [10345] / gw6 [10345] / gw7 [10345] ..........................................s................................................. [ 0%] ................x........................................................................... [ 1%] ............................................................................................ [ 2%] ............................................................................................ [ 3%] ............................................................................................. [ 4%] ............................................................................................ [ 5%] ............................................................................................ [ 6%] ............................................................................................ [ 7%] ................................................................................s........... [ 8%] ....................................................s....................................... [ 8%] ...................................................................................s........ [ 9%] .................................................................................s.......... [ 10%] ............................................................................................ [ 11%] ............................................................................................. [ 12%] ............................................................................................ [ 13%] ............................................................................................ [ 14%] ............................................................................................ [ 15%] ..............s............................................................................. [ 16%] ............................................................................................ [ 16%] ............................................................................................ [ 17%] ............................................................................................ [ 18%] ............................................................................................ [ 19%] ............................................................................................ [ 20%] ............................................................................................ [ 21%] ...............................................s............................................ [ 22%] ............................................................................................ [ 23%] ............................................................................................ [ 24%] ............................................................................................ [ 24%] ............................................................................................ [ 25%] ............................................................................................ [ 26%] ............................................................................................ [ 27%] ............................................................................................ [ 28%] ............................................................................................ [ 29%] .............................................................ss............................. [ 30%] ............................................................................................ [ 31%] ............................................................................................ [ 32%] ........................................s.................................................... [ 32%] ............................................................................................ [ 33%] ....................................s........................................................ [ 34%] ............................................................................................ [ 35%] .......................................................................s.................... [ 36%] ............................................................................................ [ 37%] ............................................................................................ [ 38%] ............................................................................................ [ 39%] ............................................................................................ [ 40%] ...............................................................s............................ [ 40%] .........................................................................................s.. [ 41%] ............................................................................................ [ 42%] ............................................................................................ [ 43%] ............................................................................................ [ 44%] ............................................................................................ [ 45%] ............................................................................................ [ 46%] ..................x..........x.............................................................. [ 47%] ............................................................................................ [ 48%] ............................................................................................ [ 48%] ....................s....................................................................... [ 49%] ........................................................................s............s....... [ 50%] ............................................................................................ [ 51%] ............................................................................................ [ 52%] ............................................................................................ [ 53%] ............................................................................................ [ 54%] ............................................................................................ [ 55%] ............................................................................................ [ 56%] ...........s..............s.....................................................s........... [ 56%] ....................................ssssss.ss................................s.............. [ 57%] ..............x.....................s...............s...........................sss........s [ 58%] ....s............x.......................s.................................................. [ 59%] .sx.......s.............ss..........s...............................................s......... [ 60%] .................s.............................................sss.......................... [ 61%] ..............sss...........sssssss.ss..s......s...s..............................s.....sss. [ 62%] s....s............sss.....ssssss..s....s...ss.........s..ssss....s..s..sss..........s......s [ 63%] .........s.s........ss.ss....s....ss.ss.s.....s.s..........ss..............s.s.............. [ 64%] s.........s....ssss.s.....sss...ssss......sssssss.....s......s...s............s.....s...s... [ 64%] ................s....ss.s.sss...ss...ssssssss.ss.......ss...ss..s...s.s.ssssss.sssssssssss.s [ 65%] ssss.sssss................ss.....s.......s................................................... [ 66%] .......................s..s..........s........s...s.....s.......ssss......ss.............sss [ 67%] sss.....s.ss.s.....................s.s.s.ss...........................................s...... [ 68%] ....................ss..ss..s..s.s..s..s.s.s..s.s.s.s..s.s.s........................ssssssss [ 69%] sssssssssssssssss.s.s.ss.s..s.s.s.ss.s.s..s..s.s.s.s.ss.s.s.s.s.s.s.ss.s.s.s.....ss.ssss..ss [ 70%] s..s.s.s.s..................................................s.s.s..ss.....s................. [ 71%] ......s........................s........................s...............s................... [ 72%] .....................................s...................................................... [ 73%] .............s.................s........ss.................................................. [ 73%] ............................................................................................. [ 74%] ............................s...s........................................................... [ 75%] ............................................................................................ [ 76%] ........................................................................s................... [ 77%] ............................................................................................. [ 78%] ..............................................................s............................. [ 79%] ............................................................................................ [ 80%] ............................................................................................ [ 81%] ............................................................................................ [ 81%] ............................................................................................ [ 82%] .s........................................................................................... [ 83%] ............................................................................................. [ 84%] ...........................................s................................................ [ 85%] ....................................................ss.s...............................s.... [ 86%] ............................................................................................ [ 87%] ............s............................................................................... [ 88%] ............................................................................F............... [ 89%] ............................................................................................ [ 89%] .s.......................................................................................... [ 90%] ............................................................................................ [ 91%] ............................................................................................ [ 92%] ............................................................................................ [ 93%] ............................................................................................ [ 94%] ............................................................................................ [ 95%] ....................................s....................................................... [ 96%] ............................................................................................ [ 97%] ...................................x......x................................................. [ 97%] ...............................................................................x............ [ 98%] .............................sssssssssssssss.ssssssssssssssssss............................. [ 99%] ............................ [100%] =========================================================================== FAILURES ============================================================================ _________________________________________________________________ TestExternal.test_c_unit_test _________________________________________________________________ [gw7] linux -- Python 3.8.10 /home/edreamleo/mypy/venv/bin/python3 self = @unittest.skipIf(sys.platform.startswith("win"), "rt tests don't work on windows") def test_c_unit_test(self) -> None: """Run C unit tests in a subprocess.""" # Build Google Test, the C++ framework we use for testing C code. # The source code for Google Test is copied to this repository. cppflags: List[str] = [] env = os.environ.copy() if sys.platform == 'darwin': cppflags += ['-mmacosx-version-min=10.10', '-stdlib=libc++'] env['CPPFLAGS'] = ' '.join(cppflags) > subprocess.check_call( ['make', 'libgtest.a'], env=env, cwd=os.path.join(base_dir, 'mypyc', 'external', 'googletest', 'make')) mypyc/test/test_external.py:27: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ popenargs = (['make', 'libgtest.a'],) kwargs = {'cwd': '/home/edreamleo/mypy/mypyc/test/../../mypyc/external/googletest/make', 'env': {'COLORTERM': 'truecolor', 'CPPFLAGS': '', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DESKTOP_SESSION': 'ubuntu', ...}} retcode = 2, cmd = ['make', 'libgtest.a'] def check_call(*popenargs, **kwargs): """Run command with arguments. Wait for command to complete. If the exit code was zero then return, otherwise raise CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute. The arguments are the same as for the call function. Example: check_call(["ls", "-l"]) """ retcode = call(*popenargs, **kwargs) if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] > raise CalledProcessError(retcode, cmd) E subprocess.CalledProcessError: Command '['make', 'libgtest.a']' returned non-zero exit status 2. /usr/lib/python3.8/subprocess.py:364: CalledProcessError --------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------- g++ -isystem ../include -I.. -g -Wall -Wextra -pthread -fPIC -c \ ../src/gtest-all.cc --------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------- make: g++: Command not found make: *** [Makefile:50: gtest-all.o] Error 127 ==================================================================== short test summary info ==================================================================== FAILED mypyc/test/test_external.py::TestExternal::test_c_unit_test - subprocess.CalledProcessError: Command '['make', 'libgtest.a']' returned non-zero exit st... ============================================== 1 failed, 9965 passed, 370 skipped, 9 xfailed in 232.83s (0:03:52) =============================================== (venv) edreamleo@edreamleo-MS-7D09:~/mypy$ ```

So it looks like g++ must be installed too.

Indeed, all tests pass for me after apt-get install g++: 9966 passed, 370 skipped, 9 xfailed

ethanhs commented 2 years ago

I think a helper to try to compile a simple file that includes Python.h in conftest that can be used to skip the mypyc tests makes sense? Unless we think that the mypyc tests should always be run, in which case we can just show a helpful error.