search

python / mypy

Optional static typing for Python
https://www.mypy-lang.org/
Other
18.58k stars 2.85k forks source link

Test failure: test_typed_pkg #4883

Closed dmoisset closed 6 years ago

dmoisset commented 6 years ago

I'm running mypy tests from master ( python/mypy@af7e834 ) with python 3.6.2; When running tests I get a failure at test_typed_package

====================================================== FAILURES =======================================================
______________________________________________ TestPEP561.test_typed_pkg ______________________________________________
[gw1] linux -- Python 3.6.2 /home/machinalis/.virtualenvs/mypy-dev/bin/python3

self = <mypy.test.testpep561.TestPEP561 testMethod=test_typed_pkg>

    def test_typed_pkg(self) -> None:
        """Tests type checking based on installed packages.

            This test CANNOT be split up, concurrency means that simultaneously
            installing/uninstalling will break tests.
            """
        test_file = 'simple.py'
        if not os.path.isdir('test-packages-data'):
            os.mkdir('test-packages-data')
        old_cwd = os.getcwd()
        os.chdir('test-packages-data')
        with open(test_file, 'w') as f:
            f.write(SIMPLE_PROGRAM)
        try:
            with self.install_package('typedpkg-stubs'):
                check_mypy_run(
                    [test_file],
                    "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                )

            # The Python 2 tests are intentionally placed after a Python 3 test to check
            # the package_dir_cache is behaving correctly.
            python2 = try_find_python2_interpreter()
            if python2:
                with self.install_package('typedpkg-stubs', python2):
                    check_mypy_run(
                        ['--python-executable={}'.format(python2), test_file],
                        "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                    )
                with self.install_package('typedpkg', python2):
                    check_mypy_run(
                        ['--python-executable={}'.format(python2), 'simple.py'],
                        "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
                    )

                with self.install_package('typedpkg', python2):
                    with self.install_package('typedpkg-stubs', python2):
                        check_mypy_run(
                            ['--python-executable={}'.format(python2), test_file],
                            "simple.py:4: error: Revealed type is 'builtins.list[builtins.str]'\n"
                        )

            with self.install_package('typedpkg'):
                check_mypy_run(
                    [test_file],
>                   "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
                )

mypy/test/testpep561.py:103: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd_line = ['simple.py'], expected_out = "simple.py:4: error: Revealed type is 'builtins.tuple[builtins.str]'\n"
expected_err = '', expected_returncode = 1

    def check_mypy_run(cmd_line: List[str],
                       expected_out: str,
                       expected_err: str = '',
                       expected_returncode: int = 1) -> None:
        """Helper to run mypy and check the output."""
        out, err, returncode = mypy.api.run(cmd_line)
>       assert out == expected_out, err
E       AssertionError: 
E       assert "simple.py:4:...ltins.str]'\n" == "simple.py:4: ...ltins.str]'\n"
E         Skipping 37 identical leading characters in diff, use -v to show
E         - 'builtins.list[builtins.str]'
E         ?            ^^^
E         + 'builtins.tuple[builtins.str]'
E         ?           +++ ^

mypy/test/testpep561.py:27: AssertionError
============================================== 1 failed in 26.85 seconds ==============================================
emmatyping commented 6 years ago

I cannot seem to repro this on either Arch or Ubuntu 17.10. This failure happens if it is finding the stub package before the typed package.

Could you give the output of python -m mypy.sitepkgs and tell me if typedpkg or typedpkg-stubs are listed in a pip freeze?

gvanrossum commented 6 years ago

Maybe checking the source could help? It looks like a simple confusion between list and tuple.

emmatyping commented 6 years ago

One package has the return type List, the other returning a Tuple so that the tests fail if the wrong package is found. This error means that the wrong package is being found first.

emmatyping commented 6 years ago

Additionally, in this specific case, based on the check being run, and the output, I can surmise that there is typedpkg-stubs installed, when it shouldn't be, and it is being picked up before the installed typedpkg package. It is possible that the uninstall failed, which I realized I don't check for (PR incoming).

gvanrossum commented 6 years ago

So maybe it's a race between different threads/subprocesses running tests?

emmatyping commented 6 years ago

All of the package install/uninstall happen in a single test case so that shouldn't happen. I designed the test case intentionally to avoid race conditions.

ilevkivskyi commented 6 years ago

Just a random idea: Maybe this is outdated info, but my instinct was always that pip uninstall doesn't actually uninstall everything (in the sense that it doesn't revert the state of everything to the same as before install). Maybe a cleaner solution would be to create a virtualenv that will be destroyed completely after the test?

emmatyping commented 6 years ago

Pip doesn't always remove everything for egg installations AIUI. However, for packages installed by pip, it should remove mostly everything. One thing it absolutely removes is the package directory, so I don't think it is an issue.

gvanrossum commented 6 years ago

It also doesn't uninstall dependencies.

On Tue, Apr 10, 2018, 19:51 Ethan Smith notifications@github.com wrote:

Pip doesn't always remove everything for egg installations AIUI. However, for packages installed by pip, it should remove mostly everything. One thing it absolutely removes is the package directory, so I don't think it is an issue.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/python/mypy/issues/4883#issuecomment-380309913, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwrMhFOtSKOLc1pYrYpJYPTqxyKiqdlks5tnW-ugaJpZM4TOlOC .

emmatyping commented 6 years ago

I just ran the test on my friends Mac in a virtualenv and this test case did not fail...

@gvanrossum and @dmoisset since you two have been able to hit this bug, could you give me some information?

It also doesn't uninstall dependencies.

Yes this is true, fortunately for my sanity, these test packages have no dependencies :)

dmoisset commented 6 years ago

@ethanhs this is my output:

(mypy-dev) machinalis@ogion ~/oss/mypy master [1]$ python -m mypy.sitepkgs
['/home/machinalis/.virtualenvs/mypy-dev/lib/python3.6/site-packages', '/home/machinalis/.virtualenvs/mypy-dev/local/lib/python3.6/dist-packages', '/home/machinalis/.virtualenvs/mypy-dev/lib/python3/dist-packages', '/home/machinalis/.virtualenvs/mypy-dev/lib/python3.6/dist-packages', '/home/machinalis/.local/lib/python3.6/site-packages']
(mypy-dev) machinalis@ogion ~/oss/mypy master $ pip freeze
apipkg==1.4
attrs==17.4.0
coverage==4.5.1
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
lxml==4.1.1
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
six==1.11.0
typed-ast==1.1.0

I don't have mypy installed in my virtualenv, I'm just running the tests from the checked out repo, I'm not sure if that might be relevant.

gvanrossum commented 6 years ago

I can still repro it, both on master and on release-0.590, using my Python 3.6.5-based venv.

alabaster==0.7.10
apipkg==1.4
attrs==17.4.0
Babel==2.5.1
certifi==2017.11.5
chardet==3.0.4
cherry-picker==0.2.6
click==6.7
coverage==4.5.1
docutils==0.14
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
gidgethub==2.4.1
idna==2.6
imagesize==0.7.1
Jinja2==2.10
junit-xml==1.8
lxml==4.1.1
MarkupSafe==1.0
mccabe==0.6.1
MonkeyType==17.12.4.dev1
more-itertools==4.1.0
mypy===0.590-dev-1fd5a30086d535d57ca87deba532ab5f36f83e45-dirty
mypy-extensions==0.3.0
mypy-protobuf==1.0
pkginfo==1.4.1
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pyannotate==1.0.5
pycodestyle==2.3.1
pyflakes==1.6.0
Pygments==2.2.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
pytz==2017.3
requests==2.18.4
requests-toolbelt==0.8.0
retype==17.12.0
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.6.5
sphinxcontrib-websupport==1.0.1
tqdm==4.19.5
twine==1.9.1
typed-ast==1.1.0
typing==3.6.4
typing-extensions==3.6.2.1
typycal==0.0.1
uritemplate==3.0.0
urllib3==1.22
virtualenv==15.1.0

(Note that the mypy refers to the mypy package in the current directory.)

JelleZijlstra commented 6 years ago

I can also repro on master on my Mac.

$ python -m mypy.sitepkgs
['/Users/jzijlstra-mpbt/py/mypy/.venv/lib/python3.6/site-packages', '/Users/jzijlstra-mpbt/Library/Python/3.6/lib/python/site-packages']
$ pip freeze
apipkg==1.4
attrs==17.4.0
coverage==4.5.1
execnet==1.5.0
flake8==3.5.0
flake8-bugbear==18.2.0
flake8-pyi==18.3.1
lxml==4.1.1
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psutil==5.4.0
py==1.5.3
pycodestyle==2.3.1
pyflakes==1.6.0
pytest==3.5.0
pytest-cov==2.5.1
pytest-forked==0.2
pytest-xdist==1.22.2
six==1.11.0
typed-ast==1.1.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
gvanrossum commented 6 years ago

So I had typedpkg and typedpkg-stubs installed in my non-venv Python 3.6.5 site-packages. No idea how that happened, but there it was. Somehow this didn't show up in the above list but it did when I deactivated the venv:

$ python3.6 -m pip freeze
certifi==2018.1.18
mypy===0.590-dev-1fd5a30086d535d57ca87deba532ab5f36f83e45-dirty
typedpkg==0.1
typedpkg-stubs==0.1
virtualenv==15.0.2

So I uninstalled these, re-activated the venv, ran the test again, and it still failed, and after deactivating the venv, typedpkg and typedpkg-stubs were there again, in the "root" python3.6!

gvanrossum commented 6 years ago

Since I don't have pytest installed in my root Python 3.6, I can't test with that, but at this point it does look like something funky's going on with my venv or with my root Python or with the relationship between the two. Note that my root Python 3.6 lives in /usr/local/bin/python3.6 which is a symlink to ../../../Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 (where the initial ../../../ is equivalent to /) and the typedpkg I found was in /Users/guido/Library/Python/3.6/lib/python/site-packages/typedpkg/

JelleZijlstra commented 6 years ago

That's a great find! I looked a bit more and found that in this code:

        # if we aren't in a virtualenv, install in the
        # user package directory so we don't need sudo
        # In a virtualenv, real_prefix is patched onto
        # sys
        if not hasattr(sys, 'real_prefix') or python_executable != sys.executable:
            install_cmd.append('--user')

The real_prefix check doesn't actually work: we call pip with --user even if I run the test inside a virtualenv. If I just comment out those lines and manually remove the typedpkg global install, the test passes.

So it looks like we need a better way to figure out whether we're in a virtualenv, or perhaps the test should just always create a new virtualenv for itself.

dmoisset commented 6 years ago

Thanks for the fix, I can confirm this has been fixed in Ubuntu too