search

ReproNim / reproman

ReproMan (AKA NICEMAN, AKA ReproNim TRD3)
https://reproman.readthedocs.io
Other
24 stars 14 forks source link

pytest fails to find/bind fixture: "AttributeError: skipif_no_docker_engine" #429

Closed yarikoptic closed 5 years ago

yarikoptic commented 5 years ago

Things were working and then they stopped... tried in another clone with the same 'success'. anyone has a clue?

Here is a full protocol, hopefully without leaking any credentials ;)

(git)hopa:~/proj/repronim/reproman-master[master]git
$> python -m pytest -s -v reproman                                                                   
===================================================== test session starts =====================================================
platform linux -- Python 3.7.2, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /home/yoh/proj/repronim/reproman-master/venvs/dev3/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/yoh/proj/repronim/reproman-master/.hypothesis/examples')
rootdir: /home/yoh/proj/repronim/reproman-master, inifile:
plugins: localserver-0.5.0, hypothesis-3.71.11
collecting 135 items / 2 errors                                                                                               2019-05-30 12:06:28,320 [ERROR  ] stderr| Error: communication error
| CEDAR:6001:Failed to connect to <127.0.0.1:9618> 
2019-05-30 12:06:28,320 [ERROR  ] Failed to run ['condor_status'] under None. Exit code=1. out= err=Error: communication error
| CEDAR:6001:Failed to connect to <127.0.0.1:9618>
|  
collected 374 items / 2 errors                                                                                                

=========================================================== ERRORS ============================================================
________________________________ ERROR collecting reproman/distributions/tests/test_docker.py _________________________________
reproman/distributions/tests/test_docker.py:23: in <module>
    @mark.skipif_no_docker_engine
reproman/tests/skip.py:207: in __getattr__
    return super(Mark, self).__getattr__(item)
reproman/tests/skip.py:166: in __getattr__
    raise AttributeError(item) from None
E   AttributeError: skipif_no_docker_engine
______________________________ ERROR collecting reproman/resource/tests/test_docker_container.py ______________________________
reproman/resource/tests/test_docker_container.py:168: in <module>
    @mark.skipif_no_docker_engine
reproman/tests/skip.py:207: in __getattr__
    return super(Mark, self).__getattr__(item)
reproman/tests/skip.py:166: in __getattr__
    raise AttributeError(item) from None
E   AttributeError: skipif_no_docker_engine
===============================================
and even fullter ``` (git)hopa:~/proj/repronim/reproman-master[master]git $> virtualenv --system-site-packages --python python3 venvs/dev3 Already using interpreter /usr/bin/python3 Using base prefix '/usr' New python executable in /home/yoh/proj/repronim/reproman-master/venvs/dev3/bin/python3 Also creating executable in /home/yoh/proj/repronim/reproman-master/venvs/dev3/bin/python Installing setuptools, pkg_resources, pip, wheel...done. 1 12721.....................................:Thu 30 May 2019 12:07:07 PM EDT:. (git)hopa:~/proj/repronim/reproman-master[master]git $> source venvs/dev3/bin/activate (dev3) 1 12723.....................................:Thu 30 May 2019 12:07:44 PM EDT:. (git)hopa:~/proj/repronim/reproman-master[master]git $> pip install -e . Obtaining file:///home/yoh/proj/repronim/reproman-master Requirement already satisfied: appdirs in /usr/lib/python3/dist-packages (from reproman==0.1.0) (1.4.3) Requirement already satisfied: attrs>=16.3.0 in /usr/lib/python3/dist-packages (from reproman==0.1.0) (18.2.0) Requirement already satisfied: humanize in /usr/lib/python3/dist-packages (from reproman==0.1.0) (0.5.1) Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from reproman==0.1.0) (3.13) Requirement already satisfied: tqdm in /usr/lib/python3/dist-packages (from reproman==0.1.0) (4.28.1) Collecting fabric>=2.3.1 (from reproman==0.1.0) Using cached https://files.pythonhosted.org/packages/d9/e4/e6fa248c94ee5d45def54b609fcf70f39d0b7f7050f2d4405c5f156b5516/fabric-2.4.0-py2.py3-none-any.whl Requirement already satisfied: cryptography>=1.5 in /usr/lib/python3/dist-packages (from reproman==0.1.0) (2.6.1) Requirement already satisfied: pytz in /usr/lib/python3/dist-packages (from reproman==0.1.0) (2018.9) Requirement already satisfied: scp in /usr/lib/python3/dist-packages (from reproman==0.1.0) (0.13.0) Requirement already satisfied: pycrypto in /usr/lib/python3/dist-packages (from reproman==0.1.0) (2.6.1) Collecting pyOpenSSL==16.2.0 (from reproman==0.1.0) Using cached https://files.pythonhosted.org/packages/ac/93/b4cd538d31adacd07f83013860db6b88d78755af1f3fefe68ec22d397e7b/pyOpenSSL-16.2.0-py2.py3-none-any.whl Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from reproman==0.1.0) (2.21.0) Requirement already satisfied: rpaths in /usr/lib/python3/dist-packages (from reproman==0.1.0) (0.13) Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from reproman==0.1.0) (2.10) Requirement already satisfied: reprozip in /usr/lib/python3/dist-packages (from reproman==0.1.0) (1.0.14) Collecting invoke<2.0,>=1.1 (from fabric>=2.3.1->reproman==0.1.0) Using cached https://files.pythonhosted.org/packages/be/9f/8508712c9cad73ac0c8eeb2c3e51c9ef65136653dda2b512bde64109f023/invoke-1.2.0-py3-none-any.whl Requirement already satisfied: paramiko>=2.4 in /usr/lib/python3/dist-packages (from fabric>=2.3.1->reproman==0.1.0) (2.4.2) Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from pyOpenSSL==16.2.0->reproman==0.1.0) (1.12.0) Installing collected packages: invoke, fabric, pyOpenSSL, reproman Found existing installation: pyOpenSSL 19.0.0 Not uninstalling pyopenssl at /usr/lib/python3/dist-packages, outside environment /home/yoh/proj/repronim/reproman-master/venvs/dev3 Can't uninstall 'pyOpenSSL'. No files were found to uninstall. Running setup.py develop for reproman Successfully installed fabric-2.4.0 invoke-1.2.0 pyOpenSSL-16.2.0 reproman (dev3) 1 12724.....................................:Thu 30 May 2019 12:07:52 PM EDT:. (git)hopa:~/proj/repronim/reproman-master[master]git $> python -m pytest -s -v reproman ===================================================== test session starts ===================================================== platform linux -- Python 3.7.3rc1, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /home/yoh/proj/repronim/reproman-master/venvs/dev3/bin/python cachedir: .pytest_cache hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/yoh/proj/repronim/reproman-master/.hypothesis/examples') rootdir: /home/yoh/proj/repronim/reproman-master, inifile: plugins: localserver-0.5.0, hypothesis-3.71.11 collecting 59 items / 1 errors 2019-05-30 12:07:57,609 [ERROR ] stderr| Error: communication error | CEDAR:6001:Failed to connect to <127.0.0.1:9618> 2019-05-30 12:07:57,609 [ERROR ] Failed to run ['condor_status'] under None. Exit code=1. out= err=Error: communication error | CEDAR:6001:Failed to connect to <127.0.0.1:9618> | collected 374 items / 2 errors =========================================================== ERRORS ============================================================ ________________________________ ERROR collecting reproman/distributions/tests/test_docker.py _________________________________ reproman/distributions/tests/test_docker.py:23: in @mark.skipif_no_docker_engine reproman/tests/skip.py:207: in __getattr__ return super(Mark, self).__getattr__(item) reproman/tests/skip.py:166: in __getattr__ raise AttributeError(item) from None E AttributeError: skipif_no_docker_engine ______________________________ ERROR collecting reproman/resource/tests/test_docker_container.py ______________________________ reproman/resource/tests/test_docker_container.py:168: in @mark.skipif_no_docker_engine reproman/tests/skip.py:207: in __getattr__ return super(Mark, self).__getattr__(item) reproman/tests/skip.py:166: in __getattr__ raise AttributeError(item) from None E AttributeError: skipif_no_docker_engine ====================================================== warnings summary ======================================================= /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:130 /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:130: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:175 /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:175: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:334 /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:334: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:494 /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:494: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:538 /usr/lib/python3/dist-packages/pyasn1/type/constraint.py:538: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/pyasn1/type/namedval.py:54 /usr/lib/python3/dist-packages/pyasn1/type/namedval.py:54: DeprecationWarning: invalid escape sequence \* """ /usr/lib/python3/dist-packages/datalad/utils.py:75 /usr/lib/python3/dist-packages/datalad/utils.py:75: DeprecationWarning: dist() and linux_distribution() functions are deprecated in Python 3.5 = platform.linux_distribution()[:2] /usr/lib/python3/dist-packages/datalad/utils.py:961 /usr/lib/python3/dist-packages/datalad/utils.py:961: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working is_decorating = not kwargs and len(args) == 1 and isinstance(args[0], collections.Callable) /usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3 /usr/lib/python3/dist-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Mapping -- Docs: https://docs.pytest.org/en/latest/warnings.html !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ============================================= 9 warnings, 2 error in 1.46 seconds ```
kyleam commented 5 years ago

Hmm, odd. I haven't had any luck triggering it on my end, including a fresh clone and new virtual environment (no system site packages). Have you tried without system site packages?

yarikoptic commented 5 years ago

Thanks for trying! I will dig

yarikoptic commented 5 years ago
just another proof that I am not insane! ```shell $> git clone http://github.com/ReproNim/reproman && cd reproman && virtualenv --system-site-packages --python python3 venvs/dev3 && source venvs/dev3/bin/activate && python -m pytest -s -v reproman Cloning into 'reproman'... warning: redirecting to https://github.com/ReproNim/reproman/ remote: Enumerating objects: 69, done. remote: Counting objects: 100% (69/69), done. remote: Compressing objects: 100% (42/42), done. remote: Total 11839 (delta 32), reused 42 (delta 27), pack-reused 11770 Receiving objects: 100% (11839/11839), 2.70 MiB | 1.79 MiB/s, done. Resolving deltas: 100% (9170/9170), done. CHANGELOG.md Dockerfile-debian Singularity formatters.py reproman.cfg setup.cfg tools/ CONTRIBUTING.md Dockerfile-redhat docker-compose.yml readthedocs.yml requirements-devel.txt setup.py* tox.ini COPYING README.md docs/ reproman/ requirements.txt setup_support.py Running virtualenv with interpreter /home/yoh/proj/repronim/reproman-master/venvs/dev3/bin/python3 Using real prefix '/usr' Path not in prefix '/home/yoh/proj/repronim/reproman-master/venvs/dev3/include/python3.7m' '/usr' New python executable in /tmp/reproman/venvs/dev3/bin/python3 Also creating executable in /tmp/reproman/venvs/dev3/bin/python Installing setuptools, pkg_resources, pip, wheel...done. ===================================================== test session starts ===================================================== platform linux -- Python 3.7.3rc1, pytest-3.10.1, py-1.7.0, pluggy-0.8.0 -- /tmp/reproman/venvs/dev3/bin/python cachedir: .pytest_cache hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/reproman/.hypothesis/examples') rootdir: /tmp/reproman, inifile: plugins: localserver-0.5.0, hypothesis-3.71.11 collecting 124 items / 1 errors 2019-05-30 15:32:11,786 [ERROR ] stderr| Error: communication error | CEDAR:6001:Failed to connect to <127.0.0.1:9618> 2019-05-30 15:32:11,786 [ERROR ] Failed to run ['condor_status'] under None. Exit code=1. out= err=Error: communication error | CEDAR:6001:Failed to connect to <127.0.0.1:9618> | collected 355 items / 4 errors =========================================================== ERRORS ============================================================ ________________________________ ERROR collecting reproman/distributions/tests/test_docker.py _________________________________ reproman/distributions/tests/test_docker.py:23: in @mark.skipif_no_docker_engine reproman/tests/skip.py:207: in __getattr__ return super(Mark, self).__getattr__(item) reproman/tests/skip.py:166: in __getattr__ raise AttributeError(item) from None E AttributeError: skipif_no_docker_engine ______________________________ ERROR collecting reproman/resource/tests/test_docker_container.py ______________________________ reproman/resource/tests/test_docker_container.py:168: in @mark.skipif_no_docker_engine reproman/tests/skip.py:207: in __getattr__ return super(Mark, self).__getattr__(item) reproman/tests/skip.py:166: in __getattr__ raise AttributeError(item) from None E AttributeError: skipif_no_docker_engine __________________________________ ERROR collecting reproman/resource/tests/test_session.py ___________________________________ ImportError while importing test module '/tmp/reproman/reproman/resource/tests/test_session.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: reproman/resource/tests/test_session.py:12: in from fabric import Connection E ModuleNotFoundError: No module named 'fabric' ___________________________________ ERROR collecting reproman/resource/tests/test_shell.py ____________________________________ ImportError while importing test module '/tmp/reproman/reproman/resource/tests/test_shell. ```
kyleam commented 5 years ago

just another proof that I am not insane!

git clone http://github.com/ReproNim/reproman && cd reproman && virtualenv --system-site-packages --python python3 venvs/dev3 && source venvs/dev3/bin/activate && python -m pytest -s -v

Hmm, looks like you don't have a pip install call above (though you do have it in the earlier snippet, so I don't think that is related to the failure).

I figured I might as well try this, but of course --system-site-packages makes it very dependent on the system. I don't even have pytest installed globally:

% git clone http://github.com/ReproNim/reproman && cd reproman && virtualenv --system-site-packages --python python3 venvs/dev3 && source venvs/dev3/bin/activate && python -m pytest -s -v
Cloning into 'reproman'...
warning: redirecting to https://github.com/ReproNim/reproman/
remote: Enumerating objects: 69, done.
remote: Counting objects: 100% (69/69), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 11839 (delta 32), reused 42 (delta 27), pack-reused 11770
Receiving objects: 100% (11839/11839), 2.70 MiB | 7.57 MiB/s, done.
Resolving deltas: 100% (9170/9170), done.
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/rman-lOPqkaU/reproman/venvs/dev3/bin/python3
Also creating executable in /tmp/rman-lOPqkaU/reproman/venvs/dev3/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
/tmp/rman-lOPqkaU/reproman/venvs/dev3/bin/python: No module named pytest
yarikoptic commented 5 years ago

ok -- the actual reason was masked away by

    def __getattr__(self, item):
        if item.startswith("skipif_"):
            try:
                return super(Mark, self).__getattr__(item[len("skipif_"):])
            except AttributeError:
                # Fall back to the original item name so that the attribute
                # error message doesn't confusingly drop "skipif_".
                pass
        return super(Mark, self).__getattr__(item)

where my system wide docker-py was probably too old or new (3.4.1-4) and the check above was throwing up with 

______________________ ERROR collecting reproman/resource/tests/test_docker_container.py _______________________
reproman/tests/skip.py:202: in __getattr__
    return super(Mark, self).__getattr__(item[len("skipif_"):])
reproman/tests/skip.py:167: in __getattr__
    return self.attr_value(condfn)
reproman/tests/skip.py:196: in attr_value
    reason, cond = condition_func()
reproman/tests/skip.py:103: in no_docker_engine
    return "docker engine not running", not is_engine_running()
reproman/tests/skip.py:97: in is_engine_running
    return DockerContainer.is_engine_running()
reproman/resource/docker_container.py:74: in is_engine_running
    session = docker.Client(base_url=base_url)
E   AttributeError: module 'docker' has no attribute 'Client'

which was happily caught in above code and "pass"ed on. IMHO

  1. this try/except should be avoided if possible or somehow match only specific exception instances
  2. check for having compatible docker module be fortified
yarikoptic commented 5 years ago

unfortunately I do not fully understand that except handling, despite the comment, so I better stay away from trying to fix it up.

kyleam commented 5 years ago

OK, thanks for the details. I'll look into it.