search

fedora-python / tox-current-env

tox plugin to run tests in current Python environment
https://pypi.org/project/tox-current-env/
MIT License
23 stars 7 forks source link

0.0.8: pytest is failing in few units #52

Open kloczek opened 1 year ago

kloczek commented 1 year ago

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tox-current-env-0.0.8-4.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tox-current-env-0.0.8-4.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra =========================================================================== test session starts ============================================================================ platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8, configfile: tox.ini plugins: xdist-3.1.0 [..] sx.........................................s...s........................................................F.F.............FF..FF......F [100%] ================================================================================= FAILURES ================================================================================= ____________________________________________________________ test_noquiet_installed_packages[--print-deps-to=-] ____________________________________________________________ [gw19] linux -- Python 3.8.16 /usr/bin/python3 flag = '--print-deps-to=-' @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"]) def test_noquiet_installed_packages(flag): flags = (flag,) if flag else () result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False) assert f"\n{NATIVE_TOXENV} installed: " in result.stdout for line in result.stdout.splitlines(): if line.startswith(f"{NATIVE_TOXENV} installed: "): packages = line.rpartition(" installed: ")[-1].split(",") break # default tox produces output sorted by package names > assert packages == sorted( packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower() ) E AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...] E At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.12' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:619: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- py38 create: /tmp/pytest-of-tkloczko/pytest-487/popen-gw19/test_noquiet_installed_package0/projdir/.tox/py38 py38 installed: Babel==2.11.0,Brlapi==0.8.3,Jinja2==3.1.2,MarkupSafe==2.1.1,PyGObject==3.42.2,Pygments==2.13.0,Sphinx==5.3.0,alabaster==0.7.12,appdirs==1.4.4,attrs==22.1.0,build==0.9.0,charset-normalizer==3.0.1,cssselect==1.1.0,distlib==0.3.6,distro==1.8.0,dnspython==2.2.1,docutils==0.19,exceptiongroup==1.0.0,execnet==1.9.0,extras==1.0.0,filelock==3.8.0,fixtures==4.0.0,gpg==1.17.1-unknown,idna==3.4,imagesize==1.4.1,importlib-metadata==5.1.0,iniconfig==1.1.1,libcomps==0.1.19,lit==15.0.6,louis==3.24.0,lxml==4.9.1,meson==0.64.1,numpy==1.23.1,packaging==21.3,pbr==5.9.0,pep517==0.13.0,pip==22.3.1,platformdirs==2.5.2,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.9,pytest==7.2.0,pytest-xdist==3.1.0,python-dateutil==2.8.2,pytz==2022.4,requests==2.28.1,rpm==4.17.0,scour==0.38.2,setuptools==65.6.3,six==1.16.0,snowballstemmer==2.2.0,sphinxcontrib-applehelp==1.0.2.dev20221204,sphinxcontrib-devhelp==1.0.2.dev20221204,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1.dev20221204,sphinxcontrib-qthelp==1.0.3.dev20221204,sphinxcontrib-serializinghtml==1.1.5,testtools==2.5.0,toml==0.10.2,tomli==2.0.1,tox==3.26.0,tox-current-env==0.0.8,urllib3==1.26.12,virtualenv==20.17.0,wheel==0.38.4,zipp==3.11.0 py38 run-test-pre: PYTHONHASHSEED='3748779245' six py ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- ______________________________________________________________ test_noquiet_installed_packages[--current-env] ______________________________________________________________ [gw23] linux -- Python 3.8.16 /usr/bin/python3 flag = '--current-env' @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"]) def test_noquiet_installed_packages(flag): flags = (flag,) if flag else () result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False) assert f"\n{NATIVE_TOXENV} installed: " in result.stdout for line in result.stdout.splitlines(): if line.startswith(f"{NATIVE_TOXENV} installed: "): packages = line.rpartition(" installed: ")[-1].split(",") break # default tox produces output sorted by package names > assert packages == sorted( packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower() ) E AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...] E At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.12' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:619: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- py38 create: /tmp/pytest-of-tkloczko/pytest-487/popen-gw23/test_noquiet_installed_package0/projdir/.tox/py38 py38 installed: Babel==2.11.0,Brlapi==0.8.3,Jinja2==3.1.2,MarkupSafe==2.1.1,PyGObject==3.42.2,Pygments==2.13.0,Sphinx==5.3.0,alabaster==0.7.12,appdirs==1.4.4,attrs==22.1.0,build==0.9.0,charset-normalizer==3.0.1,cssselect==1.1.0,distlib==0.3.6,distro==1.8.0,dnspython==2.2.1,docutils==0.19,exceptiongroup==1.0.0,execnet==1.9.0,extras==1.0.0,filelock==3.8.0,fixtures==4.0.0,gpg==1.17.1-unknown,idna==3.4,imagesize==1.4.1,importlib-metadata==5.1.0,iniconfig==1.1.1,libcomps==0.1.19,lit==15.0.6,louis==3.24.0,lxml==4.9.1,meson==0.64.1,numpy==1.23.1,packaging==21.3,pbr==5.9.0,pep517==0.13.0,pip==22.3.1,platformdirs==2.5.2,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.9,pytest==7.2.0,pytest-xdist==3.1.0,python-dateutil==2.8.2,pytz==2022.4,requests==2.28.1,rpm==4.17.0,scour==0.38.2,setuptools==65.6.3,six==1.16.0,snowballstemmer==2.2.0,sphinxcontrib-applehelp==1.0.2.dev20221204,sphinxcontrib-devhelp==1.0.2.dev20221204,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1.dev20221204,sphinxcontrib-qthelp==1.0.3.dev20221204,sphinxcontrib-serializinghtml==1.1.5,testtools==2.5.0,toml==0.10.2,tomli==2.0.1,tox==3.26.0,tox-current-env==0.0.8,urllib3==1.26.12,virtualenv==20.17.0,wheel==0.38.4,zipp==3.11.0 py38 run-test-pre: PYTHONHASHSEED='1599822279' py38 run-test: commands[0] | python -c 'import os, sys; print(os.path.realpath(sys.exec_prefix), "is the exec_prefix")' /usr is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- __________________________________________________________________ test_noquiet_installed_packages[None] ___________________________________________________________________ [gw20] linux -- Python 3.8.16 /usr/bin/python3 flag = None @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"]) def test_noquiet_installed_packages(flag): flags = (flag,) if flag else () result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False) assert f"\n{NATIVE_TOXENV} installed: " in result.stdout for line in result.stdout.splitlines(): if line.startswith(f"{NATIVE_TOXENV} installed: "): packages = line.rpartition(" installed: ")[-1].split(",") break # default tox produces output sorted by package names assert packages == sorted( packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower() ) # without a flag, the output must match tox defaults if not flag: > assert len(packages) == 3 E AssertionError: assert 4 == 3 E + where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.8']) /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:625: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- GLOB sdist-make: /tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/setup.py py38 create: /tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/py38 py38 installdeps: six, py WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. py38 inst: /tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. py38 installed: py==1.11.0,six==1.16.0,test @ file:///tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip,tox-current-env==0.0.8 py38 run-test-pre: PYTHONHASHSEED='3943385495' py38 run-test: commands[0] | python -c 'import os, sys; print(os.path.realpath(sys.exec_prefix), "is the exec_prefix")' /tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/py38 is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- ______________________________________________________________________ test_regular_run_native_toxenv ______________________________________________________________________ [gw46] linux -- Python 3.8.16 /usr/bin/python3 def test_regular_run_native_toxenv(): result = tox("-e", NATIVE_TOXENV) lines = sorted(result.stdout.splitlines()[:1]) > assert f"/.tox/{NATIVE_TOXENV} is the exec_prefix" in lines[0] E assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:479: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. /tmp/pytest-of-tkloczko/pytest-487/popen-gw46/test_regular_run_native_toxenv0/projdir/.tox/py38 is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- _________________________________________________ test_regular_after_first_print_deps_is_supported[--print-deps-to-file=-] _________________________________________________ [gw16] linux -- Python 3.8.16 /usr/bin/python3 print_deps_stdout_arg = '--print-deps-to-file=-' def test_regular_after_first_print_deps_is_supported(print_deps_stdout_arg): result = tox("-e", NATIVE_TOXENV, print_deps_stdout_arg) assert result.stdout.splitlines()[0] == "six" result = tox("-e", NATIVE_TOXENV) lines = sorted(result.stdout.splitlines()[:1]) assert "--recreate" not in result.stderr > assert f"/.tox/{NATIVE_TOXENV} is the exec_prefix" in lines[0] E assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:514: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- six py ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. /tmp/pytest-of-tkloczko/pytest-487/popen-gw16/test_regular_after_first_print0/projdir/.tox/py38 is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- ___________________________________________________ test_regular_after_first_print_deps_is_supported[--print-deps-to=-] ____________________________________________________ [gw14] linux -- Python 3.8.16 /usr/bin/python3 print_deps_stdout_arg = '--print-deps-to=-' def test_regular_after_first_print_deps_is_supported(print_deps_stdout_arg): result = tox("-e", NATIVE_TOXENV, print_deps_stdout_arg) assert result.stdout.splitlines()[0] == "six" result = tox("-e", NATIVE_TOXENV) lines = sorted(result.stdout.splitlines()[:1]) assert "--recreate" not in result.stderr > assert f"/.tox/{NATIVE_TOXENV} is the exec_prefix" in lines[0] E assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:514: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- six py ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. /tmp/pytest-of-tkloczko/pytest-487/popen-gw14/test_regular_after_first_print0/projdir/.tox/py38 is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- ___________________________________________________ test_regular_after_first_print_deps_is_supported[--print-deps-only] ____________________________________________________ [gw3] linux -- Python 3.8.16 /usr/bin/python3 print_deps_stdout_arg = '--print-deps-only' def test_regular_after_first_print_deps_is_supported(print_deps_stdout_arg): result = tox("-e", NATIVE_TOXENV, print_deps_stdout_arg) assert result.stdout.splitlines()[0] == "six" result = tox("-e", NATIVE_TOXENV) lines = sorted(result.stdout.splitlines()[:1]) assert "--recreate" not in result.stderr > assert f"/.tox/{NATIVE_TOXENV} is the exec_prefix" in lines[0] E assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:514: AssertionError --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- six py ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration. /tmp/pytest-of-tkloczko/pytest-487/popen-gw3/test_regular_after_first_print0/projdir/.tox/py38 is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) --------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------- ========================================================================= short test summary info ========================================================================== SKIPPED [1] tests/test_integration.py:136: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH SKIPPED [1] tests/test_integration.py:117: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH SKIPPED [1] tests/test_integration.py:459: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH XFAIL tests/test_integration.py::test_allenvs_print_deps_to_file_print_extras_to_other_file[True-True] - reason: Unsupported combination of parameters FAILED tests/test_integration.py::test_noquiet_installed_packages[--print-deps-to=-] - AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...] FAILED tests/test_integration.py::test_noquiet_installed_packages[--current-env] - AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...] FAILED tests/test_integration.py::test_noquiet_installed_packages[None] - AssertionError: assert 4 == 3 FAILED tests/test_integration.py::test_regular_run_native_toxenv - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-to-file=-] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-to=-] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-only] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration." =========================================================== 7 failed, 122 passed, 3 skipped, 1 xfailed in 16.99s =========================================================== ```

Here is list of installed modules in build env

```console Package Version ----------------------------- ----------------- alabaster 0.7.12 appdirs 1.4.4 attrs 22.1.0 Babel 2.11.0 Brlapi 0.8.3 build 0.9.0 charset-normalizer 3.0.1 cssselect 1.1.0 distlib 0.3.6 distro 1.8.0 dnspython 2.2.1 docutils 0.19 exceptiongroup 1.0.0 execnet 1.9.0 extras 1.0.0 filelock 3.8.0 fixtures 4.0.0 gpg 1.17.1-unknown idna 3.4 imagesize 1.4.1 importlib-metadata 5.1.0 iniconfig 1.1.1 Jinja2 3.1.2 libcomps 0.1.19 louis 3.24.0 lxml 4.9.1 MarkupSafe 2.1.1 numpy 1.23.1 packaging 21.3 pbr 5.9.0 pep517 0.13.0 pip 22.3.1 platformdirs 2.5.2 pluggy 1.0.0 py 1.11.0 Pygments 2.13.0 PyGObject 3.42.2 pyparsing 3.0.9 pytest 7.2.0 pytest-xdist 3.1.0 python-dateutil 2.8.2 pytz 2022.4 requests 2.28.1 rpm 4.17.0 scour 0.38.2 setuptools 65.6.3 six 1.16.0 snowballstemmer 2.2.0 Sphinx 5.3.0 sphinxcontrib-applehelp 1.0.2.dev20221204 sphinxcontrib-devhelp 1.0.2.dev20221204 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1.dev20221204 sphinxcontrib-qthelp 1.0.3.dev20221204 sphinxcontrib-serializinghtml 1.1.5 testtools 2.5.0 toml 0.10.2 tomli 2.0.1 tox 3.26.0 urllib3 1.26.12 virtualenv 20.17.0 wheel 0.38.4 zipp 3.11.0 ```
hroncok commented 1 year ago

Does specifying PYTHONPATH in 'passenv' in the configuration at https://github.com/fedora-python/tox-current-env/blob/master/tests/fixtures/tox.ini help?

hroncok commented 1 year ago

You should be able to run:

export TOX_TESTENV_PASSENV=PYTHONPATH

In %check to simulate that. That should get around couple of the failures.

hroncok commented 1 year ago 
        # default tox produces output sorted by package names
>       assert packages == sorted(
            packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower()
        )
E       AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...]
E         At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.12'
E         Use -v to get more diff

This seems like a difference in sorting. Maybe tox sorts capital letters first?

hroncok commented 1 year ago 
        # without a flag, the output must match tox defaults
        if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-tkloczko/pytest-487/popen-gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.8'])

This I don't fully comprehend. tox-current-env==0.0.8 is not listed when I try this.

hroncok commented 1 year ago 
        # default tox produces output sorted by package names
>       assert packages == sorted(
            packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower()
        )
E       AssertionError: assert ['Babel==2.11...=2.13.0', ...] == ['alabaster==...==0.9.0', ...]
E         At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.12'
E         Use -v to get more diff

This seems like a difference in sorting. Maybe tox sorts capital letters first?

Could you please test that assumption in https://github.com/fedora-python/tox-current-env/pull/54 ?

kloczek commented 1 year ago

Does specifying PYTHONPATH in 'passenv' in the configuration at https://github.com/fedora-python/tox-current-env/blob/master/tests/fixtures/tox.ini help?

What you mean passenv? What exactly needs to be done?

kloczek commented 1 year ago

Just tested tjose three commits from master and tjose units are still failing.

========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/test_integration.py:136: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:117: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:459: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
XFAIL tests/test_integration.py::test_allenvs_print_deps_to_file_print_extras_to_other_file[True-True] - reason: Unsupported combination of parameters
FAILED tests/test_integration.py::test_noquiet_installed_packages[--print-deps-to=-] - AssertionError: assert ['Babel==2.11...==9.3.0', ...] == ['alabaster==...=2.11.0', ...]
FAILED tests/test_integration.py::test_noquiet_installed_packages[--current-env] - AssertionError: assert ['Babel==2.11...==9.3.0', ...] == ['alabaster==...=2.11.0', ...]
FAILED tests/test_integration.py::test_regular_run_native_toxenv - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration."
FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-to=-] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration."
FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-to-file=-] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration."
FAILED tests/test_integration.py::test_noquiet_installed_packages[None] - AssertionError: assert 4 == 3
FAILED tests/test_integration.py::test_regular_after_first_print_deps_is_supported[--print-deps-only] - assert '/.tox/py38 is the exec_prefix' in "WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration."
=========================================================== 7 failed, 122 passed, 3 skipped, 1 xfailed in 17.44s ===========================================================
hroncok commented 1 year ago

Does specifying PYTHONPATH in 'passenv' in the configuration at https://github.com/fedora-python/tox-current-env/blob/master/tests/fixtures/tox.ini help?

What you mean passenv? What exactly needs to be done?

Either export TOX_TESTENV_PASSENV=PYTHONPATH or edit https://github.com/fedora-python/tox-current-env/blob/master/tests/fixtures/tox.ini to add this to testenv:

passenv =
    PYTHONPATH
hroncok commented 1 year ago

Just tested tjose three commits

What 3 commits from master? There are no commits in master related to any of the failures mentioned here.

kloczek commented 1 year ago

What 3 commits from master? There are no commits in master related to any of the failures mentioned here.

Yes commits from master.

kloczek commented 1 year ago

TOX_TESTENV_PASSENV=PYTHONPATH

OK with that ebv variable I have now three failing units

========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/test_integration.py:136: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:117: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:459: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
XFAIL tests/test_integration.py::test_allenvs_print_deps_to_file_print_extras_to_other_file[True-True] - reason: Unsupported combination of parameters
FAILED tests/test_integration.py::test_noquiet_installed_packages[--current-env] - AssertionError: assert ['Babel==2.11...==9.3.0', ...] == ['alabaster==...=2.11.0', ...]
FAILED tests/test_integration.py::test_noquiet_installed_packages[--print-deps-to=-] - AssertionError: assert ['Babel==2.11...==9.3.0', ...] == ['alabaster==...=2.11.0', ...]
FAILED tests/test_integration.py::test_noquiet_installed_packages[None] - AssertionError: assert 4 == 3
=========================================================== 3 failed, 126 passed, 3 skipped, 1 xfailed in 17.29s ===========================================================
hroncok commented 1 year ago

and if you add https://github.com/fedora-python/tox-current-env/pull/54 ?

kloczek commented 1 year ago

O with above input resources

Source:         %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
Patch:          %{VCS}/commit/d2f93448.patch#/%{name}-tox-never-gets-capitalized.patch
Patch:          %{VCS}/commit/4f97be79.patch#/%{name}-Define-and-use-a-tests-extra.patch
Patch:          %{VCS}/commit/2f100547.patch#/%{name}-Update-the-branch-for-tox3-git.patch
Patch:          %{VCS}/pull/54.patch#/%{name}-Tests-Dont-assume-tox-sorts-by-lowercase.patch

I still have one failing unit

__________________________________________________________________ test_noquiet_installed_packages[None] ___________________________________________________________________
[gw25] linux -- Python 3.8.16 /usr/bin/python3

flag = None

    @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"])
    def test_noquiet_installed_packages(flag):
        flags = (flag,) if flag else ()
        result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False)
        assert f"\n{NATIVE_TOXENV} installed: " in result.stdout
        for line in result.stdout.splitlines():
            if line.startswith(f"{NATIVE_TOXENV} installed: "):
                packages = line.rpartition(" installed: ")[-1].split(",")
                break

        # default tox produces output sorted by package names
        assert packages == sorted(
            packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0]
        )

        # without a flag, the output must match tox defaults
        if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.8'])

/home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.8/tests/test_integration.py:625: AssertionError
--------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------
GLOB sdist-make: /tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/setup.py
py38 create: /tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/.tox/py38
py38 installdeps: six, py
py38 inst: /tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip
py38 installed: py==1.11.0,six==1.16.0,test @ file:///tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip,tox-current-env==0.0.8
py38 run-test-pre: PYTHONHASHSEED='2206847846'
py38 run-test: commands[0] | python -c 'import os, sys; print(os.path.realpath(sys.exec_prefix), "is the exec_prefix")'
/tmp/pytest-of-tkloczko/pytest-681/popen-gw25/test_noquiet_installed_package0/projdir/.tox/py38 is the exec_prefix
___________________________________ summary ____________________________________
  py38: commands succeeded
  congratulations :)

--------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------

========================================================================= short test summary info ==========================================================================
SKIPPED [1] tests/test_integration.py:136: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:117: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
SKIPPED [1] tests/test_integration.py:459: This test needs python3.6, 3.7, 3.8, 3.9 and 3.10 available in $PATH
XFAIL tests/test_integration.py::test_allenvs_print_deps_to_file_print_extras_to_other_file[True-True] - reason: Unsupported combination of parameters
FAILED tests/test_integration.py::test_noquiet_installed_packages[None] - AssertionError: assert 4 == 3
=========================================================== 1 failed, 128 passed, 3 skipped, 1 xfailed in 16.97s ===========================================================
hroncok commented 1 year ago

I can reproduce that one only with TOX_TESTENV_PASSENV=PYTHONPATH set but not without.

The TOX_TESTENV_PASSENV=PYTHONPATH thing is a workaround, I guess the proper fix is to ignore the warning in the test output. Let me check.

hroncok commented 1 year ago

With https://github.com/fedora-python/tox-current-env/pull/54 and https://github.com/fedora-python/tox-current-env/pull/55 and without TOX_TESTENV_PASSENV=PYTHONPATH all the tests pass for me when PYTHONPATH is set to buildroot site-packages.

I do however not use Python 3.8.

kloczek commented 1 year ago

Don't worry about pytjon 3.8. I'll be soon moving all my modules to 3.9 and then to nect major version (I'l be going over all versions to test/perfect/polish python major version upgrade procedure).

Nevertheless as you are able to reporoduce that TOX_TESTENV_PASSENV=PYTHONPAT as I have that coded in my build procedure looks like it is not python 3.8 specyfic issue.

hroncok commented 1 year ago

Could you please try the following combination?

mtelka commented 1 year ago 
       # without a flag, the output must match tox defaults
       if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-tkloczko/pytest-487/popen->gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.8'])

This I don't fully comprehend. tox-current-env==0.0.8 is not listed when I try this.

I do have the exactly same problem:

        if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-marcel/pytest-60/popen-gw3/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.10'])
mtelka commented 1 year ago 
       # without a flag, the output must match tox defaults
       if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-tkloczko/pytest-487/popen->gw20/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.8'])

This I don't fully comprehend. tox-current-env==0.0.8 is not listed when I try this.

I do have the exactly same problem:

        if not flag:
>           assert len(packages) == 3
E           AssertionError: assert 4 == 3
E            +  where 4 = len(['py==1.11.0', 'six==1.16.0', 'test @ file:///tmp/pytest-of-marcel/pytest-60/popen-gw3/test_noquiet_installed_package0/projdir/.tox/.tmp/package/1/test-0.0.0.zip', 'tox-current-env==0.0.10'])

This problem is gone with tox-current-env-0.0.11 (and tox-3.28.0).

kloczek commented 1 year ago

As latest pytest has no llonger bundled py modue tihs issue came to the surface. Many modules aterted modify tha code to drop completely py use and test suite dependency. BTW .. the same should be done with six 😋 Kind of workaround is add standalone py to the build env however this fould be only kind of temporary/JFDI solution.

kloczek commented 1 year ago

Just found that another two units are now failing

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-tox-current-env-0.0.11-3.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-tox-current-env-0.0.11-3.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' --deselect 'tests/test_integration.py::test_noquiet_installed_packages[None]' ============================= test session starts ============================== platform linux -- Python 3.8.16, pytest-7.2.1, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.11, configfile: tox.ini plugins: xdist-3.2.0 gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I gw0 [259] / gw1 [259] / gw2 [259] / gw3 [259] / gw4 [259] / gw5 [259] / gw6 [259] / gw7 [259] / gw8 [259] / gw9 [259] / gw10 [259] / gw11 [259] / gw12 [259] / gw13 [259] / gw14 [259] / gw15 [259] / gw16 [259] / gw17 [259] / gw18 [259] / gw19 [259] / gw20 [259] / gw21 [259] / gw22 [259] / gw23 [259] / gw24 [259] / gw25 [259] / gw26 [259] / gw27 [259] / gw28 [259] / gw29 [259] / gw30 [259] / gw31 [259] / gw32 [259] / gw33 [259] / gw34 [259] / gw35 [259] / gw36 [259] / gw37 [259] / gw38 [259] / gw39 [259] / gw40 [259] / gw41 [259] / gw42 [259] / gw43 [259] / gw44 [259] / gw45 [259] / gw46 [259] / gw47 [259] s................................................s...................... [ 27%] ........................................................................ [ 55%] ...........................s...........x................................ [ 83%] ......F...........F........................ [100%]current /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.11 running in /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.11 3.26.0 imported from /usr/lib/python3.8/site-packages/tox/__init__.py registered plugins: tox-current-env-0.0.11 at /home/tkloczko/rpmbuild/BUILDROOT/python-tox-current-env-0.0.11-3.fc35.x86_64/usr/lib/python3.8/site-packages/tox_current_env/hooks.py =================================== FAILURES =================================== ________________ test_noquiet_installed_packages[--current-env] ________________ [gw13] linux -- Python 3.8.16 /usr/bin/python3 flag = '--current-env' @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"]) def test_noquiet_installed_packages(flag): flags = (flag,) if flag else () result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False) assert f"\n{NATIVE_TOXENV} installed: " in result.stdout for line in result.stdout.splitlines(): if line.startswith(f"{NATIVE_TOXENV} installed: "): packages = line.rpartition(" installed: ")[-1].split(",") break # default tox produces output sorted by package names > assert packages == sorted( packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower() ) E AssertionError: assert ['Babel==2.11...=0.7.13', ...] == ['alabaster==...==0.3.6', ...] E At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.13' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.11/tests/test_integration_tox3.py:520: AssertionError ----------------------------- Captured stdout call ----------------------------- current /tmp/pytest-of-tkloczko/pytest-375/popen-gw13/test_noquiet_installed_package0/projdir running in /tmp/pytest-of-tkloczko/pytest-375/popen-gw13/test_noquiet_installed_package0/projdir py38 create: /tmp/pytest-of-tkloczko/pytest-375/popen-gw13/test_noquiet_installed_package0/projdir/.tox/py38 py38 installed: Babel==2.11.0,Jinja2==3.1.2,MarkupSafe==2.1.2,Pygments==2.14.0,Sphinx==6.1.3,alabaster==0.7.13,attrs==22.2.0,build==0.9.0,charset-normalizer==3.0.1,distlib==0.3.6,distro==1.8.0,docutils==0.19,exceptiongroup==1.0.0,execnet==1.9.0,extras==1.0.0,filelock==3.8.2,fixtures==4.0.0,gpg==1.18.0-unknown,idna==3.4,imagesize==1.4.1,importlib-metadata==6.0.0,iniconfig==2.0.0,libcomps==0.1.19,packaging==23.0,pbr==5.9.0,pep517==0.13.0,pip==22.3.1,platformdirs==2.6.0,pluggy==1.0.0,py==1.11.0,pytest==7.2.1,pytest-xdist==3.2.0,python-dateutil==2.8.2,pytz==2022.4,requests==2.28.2,rpm==4.17.0,setuptools==65.6.3,six==1.16.0,snowballstemmer==2.2.0,sphinxcontrib-applehelp==1.0.2.dev20221204,sphinxcontrib-devhelp==1.0.2.dev20230202,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1.dev20230128,sphinxcontrib-qthelp==1.0.3.dev20230128,sphinxcontrib-serializinghtml==1.1.5,testtools==2.5.0,toml==0.10.2,tomli==2.0.1,tox==3.26.0,tox-current-env==0.0.11,urllib3==1.26.12,virtualenv==20.17.1,wheel==0.38.4,zipp==3.13.0 py38 run-test-pre: PYTHONHASHSEED='1007731709' py38 run-test: commands[0] | python -c 'import os, sys; print(os.path.realpath(sys.exec_prefix), "is the exec_prefix")' /usr is the exec_prefix ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) ----------------------------- Captured stderr call ----------------------------- ______________ test_noquiet_installed_packages[--print-deps-to=-] ______________ [gw0] linux -- Python 3.8.16 /usr/bin/python3 flag = '--print-deps-to=-' @pytest.mark.parametrize("flag", [None, "--print-deps-to=-", "--current-env"]) def test_noquiet_installed_packages(flag): flags = (flag,) if flag else () result = tox("-e", NATIVE_TOXENV, *flags, quiet=False, check=False) assert f"\n{NATIVE_TOXENV} installed: " in result.stdout for line in result.stdout.splitlines(): if line.startswith(f"{NATIVE_TOXENV} installed: "): packages = line.rpartition(" installed: ")[-1].split(",") break # default tox produces output sorted by package names > assert packages == sorted( packages, key=lambda p: p.partition("==")[0].partition(" @ ")[0].lower() ) E AssertionError: assert ['Babel==2.11...=0.7.13', ...] == ['alabaster==...==0.3.6', ...] E At index 0 diff: 'Babel==2.11.0' != 'alabaster==0.7.13' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/tox-current-env-0.0.11/tests/test_integration_tox3.py:520: AssertionError ----------------------------- Captured stdout call ----------------------------- current /tmp/pytest-of-tkloczko/pytest-375/popen-gw0/test_noquiet_installed_package0/projdir running in /tmp/pytest-of-tkloczko/pytest-375/popen-gw0/test_noquiet_installed_package0/projdir py38 create: /tmp/pytest-of-tkloczko/pytest-375/popen-gw0/test_noquiet_installed_package0/projdir/.tox/py38 py38 installed: Babel==2.11.0,Jinja2==3.1.2,MarkupSafe==2.1.2,Pygments==2.14.0,Sphinx==6.1.3,alabaster==0.7.13,attrs==22.2.0,build==0.9.0,charset-normalizer==3.0.1,distlib==0.3.6,distro==1.8.0,docutils==0.19,exceptiongroup==1.0.0,execnet==1.9.0,extras==1.0.0,filelock==3.8.2,fixtures==4.0.0,gpg==1.18.0-unknown,idna==3.4,imagesize==1.4.1,importlib-metadata==6.0.0,iniconfig==2.0.0,libcomps==0.1.19,packaging==23.0,pbr==5.9.0,pep517==0.13.0,pip==22.3.1,platformdirs==2.6.0,pluggy==1.0.0,py==1.11.0,pytest==7.2.1,pytest-xdist==3.2.0,python-dateutil==2.8.2,pytz==2022.4,requests==2.28.2,rpm==4.17.0,setuptools==65.6.3,six==1.16.0,snowballstemmer==2.2.0,sphinxcontrib-applehelp==1.0.2.dev20221204,sphinxcontrib-devhelp==1.0.2.dev20230202,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1.dev20230128,sphinxcontrib-qthelp==1.0.3.dev20230128,sphinxcontrib-serializinghtml==1.1.5,testtools==2.5.0,toml==0.10.2,tomli==2.0.1,tox==3.26.0,tox-current-env==0.0.11,urllib3==1.26.12,virtualenv==20.17.1,wheel==0.38.4,zipp==3.13.0 py38 run-test-pre: PYTHONHASHSEED='2285002353' six py ___________________________________ summary ____________________________________ py38: commands succeeded congratulations :) ----------------------------- Captured stderr call ----------------------------- =========================== short test summary info ============================ SKIPPED [1] tests/test_integration_tox4.py:24: skipping tests for tox 4 SKIPPED [1] tests/test_integration_tox3.py:55: This test needs all pythons from 3.6 to 3.11 available in $PATH SKIPPED [1] tests/test_integration_tox3.py:36: This test needs all pythons from 3.6 to 3.11 available in $PATH SKIPPED [1] tests/test_integration_tox3.py:366: This test needs all pythons from 3.6 to 3.11 available in $PATH XFAIL tests/test_integration_tox3.py::test_allenvs_print_deps_to_file_print_extras_to_other_file[True-True] - reason: Unsupported combination of parameters FAILED tests/test_integration_tox3.py::test_noquiet_installed_packages[--current-env] FAILED tests/test_integration_tox3.py::test_noquiet_installed_packages[--print-deps-to=-] ============= 2 failed, 253 passed, 4 skipped, 1 xfailed in 17.29s ============= ```
hroncok commented 1 year ago

@kloczek I've asked you to test with https://github.com/fedora-python/tox-current-env/pull/54

kloczek commented 1 year ago

OK .. one sec 😋

kloczek commented 1 year ago

That PR does nit applies on top of latest version

+ /usr/bin/tar -xof -
+ /usr/bin/gzip -dc /home/tkloczko/rpmbuild/SOURCES/python-tox-current-env-0.0.11.tar.gz
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd tox-current-env-0.0.11
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/python-tox-current-env-Tests-Dont-assume-tox-sorts-by-lowercase.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
The text leading up to this was:
--------------------------
|From 096d986fd61de9d291eac6756c2385dd43de03f1 Mon Sep 17 00:00:00 2001
|From: =?UTF-8?q?Miro=20Hron=C4=8Dok?= <miro@hroncok.cz>
|Date: Thu, 8 Dec 2022 11:48:14 +0100
|Subject: [PATCH] Tests: Don't assume tox sorts by lowercase
|
|Partial fix for https://github.com/fedora-python/tox-current-env/issues/52
|---
| tests/test_integration.py | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/tests/test_integration.py b/tests/test_integration.py
|index 09abd46..a6dd78b 100644
|--- a/tests/test_integration.py
|+++ b/tests/test_integration.py
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
hroncok commented 1 year ago

It was rebased (before your comment).