Integration tests fail with httpbin 0.10.0

[mgorny]

After upgrading to httpbin == 0.10.0, the integration tests started failing. Apparently they relied on the "main" page of httpbin including "difficult sometimes" in the past but it's been simplified now.

$ tox -e py311
py311: install_deps> python -I -m pip install ipaddress pytest pytest-cov 'pytest-httpbin>=1.0.1' PyYAML Werkzeug==2.0.3
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_editable> python -I -m pip install wheel
.pkg: build_editable> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py311: install_package_deps> python -I -m pip install PyYAML 'urllib3<2; python_version < "3.10"' wrapt yarl
py311: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/vcrpy/.tox/.tmp/package/1/vcrpy-5.1.0-0.editable-py2.py3-none-any.whl
py311: commands[0]> ./runtests.sh --cov=./vcr --cov-branch --cov-report=xml --cov-append
========================================================= test session starts =========================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
cachedir: .tox/py311/.pytest_cache
rootdir: /tmp/vcrpy
configfile: pyproject.toml
plugins: httpbin-2.0.0, cov-4.1.0
collected 255 items / 9 skipped                                                                                                       

tests/integration/test_basic.py ..F..                                                                                           [  1%]
tests/integration/test_config.py ...........                                                                                    [  6%]
tests/integration/test_disksaver.py ....                                                                                        [  7%]
tests/integration/test_filter.py ..........                                                                                     [ 11%]
tests/integration/test_ignore.py ....                                                                                           [ 13%]
tests/integration/test_matchers.py ..............                                                                               [ 18%]
tests/integration/test_multiple.py .                                                                                            [ 19%]
tests/integration/test_record_mode.py ........                                                                                  [ 22%]
tests/integration/test_register_matcher.py ....                                                                                 [ 23%]
tests/integration/test_register_persister.py .F.                                                                                [ 25%]
tests/integration/test_register_serializer.py .                                                                                 [ 25%]
tests/integration/test_request.py ..                                                                                            [ 26%]
tests/integration/test_stubs.py ....                                                                                            [ 27%]
tests/integration/test_urllib2.py ..................                                                                            [ 34%]
tests/unit/test_cassettes.py ...............................                                                                    [ 47%]
tests/unit/test_errors.py ....                                                                                                  [ 48%]
tests/unit/test_filters.py ........................                                                                             [ 58%]
tests/unit/test_json_serializer.py .                                                                                            [ 58%]
tests/unit/test_matchers.py ............................                                                                        [ 69%]
tests/unit/test_migration.py ...                                                                                                [ 70%]
tests/unit/test_persist.py ....                                                                                                 [ 72%]
tests/unit/test_request.py .................                                                                                    [ 78%]
tests/unit/test_response.py ....                                                                                                [ 80%]
tests/unit/test_serialize.py ...............                                                                                    [ 86%]
tests/unit/test_stubs.py ..                                                                                                     [ 87%]
tests/unit/test_unittest.py .........                                                                                           [ 90%]
tests/unit/test_vcr.py .......................                                                                                  [ 99%]
tests/unit/test_vcr_import.py .                                                                                                 [100%]

============================================================== FAILURES ===============================================================
_________________________________________________________ test_basic_json_use _________________________________________________________

tmpdir = local('/tmp/pytest-of-mgorny/pytest-8/test_basic_json_use0'), httpbin = <pytest_httpbin.serve.Server object at 0x7f1fa8416810>

    def test_basic_json_use(tmpdir, httpbin):
        """
        Ensure you can load a json serialized cassette
        """
        test_fixture = str(tmpdir.join("synopsis.json"))
        with vcr.use_cassette(test_fixture, serializer="json"):
            response = urlopen(httpbin.url).read()
>           assert b"difficult sometimes" in response
E           assert b'difficult sometimes' in b'<!DOCTYPE html>\n<html lang="en">\n\n<head>\n    <meta charset="UTF-8">\n    <title>httpbin.org</title>\n    <link h...        <br />\n                <br />\n            </div>\n        </section>\n    </div>\n</div>\n</body>\n\n</html>'

tests/integration/test_basic.py:42: AssertionError
-------------------------------------------------------- Captured stderr call ---------------------------------------------------------
127.0.0.1 - - [14/Aug/2023 08:56:47] "GET / HTTP/1.1" 200 9593
______________________________________________ test_load_cassette_with_custom_persister _______________________________________________

tmpdir = local('/tmp/pytest-of-mgorny/pytest-8/test_load_cassette_with_custom0')
httpbin = <pytest_httpbin.serve.Server object at 0x7f1fa8416810>

    def test_load_cassette_with_custom_persister(tmpdir, httpbin):
        """
        Ensure you can load a cassette using custom persister
        """
        my_vcr = vcr.VCR()
        my_vcr.register_persister(CustomFilesystemPersister)

        test_fixture = str(tmpdir.join("synopsis.json.test"))

        with my_vcr.use_cassette(test_fixture, serializer="json"):
            response = urlopen(httpbin.url).read()
>           assert b"difficult sometimes" in response
E           assert b'difficult sometimes' in b'<!DOCTYPE html>\n<html lang="en">\n\n<head>\n    <meta charset="UTF-8">\n    <title>httpbin.org</title>\n    <link h...        <br />\n                <br />\n            </div>\n        </section>\n    </div>\n</div>\n</body>\n\n</html>'

tests/integration/test_register_persister.py:69: AssertionError
-------------------------------------------------------- Captured stderr call ---------------------------------------------------------
127.0.0.1 - - [14/Aug/2023 08:56:57] "GET / HTTP/1.1" 200 9593

---------- coverage: platform linux, python 3.11.4-final-0 -----------
Coverage XML written to file coverage.xml

======================================================= short test summary info =======================================================
FAILED tests/integration/test_basic.py::test_basic_json_use - assert b'difficult sometimes' in b'<!DOCTYPE html>\n<html lang="en">\n\n<head>\n    <meta charset="UTF-8">\n    <title>httpbin.org...
FAILED tests/integration/test_register_persister.py::test_load_cassette_with_custom_persister - assert b'difficult sometimes' in b'<!DOCTYPE html>\n<html lang="en">\n\n<head>\n    <meta charset="UTF-8">\n    <title>httpbin.org...
============================================== 2 failed, 253 passed, 9 skipped in 14.00s ==============================================

[hartwork]

After upgrading to httpbin == 0.10.0, [..]

@mgorny could you elaborate what you mean by that? Upgrade where and how? The tox environment should not be affected by "outside" httpbin and the httpbin "inside" the tox environment is no pinned and effectively already at httpbin==0.10.1 today, according to CI logs where this particular error above does not show. Was this with 0.10.0 only and temporary and has now disappeared? Would be cool to understand, thank you!

[mgorny]

FWICS, it was fixed in f5fc7aac226d9db8520ff47cf657db326e056306.

[hartwork]

@mgorny thanks for the update, I confirm that that very commit f5fc7aac226d9db8520ff47cf657db326e056306 fixed the issue you described above :+1: