search

kiwicom / pytest-recording

A pytest plugin that allows recording network interactions via VCR.py
MIT License
425 stars 34 forks source link

[BUG] tests/test_blocking_network.py::test_pycurl* failures due to whitespace (?) #130

Closed mgorny closed 9 months ago

mgorny commented 11 months ago

Describe the bug When running the test suite via tox, I'm getting two test failures:

FAILED tests/test_blocking_network.py::test_pycurl - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
FAILED tests/test_blocking_network.py::test_pycurl_with_allowed_hosts - AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}

After adding -vv to the runpytest() method calls, I see that the mismatches seem to be on whitespace:

========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
cachedir: .tox/py311/.pytest_cache
rootdir: /tmp/pytest-recording
plugins: recording-0.13.0, httpbin-2.0.0, mock-3.12.0
collected 65 items                                                                                                                    

tests/test_blocking_network.py ............FF......                                                                             [ 30%]
tests/test_plugin.py ..........                                                                                                 [ 46%]
tests/test_recording.py ..............                                                                                          [ 67%]
tests/test_replaying.py ...............s...                                                                                     [ 96%]
tests/test_utils.py ..                                                                                                          [100%]

============================================================== FAILURES ===============================================================
_____________________________________________________________ test_pycurl _____________________________________________________________

testdir = <Testdir local('/tmp/pytest-of-mgorny/pytest-9/test_pycurl0')>

    @pytest.mark.skipif(pycurl is None, reason="Requires pycurl installed.")
    def test_pycurl(testdir):
        # When pycurl is used for network access
        testdir.makepyfile(
            r"""
    import sys
    import pytest
    import pycurl
    from io import BytesIO

    @pytest.mark.block_network
    def test_error(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, httpbin.url + "/ip")
        c.setopt(c.WRITEDATA, buffer)
        with pytest.raises(RuntimeError, match=r"^Network is disabled$"):
            c.perform()
        c.close()

    def test_work(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, httpbin.url + "/ip")
        c.setopt(c.WRITEDATA, buffer)
        c.perform()
        c.close()
        assert buffer.getvalue() == b'{"origin":"127.0.0.1"}\n'
        """
        )

        result = testdir.runpytest("-vv")
        # It should be blocked as well
>       result.assert_outcomes(passed=2)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 1} != {'passed': 2}
E         {'failed': 1} != {'failed': 0}
E         Use -v to get more diff

/tmp/pytest-recording/tests/test_blocking_network.py:364: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- /tmp/pytest-recording/.tox/py311/bin/python
cachedir: .pytest_cache
rootdir: /tmp/pytest-of-mgorny/pytest-9/test_pycurl0
plugins: recording-0.13.0, httpbin-2.0.0, mock-3.12.0
collecting ... collected 2 items

test_pycurl.py::test_error PASSED                                                                                               [ 50%]
test_pycurl.py::test_work FAILED                                                                                                [100%]

============================================================== FAILURES ===============================================================
______________________________________________________________ test_work ______________________________________________________________

httpbin = <pytest_httpbin.serve.Server object at 0x7f0e08c98990>

    def test_work(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, httpbin.url + "/ip")
        c.setopt(c.WRITEDATA, buffer)
        c.perform()
        c.close()
>       assert buffer.getvalue() == b'{"origin":"127.0.0.1"}\n'
E       assert b'{\n  "origin": "127.0.0.1"\n}\n' == b'{"origin":"127.0.0.1"}\n'
E         At index 1 diff: b'\n' != b'"'
E         Full diff:
E         - b'{"origin":"127.0.0.1"}\n'
E         + b'{\n  "origin": "127.0.0.1"\n}\n'
E         ?    ++++         +           ++

test_pycurl.py:24: AssertionError
-------------------------------------------------------- Captured stderr call ---------------------------------------------------------
127.0.0.1 - - [22/Oct/2023 09:22:19] "GET /ip HTTP/1.1" 200 28
======================================================= short test summary info =======================================================
FAILED test_pycurl.py::test_work - assert b'{\n  "origin": "127.0.0.1"\n}\n' == b'{"origin":"127.0.0.1"}\n'
===================================================== 1 failed, 1 passed in 0.56s =====================================================
___________________________________________________ test_pycurl_with_allowed_hosts ____________________________________________________

testdir = <Testdir local('/tmp/pytest-of-mgorny/pytest-9/test_pycurl_with_allowed_hosts0')>

    @pytest.mark.skipif(pycurl is None, reason="Requires pycurl installed.")
    def test_pycurl_with_allowed_hosts(testdir):
        # When pycurl is used for network access
        testdir.makepyfile(
            r"""
    import sys
    import pytest
    import pycurl
    from io import BytesIO

    @pytest.mark.block_network(allowed_hosts=["127.0.0.*", "127.0.1.1"])
    def test_allowed(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, httpbin.url + "/ip")
        c.setopt(c.WRITEDATA, buffer)
        c.perform()
        c.close()
        assert buffer.getvalue() == b'{"origin":"127.0.0.1"}\n'

    @pytest.mark.block_network(allowed_hosts=["127.0.0.*", "127.0.1.1"])
    def test_blocked(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, "http://example.com")
        c.setopt(c.WRITEDATA, buffer)
        with pytest.raises(RuntimeError, match=r"^Network is disabled$"):
            c.perform()
        c.close()
        """
        )

        result = testdir.runpytest("-s", "-vv")
        # It should be blocked as well
>       result.assert_outcomes(passed=2)
E       AssertionError: assert {'errors': 0,...pped': 0, ...} == {'errors': 0,...pped': 0, ...}
E         Omitting 4 identical items, use -vv to show
E         Differing items:
E         {'passed': 1} != {'passed': 2}
E         {'failed': 1} != {'failed': 0}
E         Use -v to get more diff

/tmp/pytest-recording/tests/test_blocking_network.py:402: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- /tmp/pytest-recording/.tox/py311/bin/python
cachedir: .pytest_cache
rootdir: /tmp/pytest-of-mgorny/pytest-9/test_pycurl_with_allowed_hosts0
plugins: recording-0.13.0, httpbin-2.0.0, mock-3.12.0
collecting ... collected 2 items

test_pycurl_with_allowed_hosts.py::test_allowed FAILED
test_pycurl_with_allowed_hosts.py::test_blocked PASSED

============================================================== FAILURES ===============================================================
____________________________________________________________ test_allowed _____________________________________________________________

httpbin = <pytest_httpbin.serve.Server object at 0x7f0e08ddf490>

    @pytest.mark.block_network(allowed_hosts=["127.0.0.*", "127.0.1.1"])
    def test_allowed(httpbin):
        buffer = BytesIO()
        c = pycurl.Curl()
        c.setopt(c.URL, httpbin.url + "/ip")
        c.setopt(c.WRITEDATA, buffer)
        c.perform()
        c.close()
>       assert buffer.getvalue() == b'{"origin":"127.0.0.1"}\n'
E       assert b'{\n  "origin": "127.0.0.1"\n}\n' == b'{"origin":"127.0.0.1"}\n'
E         At index 1 diff: b'\n' != b'"'
E         Full diff:
E         - b'{"origin":"127.0.0.1"}\n'
E         + b'{\n  "origin": "127.0.0.1"\n}\n'
E         ?    ++++         +           ++

test_pycurl_with_allowed_hosts.py:15: AssertionError
======================================================= short test summary info =======================================================
FAILED test_pycurl_with_allowed_hosts.py::test_allowed - assert b'{\n  "origin": "127.0.0.1"\n}\n' == b'{"origin":"127.0.0.1"}\n'
===================================================== 1 failed, 1 passed in 0.55s =====================================================
-------------------------------------------------------- Captured stderr call ---------------------------------------------------------
127.0.0.1 - - [22/Oct/2023 09:22:20] "GET /ip HTTP/1.1" 200 28

To Reproduce tox -e py311

Expected behavior Passing tests ;-).

Environment (please complete the following information):

Additional context I suspect this could be due to a different cURL version.

mgorny commented 11 months ago

Hmm, or perhaps it's the httpbin version.

httpbin                   0.10.1
pytest-httpbin            2.0.0
Stranger6667 commented 11 months ago

Yeah! I think we need to compare parsed values

Stranger6667 commented 9 months ago

Fixed in d56c16e7885138528205b39b62fbfcd040e39682

mgorny commented 9 months ago

Thank you!