ethereum / execution-spec-tests

A Python framework and collection of test cases to generate test vectors for Ethereum execution clients
https://ethereum.github.io/execution-spec-tests
MIT License
106 stars 72 forks source link

bug(ci): debug & fix `tox -e tests` fails with python 3.10 - reenable 3.10 matrix config in CI #852

Open danceratopz opened 2 weeks ago

danceratopz commented 2 weeks ago

850 consistently failed tox_verify with the Python 3.10 config in the test env with the following test:

=========================== short test summary info ============================
FAILED tests/cancun/eip1153_tstore/test_tstorage.py::test_run_until_out_of_gas[fork_Cancun-blockchain_test-tstore_wide_address_space]
============ 1 failed, 6232 passed, 85 skipped in 252.09s (0:04:12) ============

The error points to a resolver timeout:

=================================== FAILURES ===================================
_ test_run_until_out_of_gas[fork_Cancun-blockchain_test-tstore_wide_address_space] _
[gw3] linux -- Python 3.10.15 /home/runner/work/execution-spec-tests/execution-spec-tests/.tox/tests/bin/python3
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:468: in _make_request
    six.raise_from(e, None)
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:463: in _make_request
    httplib_response = conn.getresponse()
../../../.local/share/uv/python/cpython-3.10.15-linux-x86_64-gnu/lib/python3.10/http/client.py:1375: in getresponse
    response.begin()
../../../.local/share/uv/python/cpython-3.10.15-linux-x86_64-gnu/lib/python3.10/http/client.py:318: in begin
    version, status, reason = self._read_status()
../../../.local/share/uv/python/cpython-3.10.15-linux-x86_64-gnu/lib/python3.10/http/client.py:279: in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
../../../.local/share/uv/python/cpython-3.10.15-linux-x86_64-gnu/lib/python3.10/socket.py:717: in readinto
    return self._sock.recv_into(b)
E   TimeoutError: timed out
During handling of the above exception, another exception occurred:
.tox/tests/lib/python3.10/site-packages/requests/adapters.py:667: in send
    resp = conn.urlopen(
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:802: in urlopen
    retries = retries.increment(
.tox/tests/lib/python3.10/site-packages/urllib3/util/retry.py:552: in increment
    raise six.reraise(type(error), error, _stacktrace)
.tox/tests/lib/python3.10/site-packages/urllib3/packages/six.py:770: in reraise
    raise value
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:716: in urlopen
    httplib_response = self._make_request(
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:470: in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
.tox/tests/lib/python3.10/site-packages/urllib3/connectionpool.py:358: in _raise_timeout
    raise ReadTimeoutError(
E   urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=20)
During handling of the above exception, another exception occurred:
tests/cancun/eip1153_tstore/test_tstorage.py:284: in test_run_until_out_of_gas
    state_test(env=Environment(), pre=pre, tx=tx, post=post)
src/pytest_plugins/filler/filler.py:809: in __init__
    fixture = self.generate(
src/ethereum_test_specs/state.py:176: in generate
    return self.generate_blockchain_test().generate(
src/ethereum_test_specs/blockchain.py:731: in generate
    return self.make_fixture(t8n, fork, eips)
src/ethereum_test_specs/blockchain.py:558: in make_fixture
    header, txs, requests, new_alloc, new_env = self.generate_block_data(
src/ethereum_test_specs/blockchain.py:425: in generate_block_data
    transition_tool_output = t8n.evaluate(
src/evm_transition_tool/transition_tool.py:607: in evaluate
    return self._evaluate_server(t8n_data=t8n_data, debug_output_path=debug_output_path)
src/evm_transition_tool/transition_tool.py:426: in _evaluate_server
    response = Session().post(self.server_url, json=post_data, timeout=20)
.tox/tests/lib/python3.10/site-packages/requests/sessions.py:637: in post
    return self.request("POST", url, data=data, json=json, **kwargs)
.tox/tests/lib/python3.10/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
.tox/tests/lib/python3.10/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
.tox/tests/lib/python3.10/site-packages/requests/adapters.py:713: in send
    raise ReadTimeout(e, request=request)
E   requests.exceptions.ReadTimeout: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=20)

The strange thing is that it only failed on 3.10, even though both the 3.10 and 3.12 matrix configs both run on Ubuntu. macOS seems fine (with 3.12). It wasn't reproducible locally with 3.10. As a result, the following matrix config has been temporarily disabled pending resolution, see #850.

          - os: ubuntu-24.04
            python: "3.10"
            evm-type: "stable"
            tox-cmd: "uvx --with=tox-uv tox"  # run-parallel --parallel-no-spinner"  # TODO: disable parallelisation for uv testing

This matrix config never saw a fail:

          - os: ubuntu-24.04
            python: "3.12"
            evm-type: "stable"
            tox-cmd: "uvx --with=tox-uv tox"  # run-parallel --parallel-no-spinner"
marioevz commented 5 days ago

Now we are seeing a lot of these:

_ test_valid_inputs[fork_Cancun-blockchain_test-result_Result.SUCCESS-in_bounds_z] _
[gw2] linux -- Python 3.12.6 /home/runner/work/execution-spec-tests/execution-spec-tests/.tox/tests/bin/python3
.tox/tests/lib/python3.12/site-packages/urllib3/connectionpool.py:716: in urlopen
    httplib_response = self._make_request(
.tox/tests/lib/python3.12/site-packages/urllib3/connectionpool.py:416: in _make_request
    conn.request(method, url, **httplib_request_kw)
.tox/tests/lib/python3.12/site-packages/urllib3/connection.py:244: in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
../../../.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/http/client.py:1336: in request
    self._send_request(method, url, body, headers, encode_chunked)
../../../.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/http/client.py:1382: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
../../../.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/http/client.py:1331: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
../../../.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/http/client.py:1091: in _send_output
    self.send(msg)
../../../.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/http/client.py:1035: in send
    self.connect()
.tox/tests/lib/python3.12/site-packages/requests_unixsocket/adapters.py:35: in connect
    sock.connect(socket_path)
E   ConnectionRefusedError: [Errno 111] Connection refused

I've seen at least two instances where the failure is marked on the first two tests:

4 workers [6318 items]

FF................................................................. [  67/6318]

But I'm not sure if this is a pytest artifact or is it really the first two tests that are consistently failing.