ethereum / execution-spec-tests

A Python framework and collection of test cases to generate test vectors for Ethereum execution clients
https://ethereum.github.io/execution-spec-tests
MIT License
121 stars 77 forks source link

print an error if restricted opcodes detected in state_test function #899

Open winsvega opened 1 month ago

winsvega commented 1 month ago

I found the issue of at least one of the cases the eels resover reports an error

.venv/lib/python3.12/site-packages/requests/models.py:974: in json
    return complexjson.loads(self.text, **kwargs)
/usr/lib/python3.12/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
/usr/lib/python3.12/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
/usr/lib/python3.12/json/decoder.py:355: in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
E   json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:
tests/frontier/scenarios/test_scenarios.py:257: in test_scenarios
    state_test(env=tx_env, pre=pre, post=post, tx=tx, post_hint=hint)
src/pytest_plugins/filler/filler.py:836: in __init__
    fixture = self.generate(
src/ethereum_test_specs/state.py:188: in generate
    return self.make_state_test_fixture(t8n, fork, eips, self.post_hint)
src/ethereum_test_specs/state.py:139: in make_state_test_fixture
    transition_tool_output = t8n.evaluate(
src/ethereum_clis/transition_tool.py:487: in evaluate
    return self._evaluate_server(t8n_data=t8n_data, debug_output_path=debug_output_path)
src/ethereum_clis/transition_tool.py:314: in _evaluate_server
    output: TransitionToolOutput = TransitionToolOutput.model_validate(response.json())
.venv/lib/python3.12/site-packages/requests/models.py:978: in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
E   requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
----------------------------------------
Exception occurred during processing of request from ['local', 0]
Traceback (most recent call last):
  File "/usr/lib/python3.12/socketserver.py", line 318, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.12/socketserver.py", line 349, in process_request
    self.finish_request(request, client_address)
  File "/home/wins/.cache/ethereum-spec-evm-resolver/Cancun/src/ethereum_spec_tools/evm_tools/daemon.py", line 91, in finish_request
    super().finish_request(request, client_address)
  File "/usr/lib/python3.12/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.12/socketserver.py", line 761, in __init__
    self.handle()
  File "/usr/lib/python3.12/http/server.py", line 436, in handle
    self.handle_one_request()
  File "/usr/lib/python3.12/http/server.py", line 424, in handle_one_request
    method()
  File "/home/wins/.cache/ethereum-spec-evm-resolver/Cancun/src/ethereum_spec_tools/evm_tools/daemon.py", line 66, in do_POST
    main(args=args, out_file=out_wrapper, in_file=input)
  File "/home/wins/.cache/ethereum-spec-evm-resolver/Cancun/src/ethereum_spec_tools/evm_tools/__init__.py", line 109, in main
    return t8n_tool.run()
           ^^^^^^^^^^^^^^

so if a state test uses blockhash opcode it causes pydantic to panic and eels resolver makes issues in the logs. which is actually we should report an error. if a state test mode uses blockhash opcode it should be a blockchain test as we agreed to remove fake hash caluclations from t8n

winsvega commented 1 month ago

to reproduce

make a test with this bytecode

def test_blockhash_bug(state_test: StateTestFiller, pre: Alloc):
    """
    comment
    """
    contract = pre.deploy_contract(code=Op.BLOCKHASH(0))
    tx = Transaction(
        sender=pre.fund_eoa(),
        gas_limit=500_000_000,
        gas_price=10,
        to=contract,
        data=b"",
        value=0,
        protected=False,
    )

    state_test(env=Environment(), pre=pre, post={}, tx=tx)

here is the evmdump http://retesteth.ethdevops.io/temp/pyspecs/frontier__scenarios__test_scenarios__test_blockhash_bug/

holiman commented 1 month ago

if a state test mode uses blockhash opcode it should be a blockchain test as we agreed to remove fake hash caluclations from t8n

I object to this. Removing an opcode from statetests means that randomly generated statetests are no longer possible to execute across different clients. I don't care what it returns, as long as all clients return the same thing. Right now, they do. Why change that? It's a trivial function.

winsvega commented 1 month ago

hm. yes I see we had discussion about this. the concern is that such test cover this artificial logic of computing the hash. and we wanted to move it all to blockchain test format only to avoid that. otherwise we need to force everyone to maintain hash calculation for state tests according to this formula. ah and it seems evmone didn't like that. so yes we try to avoid using blockchain opcodes in state tests. if so we make it a blockchain test.