RpcError (unknown_seed) on `self.bake_block()` with `SandboxedNodeTestCase`

[ztepler]

Encountered RpcError: proto.018-Proxford.seed.unknown_seed during block baking using SandboxedNodeTestCase in PyTezos. Error manifests around the 15th block bake.

• Python version: 3.11.6
• PyTezos version: 3.11.1

Steps to reproduce:

Assuming there is the following test test.py containing:

from pytezos.sandbox.node import SandboxedNodeTestCase

class BaseTestCase(SandboxedNodeTestCase):
    def test_should_allow_to_bake_many_blocks(self) -> None:
        for block in range(100):
            self.bake_block()
            print(f'baked block #{block}')

Running this test with poetry run pytest test.py I am getting the following error:

===================================================== test session starts =====================================================
platform linux -- Python 3.10.13, pytest-8.0.1, pluggy-1.4.0
rootdir: /home/pytezos-unknown-seed-issue
plugins: anyio-4.2.0
collected 1 item                                                                                                              

test.py F                                                                                                               [100%]

========================================================== FAILURES ===========================================================
_____________________________________ BaseTestCase.test_should_allow_to_bake_many_blocks ______________________________________

self = <test.BaseTestCase testMethod=test_should_allow_to_bake_many_blocks>

    def test_should_allow_to_bake_many_blocks(self) -> None:
        for block in range(100):
>           self.bake_block()

test.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/sandbox/node.py:165: in bake_block
    key = get_next_baker_key(cls.get_client())
../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/sandbox/node.py:74: in get_next_baker_key
    baking_rights = client.shell.head.helpers.baking_rights()
../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/rpc/helpers.py:21: in __call__
    return self._get(
../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/rpc/query.py:124: in _get
    return self.node.get(
../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/rpc/node.py:128: in get
    return self.request('GET', path, params=params, timeout=timeout).json()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pytezos.rpc.node.RpcNode object at 0x7f95d3759f60>

Node address
http://localhost:8732, method = 'GET'
path = '/chains/main/blocks/head/helpers/baking_rights'
kwargs = {'params': {'all': None, 'cycle': None, 'delegate': None, 'level': None, ...}, 'timeout': None}
res = <Response [500]>

    def request(self, method: str, path: str, **kwargs) -> requests.Response:
        """Perform HTTP request to node.

        :param method: one of GET/POST/PUT/DELETE
        :param path: path to endpoint
        :param kwargs: requests.request arguments
        :raises RpcError: node has returned an error
        :returns: node response
        """
        logger.debug('>>>>> %s %s\n%s', method, path, json.dumps(kwargs, indent=4))
        res = requests.request(
            method=method,
            url=_urljoin(self.uri[0], path),
            headers={'content-type': 'application/json', 'user-agent': 'PyTezos', **self.headers},
            **kwargs,
        )
        if res.status_code == 401:
            logger.debug('<<<<< %s\n%s', res.status_code, res.text)
            raise RpcError(f'Unauthorized: {path}')
        if res.status_code == 404:
            logger.debug('<<<<< %s\n%s', res.status_code, res.text)
            raise RpcError(f'Not found: {path}')
        if res.status_code != 200:
            logger.debug('<<<<< %s\n%s', res.status_code, pformat(res.text, indent=4))
>           raise RpcError.from_response(res)
E           pytezos.rpc.node.RpcError: ({'id': 'proto.018-Proxford.seed.unknown_seed',
E             'kind': 'permanent',
E             'latest': 1,
E             'oldest': 0,
E             'requested': 2},)

../../../.cache/pypoetry/virtualenvs/pytezos-unknown-seed-issue-wzzsANlG-py3.10/lib/python3.10/site-packages/pytezos/rpc/node.py:117: RpcError
---------------------------------------------------- Captured stderr setup ----------------------------------------------------
Pulling image bakingbad/sandboxed-node:v19.0-1
Container started: 12c84ddbadcc
----------------------------------------------------- Captured log setup ------------------------------------------------------
INFO     testcontainers.core.container:container.py:53 Pulling image bakingbad/sandboxed-node:v19.0-1
INFO     testcontainers.core.container:container.py:64 Container started: 12c84ddbadcc
---------------------------------------------------- Captured stdout call -----------------------------------------------------
baked block #0
baked block #1
baked block #2
baked block #3
baked block #4
baked block #5
baked block #6
baked block #7
baked block #8
baked block #9
baked block #10
baked block #11
baked block #12
baked block #13
baked block #14
=================================================== short test summary info ===================================================
FAILED test.py::BaseTestCase::test_should_allow_to_bake_many_blocks - pytezos.rpc.node.RpcError: ({'id': 'proto.018-Proxford.seed.unknown_seed',
====================================================== 1 failed in 8.46s ======================================================

Here is the repository containing the code for the test shown: https://github.com/ztepler/pytezos-unknown-seed-issue Thanks.

[ztepler]

I find out that there is a request to the baking_rights RPC during each self.bake_block() call. At the time the of the block with level 16, the request to the http://localhost:8732/chains/main/blocks/16/helpers/baking_rights returns the following status 500 response:

[
  {
    "kind": "permanent",
    "id": "proto.018-Proxford.seed.unknown_seed",
    "oldest": 0,
    "requested": 2,
    "latest": 1
  }
]

Not sure what could go wrong with the seed at this step. Is it somehow related to the new staking mechanism introduced in the Oxford update maybe?