search

nolar / kopf

A Python framework to write Kubernetes operators in just a few lines of code
https://kopf.readthedocs.io/
MIT License
2.07k stars 158 forks source link

RuntimeError: Session is closed in kopf.testing.KopfRunner #877

Open retr0h opened 2 years ago

retr0h commented 2 years ago

Long story short

I encounter the following error a majority of the time when running python kubernetes queries inside the kopf.testing.KopfRunner context. Occasionally, it will succeed, but most of the time it fails as shown below.

Error:

self = <aiohttp.client.ClientSession object at 0x12258f8e0>, method = 'get', str_or_url = 'https://0.0.0.0:49746/apis/admissionregistration.k8s.io/v1'

    async def _request(
        self,
        method: str,
        str_or_url: StrOrURL,
        *,
        params: Optional[Mapping[str, str]] = None,
        data: Any = None,
        json: Any = None,
        cookies: Optional[LooseCookies] = None,
        headers: Optional[LooseHeaders] = None,
        skip_auto_headers: Optional[Iterable[str]] = None,
        auth: Optional[BasicAuth] = None,
        allow_redirects: bool = True,
        max_redirects: int = 10,
        compress: Optional[str] = None,
        chunked: Optional[bool] = None,
        expect100: bool = False,
        raise_for_status: Optional[bool] = None,
        read_until_eof: bool = True,
        proxy: Optional[StrOrURL] = None,
        proxy_auth: Optional[BasicAuth] = None,
        timeout: Union[ClientTimeout, object] = sentinel,                                                                                                                 verify_ssl: Optional[bool] = None,
        fingerprint: Optional[bytes] = None,                                                                                                                              ssl_context: Optional[SSLContext] = None,
        ssl: Optional[Union[SSLContext, bool, Fingerprint]] = None,                                                                                                       proxy_headers: Optional[LooseHeaders] = None,
        trace_request_ctx: Optional[SimpleNamespace] = None,                                                                                                              read_bufsize: Optional[int] = None,
    ) -> ClientResponse:

        # NOTE: timeout clamps existing connect and read timeouts.  We cannot
        # set the default to None because we need to detect if the user wants
        # to use the existing timeouts by setting timeout to None.

        if self.closed:
>           raise RuntimeError("Session is closed")
E           RuntimeError: Session is closed

However, if I shell out to kubectl I do not encounter this issue.

Kopf version

1.35.3

Kubernetes version

v1.19.2+k3s1

Python version

3.8.12

Code

def test_operator(_handlers, _setup):
    with kopf.testing.KopfRunner(
        [
            "run",
            "-A",
            "--verbose",
            _handlers,
        ]
    ) as runner:
        cr = CLIENT.get_namespaced_custom_object("test-stack-us-east-2b")


### Logs

_No response_

### Additional information

_No response_
nolar commented 2 years ago

Can you please show what CLIENT is? As I see by the used method named (get_namespaced_custom_object), it is the kubernetes client library, isn't it? If so, can you please show the full stack trace of the aiohttp-related exception?

retr0h commented 2 years ago

@nolar closing as it appears my k3d instance becomes wacky when my system is put to sleep and awakes a few times. Deleting the k3d cluster and re-creating prevents this error.

retr0h commented 2 years ago

Reopening as I seem to have this particular issue when running pykube-ng within the test suite executing KopfRunner.

Full stack as requested before.

====================================================================== test session starts =======================================================================
platform darwin -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/john.dewey/Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/bin/python
cachedir: .pytest_cache
rootdir: /Users/john.dewey/git/corelight-container-sensor-operator, configfile: pyproject.toml
plugins: skip-slow-0.0.2, pyfakefs-4.5.5, mock-3.7.0, clarity-1.0.1, cov-3.0.0
collected 1 item

test/test_handlers.py::test_operator FAILED                                                                                                                [100%]

============================================================================ FAILURES ============================================================================
_________________________________________________________________________ test_operator __________________________________________________________________________

_handlers = '/Users/john.dewey/git/corelight-container-sensor-operator/test/../handlers.py'
_fixtures_dir = '/Users/john.dewey/git/corelight-container-sensor-operator/test/fixtures/kubernetes/manifests'

    @pytest.mark.slow
    def test_operator(_handlers, _fixtures_dir):
        # To prevent lengthy threads in the loop executor when the process exits.
        settings = kopf.OperatorSettings()
        # settings.watching.server_timeout = 10

        # Run an operator and simulate some activity with the operated resource.
        with kopf.testing.KopfRunner(
            [
                "run",
                "-n",
                _handlers,
                "--verbose",
            ],
            settings=settings,
        ) as runner:
            SENSOR.create_namespaced_custom_object("test", "default")
>           pass

test/test_handlers.py:85:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_kits/runner.py:113: in __exit__
    raise e
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/testing.py:408: in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/core.py:1053: in main
    rv = self.invoke(ctx)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/core.py:1659: in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/core.py:1395: in invoke
    return ctx.invoke(self.callback, **ctx.params)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/core.py:754: in invoke
    return __callback(*args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/cli.py:57: in wrapper
    return fn(*args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/decorators.py:84: in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/click/core.py:754: in invoke
    return __callback(*args, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/cli.py:104: in run
    return running.run(
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_core/reactor/running.py:58: in run
    loop.run_until_complete(operator(
../../.pyenv/versions/3.8.12/lib/python3.8/asyncio/base_events.py:616: in run_until_complete
    return future.result()
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_core/reactor/running.py:135: in operator
    await run_tasks(operator_tasks, ignored=existing_tasks)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_core/reactor/running.py:416: in run_tasks
    await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_cogs/aiokits/aiotasks.py:237: in reraise
    task.result()  # can raise the regular (non-cancellation) exceptions.
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_cogs/clients/scanning.py:43: in _read_old_api
    rsp = await api.get('/api', settings=settings, logger=logger)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_cogs/clients/api.py:111: in get
    response = await request(
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_cogs/clients/auth.py:45: in wrapper
    return await fn(*args, **kwargs, context=context)
../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/kopf/_cogs/clients/api.py:78: in request
    response = await context.session.request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <aiohttp.client.ClientSession object at 0x10b6d0640>, method = 'get', str_or_url = 'https://0.0.0.0:54600/api'

    async def _request(
        self,
        method: str,
        str_or_url: StrOrURL,
        *,
        params: Optional[Mapping[str, str]] = None,
        data: Any = None,
        json: Any = None,
        cookies: Optional[LooseCookies] = None,
        headers: Optional[LooseHeaders] = None,
        skip_auto_headers: Optional[Iterable[str]] = None,
        auth: Optional[BasicAuth] = None,
        allow_redirects: bool = True,
        max_redirects: int = 10,
        compress: Optional[str] = None,
        chunked: Optional[bool] = None,
        expect100: bool = False,
        raise_for_status: Optional[bool] = None,
        read_until_eof: bool = True,
        proxy: Optional[StrOrURL] = None,
        proxy_auth: Optional[BasicAuth] = None,
        timeout: Union[ClientTimeout, object] = sentinel,
        verify_ssl: Optional[bool] = None,
        fingerprint: Optional[bytes] = None,
        ssl_context: Optional[SSLContext] = None,
        ssl: Optional[Union[SSLContext, bool, Fingerprint]] = None,
        proxy_headers: Optional[LooseHeaders] = None,
        trace_request_ctx: Optional[SimpleNamespace] = None,
        read_bufsize: Optional[int] = None,
    ) -> ClientResponse:

        # NOTE: timeout clamps existing connect and read timeouts.  We cannot
        # set the default to None because we need to detect if the user wants
        # to use the existing timeouts by setting timeout to None.

        if self.closed:
>           raise RuntimeError("Session is closed")
E           RuntimeError: Session is closed

../../Library/Caches/pypoetry/virtualenvs/sensor-core-operator-dXLDOR8d-py3.8/lib/python3.8/site-packages/aiohttp/client.py:399: RuntimeError
----------------------------------------------------------------------- Captured log call ------------------------------------------------------------------------
DEBUG    kopf._core.reactor.running:running.py:499 Starting Kopf 1.35.3.
INFO     kopf._core.engines.activities:activities.py:81 Initial authentication has been initiated.
DEBUG    kopf.activities.authentication:execution.py:275 Activity 'login_via_pykube' is invoked.
WARNING  kopf._core.reactor.running:running.py:355 OS signals are ignored: running not in the main thread.
DEBUG    kopf.activities.authentication:piggybacking.py:125 Pykube is configured via kubeconfig file.
INFO     kopf.activities.authentication:execution.py:339 Activity 'login_via_pykube' succeeded.
INFO     kopf._core.engines.activities:activities.py:93 Initial authentication has finished.
DEBUG    urllib3.connectionpool:connectionpool.py:456 https://0.0.0.0:54600 "POST /apis/corelight.com/v1/namespaces/default/sensors HTTP/1.1" 201 438
INFO     kopf._core.reactor.running:running.py:449 Stop-flag is set to True. Operator is stopping.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Resource observer is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Admission validating configuration manager is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Credentials retriever is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Namespace observer is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Admission mutating configuration manager is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Poster of events is cancelled.
DEBUG    kopf._core.reactor.orchestration:aiotasks.py:192 Streaming tasks stopping is skipped: no tasks given.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Multidimensional multitasker is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Admission webhook server is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Admission insights chain is cancelled.
DEBUG    kopf._core.reactor.running:aiotasks.py:110 Daemon killer is cancelled.
ERROR    kopf._core.reactor.observation:api.py:92 Request attempt #1/9 failed; will retry: GET https://0.0.0.0:54600/apis -> ClientConnectionError('Connector is closed.')
ERROR    kopf._core.reactor.observation:api.py:92 Request attempt #1/9 failed; will retry: GET https://0.0.0.0:54600/api -> ClientConnectionError('Connector is closed.')
DEBUG    kopf._core.reactor.running:aiotasks.py:223 Root tasks are stopped: finishing normally; tasks left: set()
DEBUG    kopf._core.reactor.observation:api.py:76 Request attempt #2/9: GET https://0.0.0.0:54600/apis
DEBUG    kopf._core.reactor.observation:api.py:76 Request attempt #2/9: GET https://0.0.0.0:54600/api
DEBUG    kopf._core.reactor.running:aiotasks.py:192 Hung tasks stopping is skipped: no tasks given.
======================================================================= 1 failed in 2.70s ========================================================================

Code snippet is slightly diff now:

@pytest.mark.slow
def test_operator(_handlers, _fixtures_dir):
    settings = kopf.OperatorSettings()

    # Run an operator and simulate some activity with the operated resource.
    with kopf.testing.KopfRunner(
        [
            "run",
            "-n",
            _handlers,
            "--verbose",
        ],
        settings=settings,
    ) as runner:
        SENSOR.create_namespaced_custom_object("test", "default")

Where SENSOR above is a pykube-ng client trying to apply a CustomResource, and the Kubernetes URL in the stack trace https://0.0.0.0:57499/api is a k3d instance I am running locally.