search

openconfig / gnmic

gNMIc is a gNMI CLI client and collector
https://gnmic.openconfig.net
Apache License 2.0
170 stars 55 forks source link

gnmi-server not sending sync_response correctly in Subscribe mode = once #502

Closed pboers1988 closed 4 weeks ago

pboers1988 commented 1 month ago

When using the gnmi-server I'm attempting to create a rudimentary rest endpoint to retrieve interface information from the network through the gnmi-server. The FastAPI endpoint makes use of a pyGNMi client and looks like so.

Endpoint

@router.get("/{target}/interfaces")
def get_capabilities(target: str, gnmi_client: gNMIclient = Depends(yield_gnmi_client)) -> list[dict[Any, Any]]:
    subscription = {
        "subscription" : [{"path" : "/interfaces"}],
        "encoding": "json",
        "mode": "once"
    }
    subscribe_handle = gnmi_client.subscribe_once(subscription, target=target)
    res = []
    while True:
        try:
            res.append(next(subscribe_handle))
        except (StopIteration):
            break
    return res

Client

def yield_gnmi_client() -> Generator:
    with gNMIclient(("gnmi-server", 57400), insecure=True) as client:
        yield client
        client.close()

When calling the endpoint I receive the following strack trace

2024-08-07 19:37:38 [info     ] Application startup complete.  [uvicorn.error] 
Collecting Capabilities...
2024-08-07 19:37:43 [info     ] Collecting Capabilities...     [pygnmi.client] 
Collection of Capabilities is successfull
2024-08-07 19:37:43 [info     ] Collection of Capabilities is successfull [pygnmi.client] 
Unable to detect supported encodings, defaulting to 'json'
2024-08-07 19:37:43 [warning  ] Unable to detect supported encodings, defaulting to 'json' [pygnmi.client] 
Parsing of telemetry information is failed.
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1351, in telemetryParser
    element_str += element
TypeError: can only concatenate str (not "TypedValue") to str
2024-08-07 19:37:43 [error    ] Parsing of telemetry information is failed. [pygnmi.client] 
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1351, in telemetryParser
    element_str += element
TypeError: can only concatenate str (not "TypedValue") to str
Exception in thread Thread-5 (enqueue_updates):
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1080, in enqueue_updates
    raise error
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1076, in enqueue_updates
    for update in subscription:
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.CANCELLED
        details = "Channel closed!"
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Channel closed!", grpc_status:1, created_time:"2024-08-07T19:37:43.325609+02:00"}"
>
2024-08-07 19:37:43 [info     ] 127.0.0.1:65154 - "GET /******/capabilities HTTP/1.1" 500 [uvicorn.access] 
2024-08-07 19:37:43 [error    ] Exception in ASGI application
 [uvicorn.error] 
Traceback (most recent call last):
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 398, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/nwastdlib/logging.py", line 121, in __call__
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/fastapi/routing.py", line 193, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/starlette/concurrency.py", line 42, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/Documents/SURF/projects/gnmic-proxy/server/api/endpoints/interface.py", line 34, in get_capabilities
    res.append(next(subscribe_handle))
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/boers001/opt/miniconda3/envs/gnmic-proxy/lib/python3.11/site-packages/pygnmi/client.py", line 1146, in __next__
    if self._once and  "sync_response" in result:
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
WARNING:  StatReload detected changes in 'server/api/endpoints/interface.py'. Reloading...

This is where the stacktrace is generated in pyGNMI https://github.com/akarneliuk/pygnmi/blob/master/pygnmi/client.py#L1178 The last message in the stream is a None that does not contain the sync_response

It would seem to me as the gnmi-server is not sending the sync_response that the protocol and client expects. It just closes the socket. Would you agree this might be an implementation error in the gnmi-server code?

karimra commented 1 month ago

The sync_response: true is being sent: https://github.com/openconfig/gnmic/blob/main/pkg/app/gnmi_server.go#L372 I also took a pcap while running a test, I could see a sync_response msg sent.

pboers1988 commented 1 month ago

Thanks! Ill look into it further and try and figure out why the pyGNMi client lib is not parsing the response correctly then.