gRPC error message during subscription

ianbarrere commented 1 month ago

Hello,

I am setting up a subscription poller using pygnmi in the following setup:

def subscribe(hostname: str, paths: List[str]):
    subscription_def = {
        "subscription": [
            {
                "path": path,
                "mode": "sample",
            }
            for path in paths
        ],
        "mode": "poll",
        "encoding": "json_ietf",
    }
    with get_gnmi_session(hostname) as session:
        subscription = session.subscribe2(subscribe=subscription_def)
        yield subscription

I noticed that every time I poll a device, if I advance to the end of the iteration (by either wrapping the generator in next() or by looping over it) it throws a rather generic looking gRPC error:

Exception in thread Thread-4 (enqueue_updates):
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/i/venv/lib/python3.11/site-packages/pygnmi/client.py", line 997, in enqueue_updates
    for update in subscription:
  File "/Users/i/venv/lib/python3.11/site-packages/grpc/_channel.py", line 540, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/Users/i/venv/lib/python3.11/site-packages/grpc/_channel.py", line 966, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.CANCELLED
        details = "Channel closed!"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-09-13T14:46:04.927971+02:00", grpc_status:1, grpc_message:"Channel closed!"}"

This error doesn't seem to disrupt the behavior of the poll, as I'm still able to get the data I'm looking for, but it seems to be hit when the session is closed.

I'm not sure if I'm using the tool wrong or if this is a known issue or something, but I just wanted to see if anybody else had run into it.

jjochum commented 1 month ago

We encountered the same issue when the session is closed. I modified client.py:1079 to work around that. Most likely, the issue is with the grpc usage, but I didn't investigate further.

        except Exception as error:
              self.error = error
              # The connection was terminated by the server. This is generally okay and shouldn't raise an exception.
              if isinstance(error, grpc._channel._MultiThreadedRendezvous) and error.code() == grpc.StatusCode.CANCELLED:
                  return
              raise error

akarneliuk commented 1 month ago

Hey @ianbarrere , Yes, GRPC library internally creates a thread and doesn't seem it is joining properly. The proposal by @jjochum seems as interesting workaround indeed. I think I can add it to the code.

akarneliuk commented 1 day ago

Fixed in #161 , thanks @jjochum

akarneliuk / pygnmi

gRPC error message during subscription #159