Thread exception when calling close on client after watcher is started

[babueter]

Description: The callback thread throws an exception when calling close on client if a watch has been started. Even if the watch is canceled the thread remains and throws a ValueError at etcd3/watch.py line 126.

Code to reproduce:

import etcd3

client = etcd3.client()
watch_id = client.watch_prefix('/')
client.cancel_watch(watch_id)

client.close()

Error message:

Exception in thread etcd3_watch_7f99646ef9b0:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/etcd3/watch.py", line 126, in _run
    metadata=self._metadata)
  File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 1084, in __call__
    event_handler, self._context)
  File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 1171, in create
    operationses_and_tags, context)
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 478, in grpc._cython.cygrpc.Channel.integrated_call
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 296, in grpc._cython.cygrpc._integrated_call
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 222, in grpc._cython.cygrpc._call
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 262, in grpc._cython.cygrpc._call
ValueError: Cannot invoke RPC: Channel closed!

python version: Python 3.6.9

pip freeze: asn1crypto==0.24.0 coverage==5.0.3 crc32c==2.0 cryptography==2.1.4 etcd3==0.11.1 grpcio==1.26.0 idna==2.6 keyring==10.6.0 keyrings.alt==3.0 numpy==1.18.1 protobuf==3.11.2 pycrypto==2.6.1 pygobject==3.26.1 pyxdg==0.25 PyYAML==5.3 rdbd==0.0.1 SecretStorage==2.3.1 six==1.13.0 tenacity==6.0.0

etcd --version: etcd Version: 3.2.17 Git SHA: Not provided (use ./build instead of go build) Go Version: go1.10 Go OS/Arch: linux/amd64

operating system: Docker version 19.03.5, build 633a0ea

Dockerfile FROM ubuntu:18.04

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes \ etcd \ python3 \ python3-pip \ supervisor && \ mkdir /python

ENV DEBUG true ENV ETCD_NAME provider ENV ETCD_DATA_DIR /var/lib/etcd ENV ETCD_INITIAL_CLUSTER_STATE new ENV ETCD_INITIAL_CLUSTER_TOKEN etcd-cluster-01 ENV ETCDCTL_API=3 ENV CLUSTER_SEED_NODES node1,node2,node3

etcd_start.sh

#!/bin/bash

set -x
CURRENT_IP=$(hostname -i)
CLUSTER_SEED_NODES=${CLUSTER_SEED_NODES:-localhost}

INITIAL_CLUSTER=""
for NODE in $(echo ${CLUSTER_SEED_NODES} | sed 's/,/ /g'); do
    NODE_ENTRY="${NODE}=http://${NODE}:2380"
    if [[ -z "${INITIAL_CLUSTER}" ]]; then
        INITIAL_CLUSTER="${NODE_ENTRY}"
    else
        INITIAL_CLUSTER="${INITIAL_CLUSTER},${NODE_ENTRY}"
    fi
done

/usr/bin/etcd \
    --name $(hostname) \
    --initial-advertise-peer-urls "http://${CURRENT_IP}:2380" \
    --listen-peer-urls "http://${CURRENT_IP}:2380" \
    --listen-client-urls "http://${CURRENT_IP}:2379,http://127.0.0.1:2379" \
    --advertise-client-urls "http://${CURRENT_IP}:2379" \
    --initial-cluster-token etcd-cluster \
    --initial-cluster ${INITIAL_CLUSTER}

[achimnol]

Same error here. There is no way to remove this exception using the public APIs. There is no callback_thread.join() and the Watcher._run() method seems to have no terminating conditions.

[achimnol]

After using an watcher, the following shutdown code can reliably reproduce the issue. Otherwise, abrupt interruption of the callback thread may print garbled exception tracebacks.

etcd.close()
if etcd.watcher._callback_thread:
    etcd.watcher._callback_thread.join()

[babueter]

See PR: https://github.com/kragniz/python-etcd3/pull/1223