projectcalico / networking-calico

Calico integration with OpenStack (Neutron)
7 stars 15 forks source link

etcdv3.py: Do not subclass etcd3gw.watch.Watcher #58

Closed tj90241 closed 3 years ago

tj90241 commented 3 years ago

etcd3gw>=0.2.2 incorporates the fix that the subclassing in this module itself was doing. However, as of new etcd3gw, this now results in an attempt to close the same file descriptor twice when stop() is invoked, which itself results in an exception that can result in log spew, as seen below:

(WARNING) etcdv3: Failed to force-close etcd3gw's watch socket: AttributeError("'HTTPResponse' object has no attribute '_fileno'",)

Pin etcd3gw in the requirements so that the version with the fix is used, and remove the fix incorporated in this module.

tj90241 commented 3 years ago

Specifically see this commit, which landed inetcd3gw=0.2.2: https://opendev.org/openstack/etcd3gw/commit/797cb2e673b9cd08c17ede032af3582b9fbb9dc9

and this code: https://github.com/projectcalico/networking-calico/blob/7be5aaba8ad1d4c5d5464a9380e07efcf28b9375/networking_calico/etcdv3.py#L43

As an aside: I'm trying to establish why we started seeing a vast increase in the frequency of etcd3gw.watch.Watcher.stop() invocations... this might be it, but while I'm investigating, here's some interim cleanup that can be done.

tj90241 commented 3 years ago

Actually, on closer look, the aforementioned function had a similar hit-rate/frequency in our older deployments; it's just that we started seeing the log-churn as a result of some python3-ism or some change in a newer networking-calico.

tl;dr: This tests well for us and does address the chatty logs that we were seeing.

nelljerram commented 3 years ago

/sem-approve