Etcd client raises a ReadTimeoutError traceback

[naveenjoy]

When the etcd client times out its watch, it is generating a traceback as seen below. In our code we are catching the TimeoutError exception. However, the etcd client is logging the exception and as a result a stack trace is printed in our stdout confusing our users.

Please fix this bug to either a) not log the exception in your code below or b) to catch and handle it.

except: _log.exception("Unexpected request failure, re-raising.") <----- offending line raise

_Traceback seen in our console ERROR etcd.client ReadTimeoutError: HTTPConnectionPool(host='X.X.X.X', port=2379): Read timed out. 2016-10-03 11:38:14.399 31774 ERROR etcd.client 2016-10-03 11:38:14.401 31774 DEBUG etcd.client [-] Issuing read for key /networking-vpp/nodes/sjo-smf-ubuntu-server-2/ports with args {'waitIndex': 485299, 'recursive': True, 'timeout': 60, 'wait': True} read /usr/local/lib/python2.7/dist-packages/etcd/client.py:521 2016-10-03 11:39:14.438 31774 ERROR etcd.client [-] Unexpected request failure, re-raising. 2016-10-03 11:39:14.438 31774 ERROR etcd.client Traceback (most recent call last): 2016-10-03 11:39:14.438 31774 ERROR etcd.client File "/usr/local/lib/python2.7/dist-packages/etcd/client.py", line 802, in wrapper 2016-10-03 11:39:14.438 31774 ERROR etcd.client = response.data 2016-10-03 11:39:14.438 31774 ERROR etcd.client File "/usr/local/lib/python2.7/dist-packages/urllib3/response.py", line 166, in data 2016-10-03 11:39:14.438 31774 ERROR etcd.client return self.read(cache_content=True) 2016-10-03 11:39:14.438 31774 ERROR etcd.client File "/usr/local/lib/python2.7/dist-packages/urllib3/response.py", line 311, in read 2016-10-03 11:39:14.438 31774 ERROR etcd.client flush_decoder = True 2016-10-03 11:39:14.438 31774 ERROR etcd.client File "/usr/lib/python2.7/contextlib.py", line 35, in exit 2016-10-03 11:39:14.438 31774 ERROR etcd.client self.gen.throw(type, value, traceback) 2016-10-03 11:39:14.438 31774 ERROR etcd.client File "/usr/local/lib/python2.7/dist-packages/urllib3/response.py", line 231, in _error_catcher 2016-10-03 11:39:14.438 31774 ERROR etcd.client raise ReadTimeoutError(self._pool, None, 'Read timed out.') 2016-10-03 11:39:14.438 31774 ERROR etcd.client ReadTimeoutError: HTTPConnectionPool(host='192.168.119.42', port=2379): Read timed out. 2016-10-03 11:39:14.438 31774 ERROR etcd.client

[lavagetto]

Without seeing your code, it seems like this should be nonetheless solved in the latest versions, but it's hard to tell. The code is doing exactly what I wanted, btw: exposing an exception so that people report those to us :)

If you want our help please report:

1) A code snippet to reproduce the problem in a controlled environment 2) The precise procedure to reproduce the bug, step by step 3) The version of the library, urllib3, and etcd you're using.

[lavagetto]

On second thoughts: it seems like your problem is happening after you got a response from the etcd server and urllib is trying to fetch the data. Specifically here:

https://github.com/jplana/python-etcd/blob/master/src/etcd/client.py#L863

so it seems you exceed your read timeout while trying to get your data. How big is the content of that key? It seems like a race condition to me.

[iawells]

From what I can tell, this is the thing I just issued a PR for - not that there's a problem with the exception, but that python-etcd thinks that it should issue ERROR-level logs when an exception flies past (despite this being the correct behaviour for python-etcd - albeit maybe an error for the app using python-etcd, but that's for it to determine). See #239.