Exception in conn.py cannot be properly handled

[stephenrs]

Hi there,

I've been using pyiqfeed for quite a while, and it has been rock solid, so thanks!

Recently I've discovered that occasionally an exception can be triggered in FeedConn._read_messages(), but as far as I can tell, the exception can't be handled outside of the pyiqfeed layer, so it causes a fatal error.

The exception seems to be triggered when iqconnect dies a violent and untimely death, and here's the stack trace (this might not be the latest version of pyiqfeed):

Exception in thread iqfeed_admin-reader:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "<pvt>/pyiqfeed/conn.py", line 200, in __call__
if self._read_messages():
File "<pvt>/pyiqfeed/conn.py", line 211, in _read_messages
data_recvd = self._sock.recv(1024).decode('latin-1')
File "<pvt>/python3.6/site-packages/gevent/_socketcommon.py", line 657, in recv
return self._sock.recv(*args)
ConnectionResetError: [Errno 104] Connection reset by peer

For some context: the reason this affects my use case is that my system is pretty resilient with regard to keeping itself running and reconnecting to iqconnect as needed under various unexpected circumstances. But, when this exception is thrown, I haven't been able to think of an elegant way to handle it.

So, do you have any suggestions, or would you consider wrapping that code in such a way that consumers of pyiqfeed can have a chance to handle exceptions that happen deep in the library?

[akapur]

Happy to accept a pull request. Please read the Readme before you start coding. Please add type annotations and document your code.

The reading is done in a separate thread so you would need to look at that piece, possibly start the reader thread yourself, catch exceptions and then disconnect cleanly, reset internal state and then reconnect when that happens. You may need to test that iqfeed.exe has restarted and can accept connections before you reconnect. If you look at service.py, you could maybe even write some sort of watchdog process that restarts iqfeed.exe when it dies by modifying it.

On Fri, May 14, 2021 at 7:40 PM stephenrs @.***> wrote:

Hi there,

I've been using pyiqfeed for quite a while, and it has been rock solid, so thanks!

Recently I've discovered that occasionally an exception can be triggered in FeedConn._read_messages(), but as far as I can tell, the exception can't be handled outside of the pyiqfeed layer, so it causes a fatal error.

The exception seems to be triggered when iqconnect dies a violent and untimely death, and happens on this line:

data_recvd = self._sock.recv(1024).decode('latin-1')

Here's the stack trace (this might not be the latest version of pyiqfeed):

Exception in thread iqfeed_admin-reader: Traceback (most recent call last): File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, *self._kwargs) File "/pyiqfeed/conn.py", line 200, in call if self._read_messages(): File "/pyiqfeed/conn.py", line 211, in _read_messages data_recvd = self._sock.recv(1024).decode('latin-1') File "/python3.6/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(args) ConnectionResetError: [Errno 104] Connection reset by peer

For some context: the reason this affects my use case is that my system is pretty resilient with regard to keeping itself running and reconnecting to iqconnect as needed under various unexpected circumstances. But, when this exception is thrown, I haven't been able to think of a way to handle it.

So, do you have any suggestions, or would you consider wrapping that code in such a way that consumers of pyiqfeed can have a chance to handle exceptions that happen deep in the library?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/akapur/pyiqfeed/issues/53, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJAVABBGOOJJWVD24XUJLTNWYIJANCNFSM445IMCCQ .

[stephenrs]

@akapur Thanks for your reply, and although I would be honored to contribute to this project, the current demands on my time unfortunately don't really leave room for it.

On the brighter side, after deeper investigation, I discovered that the exception wasn't actually being triggered by something going wrong with iqconnect. I'm able to reproduce the problem from my app by forcing a connection to die on the receiving end of the socket. Connections periodically dying is a normal event in the app, so I just needed to be more careful about how and when they die, in order to mitigate the issue.

So, thanks again, and keep up the great work.

[akapur]

Not a problem with iqfeed.