Handling hung processes

[stephenrs]

Hi,

I think I have a pretty unique situation. For some reason, occasionally (unpredictable, unreproducible), processes will hang when trying to request data from IQFeed - even when it is still possible to connect to all the IQFeed ports, as with the _is_iqfeed_running() function. An example of a hanging function call is the following:

bars = conn.request_bars_in_period(ticker=symbol,
                                                    interval_len=bar_len,
                                                    interval_type='s',
                                                    bgn_prd=start,
                                                    end_prd=end,
                                                    bgn_flt=start_time,
                                                    end_flt=end_time,
                                                    ascend=True,
                                                    timeout=5)

As you can see, I've included a timeout, but the call still sometimes hangs indefinitely. It's not clear whether this is something going wrong beyond my reach (wine, iqconnect.exe, Xvfb, OS, etc), or if it's something in pyiqfeed. I'm considering handing the problem by spawning a process to call the function (using pathos/multiprocessing), so that I can detect a hung condition and react accordingly, but I'm wondering if you have ever run into something like this, or if you have any better suggestions.

Thanks!

EDIT: I'm running this in headless mode on CentOS 7, but couldn't get xvfb_run to work properly via the service.py module, so am starting Xvfb externally, and setting the DISPLAY environment variable explicitly when wine/iqconnect is started.

[akapur]

Are you using the context managers to initiate and terminate connections as in example.py? If not and you don't connect disconnect properly, your processes won't close connections to Iqfeed cleanly and iqfeed doesn't like that? If you are, make sure its a recent wine and iqfeed and you haven't made any changes to the code. If still nothing it's likely something to do with how networking is setup or wine. Iqfeed times out and exits after a short period (configurable) of inactivity or if you launch it on a different machine. If you try to connect during the shutdown process you will stop the shutdown but Iqfeed can't send you data or anything else because it's destroyed needed data structures and you'll get a hang.

Sent from my iPad

On Jul 4, 2017, at 5:01 PM, stephenrs notifications@github.com wrote:

Hi,

I think I have a pretty unique situation. For some reason, occasionally (unpredictable, unreproducible), processes will hang when trying to request data from IQFeed - even when it is still possible to connect to all the IQFeed ports, as with the _is_iqfeed_running() function. An example of a hanging function call is the following:

bars = conn.request_bars_in_period(ticker=symbol, interval_len=bar_len, interval_type='s', bgn_prd=start, end_prd=end, bgn_flt=start_time, end_flt=end_time, ascend=True, timeout=5) As you can see, I've included a timeout, but the call still sometimes hangs indefinitely. It's not clear whether this is something going wrong beyond my reach (wine, iqconnect.exe, Xvfb, OS, etc), or if it's something in pyiqfeed. I'm considering handing the problem by spawning a process to call the function (using pathos/multiprocessing), so that I can detect a hung condition and react accordingly, but I'm wondering if you have ever run into something like this, or if you have any better suggestions.

Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[stephenrs]

I'm not using the context managers (there are cases where I don't want to close connections after each history fetch, for example, because shutdown is slow), but I am paying close attention in my code to make sure that connections are closed properly. Of course there's a chance that I'm missing something and somehow leaving connections open in some scenarios, but I'm not seeing any evidence of that. Also, I'm using a recent wine (although I've hammered 32bit wine onto a 64bit box), and have worked a bit with the iqconnect client, so know some of its "habits". I'll keep my eyes open for the next time it happens, and see if I can get more diagnostic info.

I'm seeing a perhaps related issue though...After some time running, a QuoteConn watch_trades() connection will stop working, although _is_iqfeed_running() still returns True, and it is still possible to fetch historical intervals/bars. When trying to open a new watch connection, the usual verbose messages come through, but then I get "Feed Disconnected" and nothing more - no updates follow.

From looking at some of the pyiqfeed code, I see that there's some reconnection logic, but I never see any evidence of it happening in the logging messages - for example, I never see "Feed Reconnect Failed". The only way to get watching to work again is to restart iqconnect.

Does this sound familiar? Any thoughts?

Thanks again for your work on this project, and your help with the issues I'm finding.

[stephenrs]

I just want to also note that I'm building a "heartbeat" service that monitors for any bad conditions that I have identified, and responds (usually by restarting iqconnect), but it would be nice to be able to address the root causes of the problems that have surfaced, if possible.

[akapur]

You are literally the only person who has reported that he has this problem. So if there is a "root cause", it’s likely not in the library. I use the library pretty extensively myself and have no such issues so long as the library is used as intended as described in the docs and the previous email. If you find a bug in the library, I would really appreciate a fix and a pull request.

If you find a heartbeat service necessary and you develop the heartbeat service, let me know. If it’s useful I’ll be happy to link to it from the pyiqfeed README file.

Regards.

Ashwin

On Jul 6, 2017, at 1:43 PM, stephenrs notifications@github.com wrote:

I just want to also note that I'm building a "heartbeat" service that monitors for any bad conditions that I have identified, and responds (usually by restarting iqconnect), but it would be nice to be able to address the root causes of the problems that have surfaced, if possible.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/akapur/pyiqfeed/issues/21#issuecomment-313468480, or mute the thread https://github.com/notifications/unsubscribe-auth/AAyQVMYio1nRq7gCVNzEh76bPVC60rTsks5sLRzXgaJpZM4ONrrB.

[stephenrs]

Every issue has a first person to report it ;)

I didn't mean to suggest that there are problems with the library, but given your experience with iqconnect, I just wanted to know if any of my symptoms sounded familiar to you. I agree, the root cause is likely lower down in the system, not in the python code - my first suspicion is something going on with 32-bit wine (needed for IQFeed) running on my 64-bit system. Shame there isn't a 64-bit version, or 64-bit wine could run 32-bit apps.

Also, I have pretty high confidence that I'm using the library properly. For example, I checked the context manager code to make sure that I wasn't missing anything (all they do is connect/disconnect, and I've confirmed that I'm doing that at the right times). And since the verbose listeners seem pretty comprehensive in what conditions they can alert to, it seems probable that something is happening inside the wine container that pyiqfeed/DTN doesn't know about at the ends of the chain.

I've built a first iteration of a heartbeat service...all it does is check periodically to make sure ports are working (_is_iqfeed_running), and it makes sure it can fetch historical data. Now I just need to add something to check whether watching is still receiving data.

A heartbeat is not ideal, but fine for my purposes, particularly since I'm running out of ideas regarding what the root cause could be.

[akapur]

If you’d like to send me some code to run to see if I get hangs on my system, I’d be happy to try it out. I would prefer the smallest bit of code you can send, ideally only stuff that exercises pyiqfeed.

On Jul 6, 2017, at 7:41 PM, stephenrs notifications@github.com wrote:

Every issue has a first person to report it ;)

I didn't mean to suggest that there are problems with the library, but given your experience with iqconnect, I just wanted to know if any of my symptoms sounded familiar to you. I agree, the root cause is likely lower down in the system, not in the python code - my first suspicion is something going on with 32-bit wine (needed for IQFeed) running on my 64-bit system. Shame there isn't a 64-bit version, or 64-bit wine could run 32-bit apps.

Also, I have pretty high confidence that I'm using the library properly. For example, I checked the context manager code to make sure that I wasn't missing anything (all they do is connect/disconnect, and I've confirmed that I'm doing that at the right times). And since the verbose listeners seem pretty comprehensive in what conditions they can alert to, it seems probable that something is happening inside the wine container that pyiqfeed/DTN doesn't know about at the ends of the chain.

I've built a first iteration of a heartbeat service...all it does is check periodically to make sure ports are working (_is_iqfeed_running), and it makes sure it can fetch historical data. Now I just need to add something to check whether watching is still receiving data.

A heartbeat is not ideal, but fine for my purposes, particularly since I'm running out of ideas regarding what the root cause could be.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/akapur/pyiqfeed/issues/21#issuecomment-313548572, or mute the thread https://github.com/notifications/unsubscribe-auth/AAyQVCkJdyRl4nEuW8dNpdJpw2KJtihrks5sLXCwgaJpZM4ONrrB.

[stephenrs]

My sincere thanks for your offer, but I don't think that would actually be a good use of either of our time - the problem only manifests over hours or days (even when being used very lightly), so I'm pretty sure it's something system-dependent, rather than something you could see in the code or reproduce with a test case alone.

As a last step before I fallback to the heartbeat as "the solution" and I move on, right now I'm rebuilding wine using this approach: https://www.systutorials.com/239913/install-32-bit-wine-1-8-centos-7/ (which didn't exist when I first hacked wine onto CentOS 7).

Other than this issue, things are working great, and it's been a pleasure to work with pyiqfeed, so thanks again, and wish me luck!