cron job for trades: timeout and ping interval

[multidis]

Cron job execution (particularly in the ratio mode) for trades hangs if no response is received. Possible solution: introduce a reasonable timeout period (say 30 sec) for receiving a ping result. That way, if one of the queried exchanges is not responding, the loop can still continue with others.

Testing was done in a crude way by disabling networking while cron was running. The results were not very reproducible, but most commonly the script hangs beyond the scheduled cron time. Sometimes, reconnecting resumes execution in order-mode, but rarely in velocity mode.

Possible extension: since exchanges limit API call frequency, it would help to enforce a minimum time interval between pings of the same exchange (e.g. 1.1 sec). Perhaps this check can be combined with the maximum runtime check of a ping.

Suggestion summary/objectives:

• allow cron loop to proceed if one of the pings did not get a response nor error
• ensure ping calls are not violating exchange API call frequency (currently they can in some test cases)

[rosspalmer]

Added simple API limiter currently set to one ping per second.

Commit

I will start looking into the no response issue, I would probably want a much smaller timeout period but its a good idea.

[rosspalmer]

I've been really busy this week and will be on vacation all next. I will tackle this issue when I get back :)

[multidis]

Sure, it would be good to debug what is happening eventually. In principle, this can be dealt with by setting a timeout in the cron command that runs bitquant. That way the process will be terminated on time even if the code "hangs" without a response. The downside of course is no data written to the DB during that call. Have a good vacation!

[multidis]

For the ping interval: the following code queried coinbase 138 times in 50 seconds:

import bitquant as bq
c = bq.cron()
c.add_tjob('coinbase','btcusd',limit=100)
c.run(50, log='yes')

This is about twice the ping limit (a bit more). When the ping limit default is reduced significantly in api.py, exchange is queried less often. Overall, it appears that some ping interval limit is enforced in the current code, but the numerical value of pings/min is different from the setting.

[rosspalmer]

I still need to address the issue up top but I think the high ping rate while using bq directly was due to an outdated version on PyPi without the limiter. I just uploaded a new version, you should try an update and see if the same thing happens.

[multidis]

I was testing with the latest source using a virtualenv with pip install --upgrade -e git+https://github......git#egg=bitQuant, hoping to avoid PyPi delays (not sure about PyPi anyway, but fairly certain that pip command compiles the latest source). Will check again, thanks.

[multidis]

Did some more testing: the ping interval seems to be enforced in cron-order mode. Not fully sure about the velocity mode, but this issue probably can be closed for now.

[rosspalmer]

On a related note, I have been working the past week or so on a major restructure. All actions will be done by either a API, SQL, Data class. One of the benefits is that the API ping limiter is completely embedded in the API class so that all pings have the limiter applied automatically and each job can have custom ping limits.

Update should be out in a week.