Feed returns wrong ohlcv values and makes many `fetch_ohlcv()` requests to the exchange

[sandroboehme]

1. As reproduceable in this gist the CCXTFeed continuously fetches ohlcv data from the exchange and thus consumes additional downstream data volume. Is that really needed for something or is that an error?
2. Sometimes the same value for ohlc is returned which is most probably wrong. The output from the above gist looks like that in this case:

   ---- NEW REQUEST ----
   2019-03-12 19:10:03.585799 - Requesting: Since TS 1552417740000 Since date 2019-03-12 19:09:00 granularity 1m, limit 2, params
   2019-03-12 19:10:04.374471 - Data 0: 2019-03-12 19:09:00 - TS 1552417740000 Time 1552417804374.4797
   2019-03-12 19:10:04.374536 - Data 1: 2019-03-12 19:10:00 - TS 1552417800000 Time 1552417804374.5388
   ---- REQUEST END ----
   ----     LOAD    ----
   2019-03-12 19:10:04.374660 Load OHLCV Returning: True
   2019-03-12T19:10:00 open: 15.0262
   2019-03-12T19:10:00 high: 15.0262
   2019-03-12T19:10:00 low: 15.0262
   2019-03-12T19:10:00 close: 15.0262

   When I set a conditional breakpoint in binance.py:fetch_ohlcv() after publicGetKlines() I can inspect the variables for this scenario but it's still not clear to me where that comes from. The breakpoint condition I use is: len(response) == 2 and response[1][1] == response[1][2] and response[1][2] == response[1][3] and response[1][3] == response[1][4]

To make the gist work the code from the #5 pull request needs to be used to avoid needing a Binance API key.

[sandroboehme]

It's actually most of the time that the ohlc data has the same value.

[sandroboehme]

Any hints about why this might happen is highly appreciated!

[Dave-Vallance]

@sandroboehme

Because we are using a rest API, the exchange is constantly polled for new data at the exchanges rate limit. It will attempt this every few seconds.

So yes, it is a little wasteful but trying to doe it once per minute (or whatever timeframe you are using) is a little problematic. First of all, you would need to sychronize to the minute, hour or whatever depending on the timeframe. Next, the exchanges tend not to update their Rest API right on the second. For example, a new bar might only be returned from the API 4-5 seconds after the bar has closed.

Using the exchanges Websocket API would be the resolution for this.

Let me know if I missed the point!

[Dave-Vallance]

Closing for housekeeping as there has not been a response for a while. Please re-open it if needed!