Incorrect ohlc on fetching during live trading

[dimnorin]

While running live trading ohlc data incorrectly pushed to lines.

Please view the image. The left part is prefetched data, the right part - live data.

[JiazhengChai]

Hi @dimnorin, it works great with your fix. I also had the same issue and with your fix, now it works. However, when I resample data into other timeframes, the following errors occur.

It is actually related to bug #17 and it could be fixed by adding the following codes in ccxtfeed.py -> _load as suggested by @LatchRazu :

  start = datetime.utcnow()

   if start.second < 15:
        time.sleep(15 - start.second)

  self._fetch_ohlcv()

Details can be found in issue #17. Unfortunately, the above fix is not compatible with your fix. When I combined both, the min() arg error occurs. Do you have any idea why it is not compatible with your fix?

[LatchRazu]

This is only an issue on certain exchanges right? I run on bitmex and get all the ohlcv data.

@dimnorin I think your fix might be redundant. If I'm not mistaken ccxtfeed's 'drop_newest' param is supposed to fix this

@JiazhengChai This is a well known issue with backtrader. Any manipulation of the datafeed tends to screw things up when resampling data. (activating memory savings with resampled data causes the same problem)

You can prevent the crash by adding this piece of code in Cerebro but I haven't tested if it screws up your data.

        for d in datas:
            qlapse = datetime.datetime.utcnow() - qstart
            d.do_qcheck(newqcheck, qlapse.total_seconds())
            drets.append(d.next(ticks=False))

        # CRASH PREVENT
        if drets[0] is None and any(drets[1:]):
            self.log.critical('resampling  crash prevented')
            for idx, dr in enumerate(drets):
                if dr:
                    drets[idx] = False
        # CRASH PREVENT

        d0ret = any((dret for dret in drets))
        if not d0ret and any((dret is None for dret in drets)):
            d0ret = None

So try out if this fix and see if your data get scrambled or not. If it does, I'd suggest moving to an exchange which doesn't provide incomplete candles.

I'll try debugging it for a bit but it's an absolute nightmare so don't get your hopes up.

[JiazhengChai]

@LatchRazu I see, so it is an exchange-based issue? I didn't know that. I am currently trying Bybit but so far encountering quite a few issues.

Yeah, I agree with the resampling problem when saving memory.

So your crash prevention fix is to set all other timeframe's dret to False when the base timeframe value is None, am I right? Thanks for your help. I appreciate it.

[LatchRazu]

I've been trading/testing on bitmex for months and after adding the sleep fix and adding extensive error handling in the retry method, I haven't experienced any crashes or issues from the ccxt store.