Incorrect candle data

[ninadpachpute]

Hi, I got the bot running with no hassle. But before everything else, I was checking the 1m datafeed. Below is my strategy which only prints ohlcv data.

After running data feed for half a day; starting @ 25-05-19 09:20:00 I started getting the datafeed such that all ohlc values are exactly same for many candles if not all.

I then turned the bot off and started back on which getting 15m worth 1m data on restart. On restart I see correct data.

For example @ 25-05-19 09:29:00, see data in 1st run and the 2nd run. Data from 2nd run (restart) is correct.

The problem is I don't even know where to start debugging this. Any pointers?

class Basic01(bt.Strategy):

    def __init__(self):
        # Keep a reference to the "close" line in the data[0] dataseries
        self.o = self.datas[0].open
        self.h = self.datas[0].high
        self.l = self.datas[0].low
        self.c = self.datas[0].close
        self.v = self.datas[0].volume

    def log(self, txt, send_telegram=False, color=None):
        dt = self.datas[0].datetime.datetime(0)
        print('%s : %s' % (dt.strftime("%d-%m-%y %H:%M:%S"), txt))

    def next(self):
        # Simply log the closing price of the series from the reference
        self.log('O:%.8f H:%.8f L:%.8f C:%.8f V:%.8f' %
                 (self.o[0], self.h[0], self.l[0], self.c[0], self.v[0]))

25-05-19 09:01:00 : O:8016.44000000 H:8017.72000000 L:8005.37000000 C:8013.84000000 V:26.76449600 25-05-19 09:02:00 : O:8003.23000000 H:8005.10000000 L:8003.23000000 C:8005.02000000 V:0.21315400 25-05-19 09:03:00 : O:7998.50000000 H:7998.50000000 L:7998.50000000 C:7998.50000000 V:0.01875000 25-05-19 09:04:00 : O:8000.08000000 H:8000.08000000 L:8000.08000000 C:8000.08000000 V:0.02967000 25-05-19 09:05:00 : O:7993.50000000 H:7993.50000000 L:7993.50000000 C:7993.50000000 V:0.01826500 25-05-19 09:06:00 : O:7988.68000000 H:7988.68000000 L:7988.68000000 C:7988.68000000 V:0.02487400 25-05-19 09:07:00 : O:7995.38000000 H:7995.38000000 L:7995.38000000 C:7995.38000000 V:0.00500000 25-05-19 09:08:00 : O:7995.67000000 H:7995.67000000 L:7995.67000000 C:7995.67000000 V:0.00183700 25-05-19 09:09:00 : O:7991.58000000 H:7991.58000000 L:7991.58000000 C:7991.58000000 V:0.05152600 25-05-19 09:10:00 : O:7988.93000000 H:7990.84000000 L:7988.87000000 C:7990.84000000 V:0.16319400 25-05-19 09:11:00 : O:7986.82000000 H:7986.82000000 L:7985.05000000 C:7985.05000000 V:1.01830900 25-05-19 09:12:00 : O:7982.56000000 H:7982.56000000 L:7982.56000000 C:7982.56000000 V:0.04744300 25-05-19 09:13:00 : O:7983.98000000 H:7983.98000000 L:7983.98000000 C:7983.98000000 V:0.15108800 25-05-19 09:14:00 : O:7997.36000000 H:7997.36000000 L:7997.36000000 C:7997.36000000 V:0.00898900 25-05-19 09:15:00 : O:7993.19000000 H:7993.19000000 L:7993.19000000 C:7993.19000000 V:0.03065000 25-05-19 09:16:00 : O:7992.53000000 H:7992.53000000 L:7992.53000000 C:7992.53000000 V:0.00853100 25-05-19 09:17:00 : O:7984.85000000 H:7984.85000000 L:7984.83000000 C:7984.85000000 V:0.22422500 25-05-19 09:18:00 : O:7988.74000000 H:7988.74000000 L:7988.74000000 C:7988.74000000 V:0.10418400 25-05-19 09:19:00 : O:7994.00000000 H:7995.71000000 L:7994.00000000 C:7994.43000000 V:0.54700800 25-05-19 09:20:00 : O:7994.98000000 H:7994.98000000 L:7994.98000000 C:7994.98000000 V:0.00149400 25-05-19 09:21:00 : O:7993.01000000 H:7993.01000000 L:7993.01000000 C:7993.01000000 V:0.00294500 25-05-19 09:22:00 : O:7995.16000000 H:7996.13000000 L:7995.14000000 C:7995.14000000 V:0.78172200 25-05-19 09:23:00 : O:7994.89000000 H:7994.89000000 L:7994.89000000 C:7994.89000000 V:0.00149800 25-05-19 09:24:00 : O:7987.08000000 H:7987.79000000 L:7987.08000000 C:7987.79000000 V:0.48928000 25-05-19 09:25:00 : O:7986.74000000 H:7986.74000000 L:7986.74000000 C:7986.74000000 V:0.04167200 25-05-19 09:26:00 : O:7986.10000000 H:7986.10000000 L:7986.10000000 C:7986.10000000 V:0.00469700 25-05-19 09:27:00 : O:7981.91000000 H:7981.91000000 L:7981.91000000 C:7981.91000000 V:0.07297500 25-05-19 09:28:00 : O:7987.57000000 H:7987.57000000 L:7987.57000000 C:7987.57000000 V:0.09389700 25-05-19 09:29:00 : O:7980.57000000 H:7980.57000000 L:7980.57000000 C:7980.57000000 V:0.12323600 finished.

C:\Users\521876\Documents\personal\bot>python main.py ENV = production Starting Portfolio Value: 574.62 25-05-19 09:15:00 : O:7993.19000000 H:7995.59000000 L:7986.00000000 C:7992.54000000 V:22.08483100 25-05-19 09:16:00 : O:7992.53000000 H:7995.08000000 L:7982.43000000 C:7984.86000000 V:21.74773600 25-05-19 09:17:00 : O:7984.85000000 H:7992.11000000 L:7983.26000000 C:7988.00000000 V:18.14793700 25-05-19 09:18:00 : O:7988.74000000 H:7995.70000000 L:7985.57000000 C:7993.53000000 V:20.94795900 25-05-19 09:19:00 : O:7994.00000000 H:7997.34000000 L:7991.02000000 C:7994.66000000 V:14.31850900 25-05-19 09:20:00 : O:7994.98000000 H:7994.98000000 L:7988.81000000 C:7990.53000000 V:13.49096400 25-05-19 09:21:00 : O:7993.01000000 H:7996.55000000 L:7991.14000000 C:7994.94000000 V:24.40371700 25-05-19 09:22:00 : O:7995.16000000 H:7999.00000000 L:7991.45000000 C:7993.78000000 V:23.31974600 25-05-19 09:23:00 : O:7994.89000000 H:7994.89000000 L:7985.00000000 C:7987.06000000 V:35.50745000 25-05-19 09:24:00 : O:7987.08000000 H:7989.78000000 L:7985.21000000 C:7988.02000000 V:18.20830500 25-05-19 09:25:00 : O:7986.74000000 H:7988.52000000 L:7981.21000000 C:7986.12000000 V:18.09139700 25-05-19 09:26:00 : O:7986.10000000 H:7987.90000000 L:7981.91000000 C:7981.92000000 V:17.93937100 25-05-19 09:27:00 : O:7981.91000000 H:7993.80000000 L:7977.02000000 C:7992.16000000 V:80.66655800 25-05-19 09:28:00 : O:7987.57000000 H:7989.54000000 L:7978.01000000 C:7978.21000000 V:16.02878300 25-05-19 09:29:00 : O:7980.57000000 H:7981.62000000 L:7978.68000000 C:7980.55000000 V:3.99554100 25-05-19 09:30:00 : O:7977.41000000 H:7977.98000000 L:7977.41000000 C:7977.98000000 V:0.19824800 25-05-19 09:31:00 : O:7978.00000000 H:7984.00000000 L:7976.27000000 C:7980.06000000 V:20.46780000 25-05-19 09:32:00 : O:7979.95000000 H:7989.20000000 L:7979.70000000 C:7989.02000000 V:19.21879300 25-05-19 09:33:00 : O:7988.64000000 H:7994.00000000 L:7986.07000000 C:7993.91000000 V:21.11476500 25-05-19 09:34:00 : O:7992.70000000 H:7997.68000000 L:7992.10000000 C:7994.39000000 V:18.21645000 25-05-19 09:35:00 : O:7993.65000000 H:7997.99000000 L:7992.20000000 C:7993.08000000 V:10.55074700 25-05-19 09:36:00 : O:7993.08000000 H:7993.57000000 L:7986.48000000 C:7990.94000000 V:15.25538600 25-05-19 09:37:00 : O:7991.98000000 H:7991.98000000 L:7989.51000000 C:7991.98000000 V:0.01849800 finished.

[rodrigo-brito]

Hi @ninadpachpute, thank you for the report. Is recommended to wait for the LIVE signal to start the bot: https://github.com/rodrigo-brito/backtrader-binance-bot/blob/master/strategies/base.py#L26-L30

Is very strange the close, open, low, and high with the same value, like this: 25-05-19 09:18:00 : O:7988.74000000 H:7988.74000000 L:7988.74000000 C:7988.74000000 V:0.10418400

The bt-ccxt-broker fetch values by Tick time frame, and reduce it in minutes on Feed, are you using some compression in the main function. Can you share it with me?

I also recommend you to open an issue in the repository https://github.com/Dave-Vallance/bt-ccxt-store That is responsible to fetch the data feed. Are you using Binance?

[ninadpachpute]

@rodrigo-brito Thanks for quick response. Yes, I am using Binance. Also the data is being printed on the screen if you see the strategy. I am using all your code except the strategy which I started writing from scratch (code sis in original post).

After doing some investigation, I think it could well be a ccxt version issue. I sourced your requirement.txt at first but later updated all python packages. Let me also start using conda which will take care of version issues.

My plan is to log LIVE data coming through the bot, store it and compare later with say historical data fetched only using ccxt. Both should match to every decimal point which will give me confidence.

[rodrigo-brito]

In a bigger time frame, this kind of problems doesn't affect on average or become insignificant in the final result. It will be great, this check with the original CCXT library can help with the library validation. Please, notify me if you found some information about it. Thank you!

I checked it in a bigger time frame (1h) and the final value is very close to the original.

[ninadpachpute]

How I see it is, prices from two different sources should be exactly same to a decimal point. For example, ccxt vs tradingview prices. But what I have seen it they are not always exactly the same. I do not know the reason; if that is the way it is designed and you know why that is - would be great if you can shed some light.

[rodrigo-brito]

CCXT library has different precision settings. Look this: https://github.com/ccxt/ccxt/wiki/Manual#precision-and-limits

[ninadpachpute]

Thanks. This helps a lot. Also, the issue is already on bt-ccxt-store. https://github.com/Dave-Vallance/bt-ccxt-store/issues/10