search

askmike / gekko

A bitcoin trading bot written in node - https://gekko.wizb.it/
MIT License
10.07k stars 3.94k forks source link

Unable to use locally stored candles #362

Closed BarnumD closed 8 years ago

BarnumD commented 8 years ago

Hi. Thanks for all your work on gekko.

I'm having two similar issues when I start gekko. When i start node gekko it seems to be working alright, and it puts data into the database but i get the following error. I'm not sure if it is connected with the second issues i'm having. The program precedes on from there and seems to run fine - collecting trades and giving advice. 2016-07-02 01:59:37 (DEBUG): Exchange has data spanning 16090 seconds, we need at least 90000 seconds. 2016-07-02 01:59:37 (INFO): Unable to use locally stored candles. 2016-07-02 01:59:37 (INFO): Using the trading method: MACD

The second issue I'm having is when running node docker --backtrace I get the following error. Gekko encountered an error and can't continue Error: Query returned no candles (do you have local data for the specified range?)

My backtest settings are: from: moment.utc("2016-06-23 20:56:00") and to: moment.utc("2016-07-1 16:00:00") and do have data between those ranges. I don't know why it seems to be having trouble with the data. I'm running this in a docker container with the database volume mounted. I can share my gekko database with you and my docker run line if you like.

BarnumD commented 8 years ago

On the other hand I have a second instance going (using a separate database), and it is able to use the db (Using locally stored historical candles) when the node gekko process starts. But when I try --backtest it still gives me the same error.

askmike commented 8 years ago

Gekko basically pipes market data into a trading method. However most trading strategies cannot start reliably straight away. They need to build up a history first. How much depends on 2 settings you configured in the tradingAdvisor config: historySize and candleSize (historySize is the amount of candles of history the trading method needs. CandleSize the size of the candles, eg. 60 for hourly candles).

2016-07-02 01:59:37 (DEBUG): Exchange has data spanning 16090 seconds, we need at least 90000 seconds.

This means that you configured the historySize and the candleSize to add up to 90000 seconds. Gekko will try to get this data straight away: But apparently the exchange was only able to give data over the past 16090 seconds, so:

(I agree that this is cryptic because you are not quite sure what that means, it just means that the history required by the trading method cannot be given on start and it will take longer before you will get advice).

If you don't want to see:

2016-07-02 01:59:37 (DEBUG): Exchange has data spanning 16090 seconds, we need at least 90000 seconds.
2016-07-02 01:59:37 (INFO): Unable to use locally stored candles.

turn of the tradingAdvisor (if you only care about storing data for future backtests) or lower the historySize in the config. Or accept the fact that you have to wait longer for advice.

askmike commented 8 years ago

The second issue I'm having is when running node docker --backtrace I get the following error. Gekko encountered an error and can't continue Error: Query returned no candles (do you have local data for the specified range?)

That is a seperate issue, could you please email me your config (without API keys) and local database? You can send them to mike [at] mvr [dot] me.

askmike commented 8 years ago

Regarding the message unable to use locally stored candles. I now changed the output too:

If there is no local history available

2016-07-02 13:57:04 (INFO): Setting up:
2016-07-02 13:57:04 (INFO):    Trading Advisor
2016-07-02 13:57:04 (INFO):    Calculate trading advice
2016-07-02 13:57:04 (DEBUG):  The trading method requests 7200 seconds of historic data. Checking availablity..
2016-07-02 13:57:04 (DEBUG):  Exchange has data spanning 3590 seconds, we need at least 7200 seconds.
2016-07-02 13:57:04 (INFO):    Unable to seed the trading method with locally stored historical candles (Gekko needs more time before it can give advice).

If there is partial local history available

2016-07-02 13:53:52 (INFO): Setting up:
2016-07-02 13:53:52 (INFO):    Trading Advisor
2016-07-02 13:53:52 (INFO):    Calculate trading advice
2016-07-02 13:53:52 (DEBUG):  The trading method requests 720000 seconds of historic data. Checking availablity..
2016-07-02 13:53:52 (DEBUG):  Exchange has data spanning 3566 seconds, we need at least 720000 seconds.
2016-07-02 13:53:52 (DEBUG):  Partial history locally available. But 630120 seconds are missing. Seeding the trading method with partial historical data  (Gekko needs more time before it can give advice).

If full local history is available

2016-07-02 13:54:19 (INFO): Setting up:
2016-07-02 13:54:19 (INFO):    Trading Advisor
2016-07-02 13:54:19 (INFO):    Calculate trading advice
2016-07-02 13:54:19 (DEBUG):  The trading method requests 7200 seconds of historic data. Checking availablity..
2016-07-02 13:54:20 (DEBUG):  Exchange has data spanning 3598 seconds, we need at least 7200 seconds.
2016-07-02 13:54:20 (DEBUG):  Full history locally available. Seeding the trading method with all required historical candles.

If think this fixed the issue (the message) so I am closing this now. If you think there is a bug in the code,just comment here.

askmike commented 8 years ago

@BarnumD I just pushed a new feature to the develop branch which might be of interest. If you change your config.backtest.daterange to scan, Gekko will dynamically figure out what dateranges are available to you:

2016-07-07 00:33:33 (INFO): Scanning local history for backtestable dateranges.
2016-07-07 00:33:33 (INFO): The database has 3 candles missing, Figuring out which ones...
2016-07-07 00:33:33 (INFO): Gekko detected multiple dateranges in the locally stored history. Please pick the daterange you are interested in testing:
2016-07-07 00:33:33 (INFO):      OPTION 1:
2016-07-07 00:33:33 (INFO):    from: 2016-07-03 08:50:00
2016-07-07 00:33:33 (INFO):    to: 2016-07-03 12:50:00
2016-07-07 00:33:33 (INFO):      OPTION 2:
2016-07-07 00:33:33 (INFO):    from: 2016-07-03 13:50:00
2016-07-07 00:33:33 (INFO):    to: 2016-07-03 21:50:00
prompt: option:

And then you type either 1 or 2 (and hit enter), depending on what data you want to Gekko to backtest against.

BarnumD commented 8 years ago

Thanks. That looks promising. Although, I'd also like to figure out how I can manually edit some of the gaps I might have (not to mention trying to figure out the importer - but that's another topic.)

I built a container with the new code. Updated my config.js: config.backtest = { adapter: 'sqlite', daterange: 'scan', batchSize: 50 }

I'm getting an error when I run gekko now: `/usr/src/gekko/core/budfox/marketFetcher.js:22 var provider = config.watch.exchange.toLowerCase(); ^

TypeError: Cannot read property 'watch' of undefined `

askmike commented 8 years ago

Although, I'd also like to figure out how I can manually edit some of the gaps I might have

The basic gists about gaps is:

If you start Gekko it stores candles in a sqlite like so:

candle: { start 1, open, high, low, close, etc. }
candle: { start 2, open, high, low, close, etc. }
candle: { start 3, open, high, low, close, etc. }
candle: { start 4, open, high, low, close, etc. }

If you stop Gekko and start it again a week later you insert new candles like so:

candle: { start 11, open, high, low, close, etc. }
candle: { start 12, open, high, low, close, etc. }
candle: { start 13, open, high, low, close, etc. }
candle: { start 14, open, high, low, close, etc. }

This works fine, but the problem lies in using this data (for backtesting for example). You don't want the backtest to go from candle 4 to candle 11 as if that was the next candle, because then you are just calculating over garbage instead of real market data.

not to mention trying to figure out the importer - but that's another topic.

For supported exchanges, it should be a simple as configuring a config.import.datarange:

node gekko --import

See here for details. If something broke let me know, for me it is working fine.

I'm getting an error when I run gekko now: `/usr/src/gekko/core/budfox/marketFetcher.js:22 var provider = config.watch.exchange.toLowerCase(); ^

TypeError: Cannot read property 'watch' of undefined

Could you post your config.watch?

BarnumD commented 8 years ago

config.watch = { //exchange: 'poloniex', // see gekko/docs/supported_exchanges.md exchange: 'bitfinex', // see gekko/docs/supported_exchanges.md currency: 'USD', asset: 'BTC' }

BarnumD commented 8 years ago

I tracked it down to commit 4df979dd2e298a2cbd14e5b574948b42d610065e that broke things for me. All the commits up to that point run. But when I update to that commit it has the error: `/usr/src/gekko/core/budfox/marketFetcher.js:22 var provider = config.watch.exchange.toLowerCase(); ^

TypeError: Cannot read property 'watch' of undefined at new Fetcher (/usr/src/gekko/core/budfox/marketFetcher.js:22:24) at new Manager (/usr/src/gekko/core/budfox/marketDataProvider.js:18:17) at new BudFox (/usr/src/gekko/core/budfox/budfox.js:28:29) at /usr/src/gekko/core/pipeline.js:160:20 at /usr/src/gekko/node_modules/async/lib/async.js:232:13 at /usr/src/gekko/node_modules/async/lib/async.js:142:25 at /usr/src/gekko/node_modules/async/lib/async.js:229:17 at /usr/src/gekko/node_modules/async/lib/async.js:556:34 at prepareMarket (/usr/src/gekko/core/pipeline.js:143:7) at /usr/src/gekko/node_modules/async/lib/async.js:551:21 `

BarnumD commented 8 years ago

You should be able to duplicate it by building a docker container like:

`FROM node:6

RUN mkdir -p /usr/src/gekko WORKDIR /usr/src/gekko

Download & setup Gekko trading bot

RUN git clone -b new-bitfinex https://github.com/askmike/gekko.git .; \

RUN git clone -b develop https://github.com/askmike/gekko.git .; \ git reset --hard 4df979dd2e298a2cbd14e5b574948b42d610065e; \ npm install

Enable redis plugin

RUN npm install redis@0.10.0

CMD [ "node", "gekko" ]`

NOTE: Above code block truncates hash.

askmike commented 8 years ago

I moved this issue to #374.

askmike commented 8 years ago

(and I pushed an update, could you test develop a184473?)

stefano-neri commented 6 years ago

I might have found the reason why datasets cannot be stitched: I changed line 115 of dataStitcher.js from var stitchable = localData && exchangeData.from <= localData.to; to var stitchable = localData.from && exchangeData.from <= localData.to;

It now seems to work for me: 2018-01-07 12:22:10 (DEBUG): Available local data: 2018-01-07 12:22:10 (DEBUG): from: 60 minutes ago 2018-01-07 12:22:10 (DEBUG): to: 5 minutes ago 2018-01-07 12:22:10 (INFO): Usable local data available, trying to match with exchange data.. 2018-01-07 12:22:10 (DEBUG): Fetching exchange data since 20 minutes ago 2018-01-07 12:22:10 (DEBUG): Available exchange data: 2018-01-07 12:22:10 (DEBUG): from: 13 minutes ago 2018-01-07 12:22:10 (DEBUG): to: 0 minutes ago 2018-01-07 12:22:10 (DEBUG): Stitching datasets 2018-01-07 12:22:10 (INFO): Full history locally available. Seeding the trading method with all required historical candles. 2018-01-07 12:22:10 (DEBUG): Seeding with: 2018-01-07 12:22:10 (DEBUG): from: 60 minutes ago 2018-01-07 12:22:10 (DEBUG): to: 15 minutes ago 2018-01-07 12:22:10 (INFO): Setting up: 2018-01-07 12:22:10 (INFO): Advice logger 2018-01-07 12:22:10 (INFO): 2018-01-07 12:22:10 (INFO): Setting up: 2018-01-07 12:22:10 (INFO): Trader 2018-01-07 12:22:10 (INFO): Follows the advice and create real orders. 2018-01-07 12:22:10 (DEBUG): getting ticker, balance & fee from Bitfinex 2018-01-07 12:22:10 (INFO):