Impossible to merge local datasets

[sheerlox]

Hi,

I'm trying to merge multiples imports into one so my live gekko who needs 3 days of data history can run. But I'm facing this problem:

Even if I create new imports, there is still 1 hour between each old import, they won't merge.

Thanks in advance !

[askmike]

Were you importing multiple imports at the same time (if so, how many)?

Might be related to #486 and #700.

[sheerlox]

Well this discussion is in case irrelevant because of the memory leak present in the backtest. My algorithm works on 30 minutes candle but the backtest can't handle 5 months so ^^ Thanks anyway !

[askmike]

Note that if you had one big dataset, you can specify a subrange. Merging datasets always give more flexibility.

What memory leak do you refer to? Are you running strats with native + talib indicators?

[sheerlox]

nope the only indicator I use is a mathematical function I hardcoded in my strat ...

[askmike]

What memory leak do you mean?

[thegamecat]

Sherlock, what happens re the memory leak? Is there a memory exhaustion reported in node console?

[thegamecat]

Sherlock, this maybe a more sensible place to discuss memory issues:

https://github.com/askmike/gekko/issues/929

[sheerlox]

@askmike how do you use local history on live papertrader ? I can't get it :/

[sheerlox]

[askmike]

how do you use local history on live papertrader ? I can't get it :/

• using the UI -> at this moment that's only possible if the market watcher has been running for a while (if you start a new paper trader without a watcher a new one is created).
• using the commandline -> Gekko will merge histories together if you run a paper trader over the commandline.

[askmike]

live gekko who needs 3 days of data history can run.

note that you can configure this yourself, you want your strat to use 3 days of data first?

[sheerlox]

@askmike thx I'll try using CLI, yes as it needs 120 candles to be accurate and works on 30 min interval :)

[sheerlox]

@askmike nice job with CLI, UI still needs some improvements :) I'll work on it with you :+1: Can I invite you on some chat ? Gitter, Telegram or so ?

[sheerlox]

@askmike seems local data merging is not working for me.

config.js (relevant infos only):

config.debug = true;

config.watch = {
  exchange: 'poloniex',
  currency: 'BTC',
  asset: 'LSK'
}

config.tradingAdvisor = {
  enabled: true,
  method: 'MD',
  candleSize: 30,
  historySize: 120,
  adapter: 'mongodb',
  talib: {
    enabled: false,
    version: '1.0.2'
  }
}

config.paperTrader = {
  enabled: true,
  reportInCurrency: true,
  simulationBalance: {
    asset: 0,
    currency: 1,
  },
  fee: 0.25,
  slippage: 0.05,
}

config.trader = {
  enabled: false,
  key: '',
  secret: '',
  username: '',
  passphrase: ''
}

config.adapter = 'mongodb';

config.importer = {
  daterange: {
    from: "2017-08-01 00:00:00"
  }
}

config['I understand that Gekko only automates MY OWN trading strategies'] = true;

console:

node gekko --config config.js --import
...
node gekko --config config.js
2017-08-11 12:34:54 (INFO): Setting up Gekko in realtime mode
2017-08-11 12:34:54 (INFO): 
2017-08-11 12:34:54 (INFO): Setting up:
2017-08-11 12:34:54 (INFO):      Trading Advisor
2017-08-11 12:34:54 (INFO):      Calculate trading advice
2017-08-11 12:34:54 (INFO):      Using the strategy: MD
2017-08-11 12:34:54 (INFO):     The trading method requests 3600 minutes of historic data. Checking availablity..
2017-08-11 12:34:54 (INFO): 

2017-08-11 12:34:55 (DEBUG):        Available local data:
2017-08-11 12:34:55 (DEBUG):            from: 101 minutes ago
2017-08-11 12:34:55 (DEBUG):            to: 2 minutes ago
2017-08-11 12:34:55 (INFO):     Usable local data available, trying to match with exchange data..
2017-08-11 12:34:55 (DEBUG):        Fetching exchange data since 17 minutes ago
2017-08-11 12:34:55 (DEBUG):        Available exchange data:
2017-08-11 12:34:55 (DEBUG):            from: 13 minutes ago
2017-08-11 12:34:55 (DEBUG):            to: 1 minutes ago
2017-08-11 12:34:55 (DEBUG):        Stitching datasets
2017-08-11 12:34:55 (INFO):     Partial history locally available, but 3499 minutes are missing.
2017-08-11 12:34:55 (INFO):     Seeding the trading method with partial historical data (Gekko needs more time before it can give advice).
2017-08-11 12:34:55 (DEBUG):        Seeding with:
2017-08-11 12:34:55 (DEBUG):            from: 101 minutes ago
2017-08-11 12:34:55 (DEBUG):            to: 14 minutes ago
2017-08-11 12:34:55 (INFO): Setting up:
2017-08-11 12:34:55 (INFO):      Paper Trader
2017-08-11 12:34:55 (INFO):      Paper trader that simulates fake trades.
2017-08-11 12:34:55 (INFO): 

2017-08-11 12:34:55 (INFO): Setting up:
2017-08-11 12:34:55 (INFO):      Performance Analyzer
2017-08-11 12:34:55 (INFO):      Analyzes performances of trades
2017-08-11 12:34:55 (INFO): 

2017-08-11 12:34:55 (INFO): Starting to watch the market: Poloniex LSK/BTC
2017-08-11 12:34:55 (DEBUG):    scheduling ticks
2017-08-11 12:34:55 (DEBUG):    Requested LSK/BTC trade data from Poloniex ...
2017-08-11 12:34:56 (DEBUG):    Processing 8 new trades. From 2017-08-11 12:21:03 UTC to 2017-08-11 12:33:15 UTC. (12 minutes)
2017-08-11 12:35:15 (DEBUG):    Requested LSK/BTC trade data from Poloniex ...
2017-08-11 12:35:16 (DEBUG):    No new trades.

[sheerlox]

Running with sqlite:

2017-08-11 12:49:15 (INFO): Setting up Gekko in realtime mode
2017-08-11 12:49:15 (INFO): 
2017-08-11 12:49:15 (INFO): Setting up:
2017-08-11 12:49:15 (INFO):      Trading Advisor
2017-08-11 12:49:15 (INFO):      Calculate trading advice
2017-08-11 12:49:15 (INFO):      Using the strategy: MD
2017-08-11 12:49:15 (INFO):     The trading method requests 3600 minutes of historic data. Checking availablity..
2017-08-11 12:49:15 (INFO): 

2017-08-11 12:49:15 (INFO):     No usable local data available, trying to get as much as possible from the exchange..
2017-08-11 12:49:15 (INFO):     Preventing Gekko from requesting 3600 minutes of history.
2017-08-11 12:49:15 (DEBUG):        Fetching exchange data since 240 minutes ago

Notice the: Preventing Gekko from requesting 3600 minutes of history.

[askmike]

Partial history locally available, but 3499 minutes are missing.

Is the problem, your local data looks too old. Maybe do a import before you run it?

[sheerlox]

@askmike Ahah my bad haven't activated candleWriter retrying

[sheerlox]

YES !!

2017-08-11 12:57:41 (INFO): Setting up Gekko in realtime mode
2017-08-11 12:57:41 (INFO): 
2017-08-11 12:57:41 (INFO): Setting up:
2017-08-11 12:57:41 (INFO):      Candle writer
2017-08-11 12:57:41 (INFO):      Store candles in a database
2017-08-11 12:57:41 (INFO): 

2017-08-11 12:57:41 (INFO): Setting up:
2017-08-11 12:57:41 (INFO):      Trading Advisor
2017-08-11 12:57:41 (INFO):      Calculate trading advice
2017-08-11 12:57:41 (INFO):      Using the strategy: MD
2017-08-11 12:57:42 (INFO):     The trading method requests 3600 minutes of historic data. Checking availablity..
2017-08-11 12:57:42 (INFO): 

2017-08-11 12:57:42 (DEBUG):        Available local data:
2017-08-11 12:57:42 (DEBUG):            from: 3600 minutes ago
2017-08-11 12:57:42 (DEBUG):            to: 103 minutes ago
2017-08-11 12:57:42 (INFO):     Usable local data available, trying to match with exchange data..
2017-08-11 12:57:42 (DEBUG):        Fetching exchange data since 118 minutes ago
2017-08-11 12:57:42 (DEBUG):        Available exchange data:
2017-08-11 12:57:42 (DEBUG):            from: 117 minutes ago
2017-08-11 12:57:42 (DEBUG):            to: 0 minutes ago
2017-08-11 12:57:42 (DEBUG):        Stitching datasets
2017-08-11 12:57:42 (INFO):     Full history locally available. Seeding the trading method with all required historical candles.
2017-08-11 12:57:42 (DEBUG):        Seeding with:
2017-08-11 12:57:42 (DEBUG):            from: 3600 minutes ago
2017-08-11 12:57:42 (DEBUG):            to: 118 minutes ago
2017-08-11 12:57:42 (INFO): Setting up:
2017-08-11 12:57:42 (INFO):      Paper Trader
2017-08-11 12:57:42 (INFO):      Paper trader that simulates fake trades.
2017-08-11 12:57:42 (INFO): 

2017-08-11 12:57:42 (INFO): Setting up:
2017-08-11 12:57:42 (INFO):      Performance Analyzer
2017-08-11 12:57:42 (INFO):      Analyzes performances of trades
2017-08-11 12:57:42 (INFO): 

2017-08-11 12:57:42 (INFO): Starting to watch the market: Poloniex LSK/BTC
2017-08-11 12:57:42 (DEBUG):    scheduling ticks
2017-08-11 12:57:42 (DEBUG):    Requested LSK/BTC trade data from Poloniex ...
2017-08-11 12:57:42 (DEBUG):    Processing 126 new trades. From 2017-08-11 11:00:38 UTC to 2017-08-11 12:57:26 UTC. (2 hours)

[askmike]

great, So If have more questions feel free to ask.

But I am closing this as the main issue is a dup of #486.

[askmike]

@SherlockStd

nice job with CLI, UI still needs some improvements :) I'll work on it with you 👍 Can I invite you on some chat ? Gitter, Telegram or so ?

Thanks, the CLI interface is a few years old, whereas the UI was only added a few months back. My first proposed improvements for live gekkos can be found here: #911.

There is a discord group here: https://discord.gg/q6tq9s

But I am working now so won't be there for at least the next few hours.

[sheerlox]

@askmike no problem ;) Joining the Discord group !

Another question: Does tickrate influate on the candle size (in my case 30 min) ?

[askmike]

no, tickrate specifies how often Gekko polls the exchange for trades, this needs to be fast enough not to miss any trades.

But everything is piped into a trade filter, candle aggregator and in your case a candle batcher (turns 1 min candles into 30 min candles). In practise you might see your candle a few seconds earlier by making the tickrate smaller (this only explains the a CLI live gekko, the UI uses two gekkos to live/paper trade so there is an additional polling layer here).

[sheerlox]

Thanks a lot !