Holes in dataset

[tuxitor]

Stops and starts of gekko often leads to holes in the dataset. It happens even with short stops and difficult to avoid in the long run. Several situations requires restart.

• Updating computer software that requires restart
• Gui Hangs that requires restart
• Configuration changes with gekko restart
• Updating gekko software with restart, which happens a lot during development

Certain strategies requires a dataset wihout holes. The issue has been discussed in several issues here. Here is one: #700. It is also discussed breafly here: #650

For exchanges that have an API with history data, this shouldn't pose much of a problem. A utility that fill the holes from history should do the trick. For exchanges without trading history the case is different.

As is seen here, the datasets are quite fragmented:

My idea is to separate the the data collecting and store it in the database. This program should have a simple API. The data needed from the exchange is relatively simple. And it should continue to run undisturbed even if the main program needs to be stopped for some reason. It should have a small footprint and be started as a daemon on system boot.

The communication with gekko could be done through UNIX sockets, shared memory, REST or WS. With A WS API gekko could simply subscribe to either candles or trade data. This API could deliver JSON or csv data, and other programs could also take advantage of this API. The communication delays would be neglectable even if the program ran on another computer/server.

Gekko is well structured, and it shouldn't be to complicated to do this.

I invite people in here to think about this and give some feedback.

[zfsamzfsam]

bitcoin exchange data->watch(hemera)->NATS->hemera(candleWriter)->database(mysql/.../influxdb) ->hemera(tradingAdvisor) ->hemera(trader) ->hemera(...)

Hemera
A Node.js microservices toolkit for NATS. https://github.com/hemerajs/hemera

node-nats
    Node.js client for NATS
    https://github.com/nats-io/node-nats

NATS NATS Server is a simple, high performance open source messaging system for cloud native applications, IoT messaging, and microservices architectures. https://nats.io/

http://nats.io/download/
Here you will find the NATS Server (for simple, fast publish-subscribe) , 
    and officially supported clients. 
Also available are a wide variety of community contributed clients, connectors, and utilities. 

influxdb——processing millions of rows/sec on modest hardware https://docs.influxdata.com/influxdb/v1.2/tools/api_client_libraries/ node-influx The InfluxDB Client for Node.js and Browsers https://github.com/node-influx/node-influx

    Features
        Performant, processing millions of rows/sec on modest hardware
        Zero dependencies

        Simple API for nearly all Influx operations
        Fully supported in Node and the browser

[LooOOooM]

I wrote a bash script which fixes that issue: http://gekkowarez.com/download-category/trading-data/

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If you feel this is very a important issue please reach out the maintainer of this project directly via e-mail: gekko at mvr dot me.