Unable to get historical data from CEX.io

askmike commented 10 years ago

Since a couple of days the CEX.io API does not give back old trades anymore (using the since parameter). After contacting support they stated:

Hi Mike,

We have limited our trade_history call to 1000 trades now, We will be implementing a daily trade static file in a future update I apologize for the inconvenence and we will try to get this out ASAP.

Which means that until the static file is offered we need to redesign CEX.io implementation to:

on gekko start, start gathering trades and save them in the database (this functionality is already there). But don't calculate EMA's or give any advice yet.
Monitor how many trades we have stored in the database, as soon as the oldest data we have is old enough to calculate the start EMA's (config.EMA.candles * config.EMA.interval) start giving advice and trade.

nicobrion commented 10 years ago

looking forward for the CEX.io historical data fix. I truly congratulate you for an outstanding dev, this project is awesome! Kudos,

Nico

askmike commented 10 years ago

Thanks!

I am working on a new version of Gekko that is able to store history locally, this means it is able to monitor:

CEX.io
BTC-e (including altcoin markets)
Bitstamp (was broken for a long time)
All other API's that don't offer historical data.

So far I have written the functionality that can monitor the exchange and store candles. If people could test it that would be awesome!

Test new Gekko

Go to the dev branch and download it.
Cd to it & npm install
Set the config to either cexio, bitstamp or btce. (Mt. Gox not implemented yet, just as: real trading / mailer / advice). [edit: and pick a new market here]
Run it
History is now saved in the history folder.
Give me feedback!

fmjensen commented 10 years ago

Is this okay ? I dont know what to expect, so you just got the output here :

gekko-localDB : node gekko.js 
2013-12-24 10:48:25 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-24 10:48:25 (INFO): Let me show you some Exponential Moving Averages.
2013-12-24 10:48:25 (INFO): Using normal settings to monitor the live market
2013-12-24 10:48:25 (INFO): NOT trading with real money
2013-12-24 10:48:25 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-24 10:48:25 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:48:25 (DEBUG):    Fetched 1001 new trades, from 09:29:18 (UTC) to 09:48:23 (UTC)
2013-12-24 10:48:25 (DEBUG):    Creating a new daily database for day 2013-12-24
2013-12-24 10:48:25 (DEBUG):    inserting candle 569 (09:29:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 570 (09:30:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 571 (09:31:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 572 (09:32:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 573 (09:33:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 574 (09:34:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 575 (09:35:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 576 (09:36:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 577 (09:37:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 578 (09:38:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 579 (09:39:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 580 (09:40:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 581 (09:41:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 582 (09:42:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 583 (09:43:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 584 (09:44:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 585 (09:45:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 586 (09:46:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 587 (09:47:00 UTC)
2013-12-24 10:48:25 (DEBUG):    Leftovers: 588
2013-12-24 10:49:26 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:49:26 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:49:25 (UTC)
2013-12-24 10:49:26 (DEBUG):    inserting candle 588 (09:48:00 UTC)
2013-12-24 10:49:26 (DEBUG):    Leftovers: 589
2013-12-24 10:50:27 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:50:27 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:50:26 (UTC)
2013-12-24 10:50:27 (DEBUG):    inserting candle 589 (09:49:00 UTC)
2013-12-24 10:50:27 (DEBUG):    Leftovers: 590
2013-12-24 10:51:27 (DEBUG):    Scheduling next fetch: in 1 minutes

askmike commented 10 years ago

Thanks for testing :)

Looks great, I'll explain some things:

Why so much logging? I am still developing it and I want all feedback I can get.
You now have a new history folder (inside Gekko) that has files like cexio-BTC-GHS-2013-12-24.db, this is all the trade information Gekko was able to store!
The times are off: no they are not, they're in in UTC. The reason for this is that everyone from over the world can share historical candles and the timezones doesn't mess everything up.

One small thing I am going to look into though: It shouldn't fetch once every minute (doesn't cause any trouble, but a lot of overhead).

fmjensen commented 10 years ago

I configured it to fetch every minute, and run in debug mode to get some activity on stdout ;-) The time fits my timezone. Im running on OSX btw...

askmike commented 10 years ago

Ah, that explains it. And the answers were also for clarifying the new logic for everyone.

On CEX.io [and BTC-e] you only get the last 1000 [or 150] trades, whether they happened in a minute or over the last hour. This makes it quite hard to determine when to fetch. But I quite like how the current setup deals with it: after every fetch it calculates a new fetchtime based on trades/min from the previous fetch. It appears to do well but it needs to prove itself over langer periods of time.

Here is a log where you can see this in action:

Fetched 1001 new trades, from 12:06:49 (UTC) to 12:07:19 (UTC)
inserting candle 726 (12:06:00 UTC)
Leftovers: 727
Scheduling next fetch: in 1 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:08:17 (UTC)
inserting candle 727 (12:07:00 UTC)
Leftovers: 728
Scheduling next fetch: in 2 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:09:22 (UTC)
inserting candle 728 (12:08:00 UTC)
Leftovers: 729
Scheduling next fetch: in 3 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:11:00 (UTC)
inserting candle 729 (12:09:00 UTC)
inserting candle 730 (12:10:00 UTC)
Leftovers: 731
Scheduling next fetch: in 5 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:13:52 (UTC) 
inserting candle 731 (12:11:00 UTC)
inserting candle 732 (12:12:00 UTC)
Leftovers: 733
Scheduling next fetch: in 8 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:18:34 (UTC)
inserting candle 733 (12:13:00 UTC)
inserting candle 734 (12:14:00 UTC)
inserting candle 735 (12:15:00 UTC)
inserting candle 736 (12:16:00 UTC)
inserting candle 737 (12:17:00 UTC)
Leftovers: 738
Scheduling next fetch: in 13 minutes

I do want to make it more aggressive, missing trades is not acceptable.

fmjensen commented 10 years ago

I like it ! But... i miss the ADVICE in stdout - and can't figure out if it can be enabled somehow?!

askmike commented 10 years ago

Well this new change means that I need to rewrite how Gekko deals with data under the hood. Both the historicalCandleFetcher.js and the realtimeCandleFetcher.js are going to be obsolete and they provided basically all internal communication (from fetching to passing to trading method, etc).

I am working on it, but it's a lot of work.

nicobrion commented 10 years ago

I downloaded the localDB version and ran some test, but it only seems to monitor candle behaviour, is that right? Should I send you the logs? if you need a volunteer to test the trading feature, count me in.

bppsp1 commented 10 years ago

Is the test version able to trade yet or is it just at the candle/advice section for now ?

streetlogics commented 10 years ago

Hey - CEX seems to be running fine, but I get this error when I try to launch with Mt. Gox now (this is with debug turned on):

2013-12-26 08:37:43 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-26 08:37:43 (INFO): Let me show you some Exponential Moving Averages.

2013-12-26 08:37:43 (INFO): Using normal settings to monitor the live market
2013-12-26 08:37:43 (INFO): NOT trading with real money
2013-12-26 08:37:43 (INFO): Starting to watch the market: MtGox USD/BTC
either start looping right away (`since`) or first determine starting point dynamically

askmike commented 10 years ago

@bppsp1 nope, not yet. I'll gladly accept pull requests though :)

@streetlogics, the code works in that it takes different paths for different situations, the situations are:

No historical data available. (BTC-e, bitstamp, CEX.io)
Historical data available request-able via tid, while we need trades based on a point in time. (BTCChina I think)
Historical data available request-able via timestamp (MtGox)

I am still finishing up the first scenario. Though because you can always request historical data from Mt. Gox, you can do anything I'm trying to achieve already (except store the historical data) using the master branch.

streetlogics commented 10 years ago

Cool cool - no worries if you're aware of it then, just wanted to let you know just in case. Happy to test again on the branch once it's implemented.

djmuk commented 10 years ago

Mike - looking forward to the trade engine implementation... Have been running this version and just noticed that it didn't put any data in the new day file - it created them but they just had the "{"$$indexCreated":{"fieldName":"s","unique":true,"sparse":false}}" line in them. Restarting gekko gave me a rash of 'creating new realtime candle' and todays file got populated... Having been running since 26th, I have an empty file for the 27th and todays only got populated when I restarted.

Just another suggestion while looking at candle generation - would it be possible to also store volume weighted price (VWP) as well and also to give the option to use this in calcs rather than the centre price? This would mean iterating the trade data set but this could be calculated when assigning them to candles (and wouldn't it be as easy to pick up the OHCL figures at this point rather than using first/last/min/max later?)

Thanks

kkwestt commented 10 years ago

kwest$ node gekko 2013-12-28 16:37:15 (INFO): I'm gonna make you rich, Bud Fox. 2013-12-28 16:37:15 (INFO): Let me show you some Exponential Moving Averages.

2013-12-28 16:37:15 (INFO): Using normal settings to monitor the live market 2013-12-28 16:37:15 (INFO): NOT trading

/Users/kwest/gekko/gekko.js:60 throw invalid; ^ At this moment Gekko can't monitor CEX.io, find out more info here:

https://github.com/askmike/gekko/issues/90

askmike commented 10 years ago

@kkwestt did you read the issue? The rest of the discussion is not about a stable version. Read more here.

@djmuk Thanks. I pushed a new version a couple of hours ago that should fix the daily DB creation. I like the idea for the VWP, will add it to the candles.

kkwestt commented 10 years ago

ok i run new gekko

172-0-0-7:gekko2 kwest$ node gekko
2013-12-28 17:03:02 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-28 17:03:02 (INFO): Let me show you some Exponential Moving Averages.

2013-12-28 17:03:02 (INFO): Using normal settings to monitor the live market
2013-12-28 17:03:02 (INFO): NOT trading with real money
2013-12-28 17:03:02 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-28 17:03:03 (DEBUG):    Scheduling next fetch: in 5 minutes
2013-12-28 17:03:03 (DEBUG):    Fetched 1001 new trades, from 2013-12-28 12:17:56 (UTC) to 2013-12-28 13:02:45 (UTC)
TRADES 1001
TRADES 12:17:56
TRADES 13:02:45
MINIMUM: 2013-12-28 13:00:00
FILTERED TRADES 16
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:04
FILTERED TRADES 2013-12-28 13:00:05
FILTERED TRADES 2013-12-28 13:00:23
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:34
FILTERED TRADES 2013-12-28 13:00:43
FILTERED TRADES 2013-12-28 13:01:03
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:04
FILTERED TRADES 2013-12-28 13:01:20
FILTERED TRADES 2013-12-28 13:02:03
FILTERED TRADES 2013-12-28 13:02:04
FILTERED TRADES 2013-12-28 13:02:13
FILTERED TRADES 2013-12-28 13:02:45
FILTERED TRADES 2013-12-28 13:02:45
2013-12-28 17:03:03 (DEBUG):    inserting candle 780 (13:00:00 UTC) vol: 3.43184006
2013-12-28 17:03:03 (DEBUG):    inserting candle 781 (13:01:00 UTC) vol: 0.99916237
2013-12-28 17:03:03 (DEBUG):    Leftovers: 782

askmike commented 10 years ago

@kkwestt I consider it stable when it runs a while on busy markets as well as empty ones. Could you let me know if it crashes or anything like that?

I think I got it almost stable now:

It now also adds empty candles, it does this to differentiate between times when gekko was off and times when there were no trades at the exchange. The result is that all candles generated before (unless there was a trade in every minute) are now considered corrupt.
fixed lots of bugs around filling gaps with empty candles (between two fetches, around midnight, etc.)
added VWP
lots of other bug fixes.

dasher commented 10 years ago

The latest tip of localdb is borking out with JS Allocation failed - process out of memory - trying to track down what & where currently..

askmike commented 10 years ago

@dasher very strange? Could you post (or link to) the output with debug enabled? Haven't seen that behaviour yet. Also: on what market is this happening and do you have a history build up with previous versions?

dasher commented 10 years ago

Yep - I have debug on at all times at the mo - https://gist.github.com/dasher/b564fcb61008dbcb192a Market: cex.io GHS/BTC

I have 2 days worth of histories from other runs - I didn't clean them out when I updated.

dasher commented 10 years ago

Just renamed the history/cex* files to ensure it has a clean run this time around (I noticed you'd changed it a bit in the last day) and running it again now

dasher commented 10 years ago

Renaming the old history files & rerunning appears to keep gekko happy - no crashes yet :)

bizio55 commented 10 years ago

Is the bot supposed to create the history files as soon as you start it? I'm asking because i've been running it for some hours but after telling me that "No history found, starting to build one now" it just started fetching as it should without errors but it still hasn't created any database file.

iceydee commented 10 years ago

I've been running the localDB version roughly 12 hours now and it seems to be collecting the cex.io data just fine. However, when it passed midnight it crashed (and will no longer run). Here's the stacktrace:

2013-12-29 00:18:17 (DEBUG):    Fetched 1001 new trades, from 2013-12-28 23:34:16 (UTC) to 2013-12-29 00:18:09 (UTC)
TRADES 1001
TRADES 23:34:16
TRADES 00:18:09
2013-12-29 00:18:17 (DEBUG):    MINIMUM TRADE TRESHOLD: 2013-12-28 00:00:00
2013-12-29 00:18:18 (DEBUG):    This batch includes trades for a new day.

/home/ubuntu/gekko/databaseManager.js:457
      var max = mom.minutes + 1;
                ^
ReferenceError: mom is not defined
    at /home/ubuntu/gekko/databaseManager.js:457:17
    at Function.forEach (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:3297:15)
    at Manager.addEmtpyCandles (/home/ubuntu/gekko/databaseManager.js:447:5)
    at bound [as addEmtpyCandles] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Manager.processTrades (/home/ubuntu/gekko/databaseManager.js:351:23)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Fetcher.processTrades (/home/ubuntu/gekko/tradeFetcher.js:163:8)
    at bound (/home/ubuntu/gekko/node_modules/lodash/dist/lodash.js:729:21)

Sounds like it fails to create the new history file?

bizio55 commented 10 years ago

It's still not creating any database for me. I've tried to run it on both windows and linux and i've tried with different versions of node.js but the result it's still the same: the bot just continues to fetch and schedule without ever creating the database and without throwing any error.

djmuk commented 10 years ago

@bizio The history directory is created in the current working directory by default. Check your config.js and put an explicit path in there? Askmike - is it worth printing / logging the full path of the history file when it is created?

bizio55 commented 10 years ago

The path for the history directory is the default one and the folder gets created automatically if it doesn't exist or i delete it manually, the problem is that the bot just doesn't create the database files.

askmike commented 10 years ago

@bizio55 There was a bug in the latest version I pushed 10 hours ago, should be good now.

@iceydee That also has been fixed, thanks for the feedback!

@djmuk Since you can now configure the location of the history directory I think this is a great idea. But note that the massive logging we have right now is just to make it easier to debug. As soon as this runs stable I'll remove most of the logging (but creating a new DB is something that happens once a day top)

gabbello commented 10 years ago

I'm using the main gekko branch for trading at btc-e and downloaded this one for testing. Running both on a rasbperry pi (node 10.2).

Dev branch seems to have started fine:

2013-12-29 13:02:06 (INFO): Using normal settings to monitor the live market
2013-12-29 13:02:06 (INFO): NOT trading with real money
2013-12-29 13:02:08 (INFO): Starting to watch the market: BTC-e USD/BTC
2013-12-29 13:02:10 (DEBUG):    Scheduling next fetch: in 2 minutes
2013-12-29 13:02:10 (DEBUG):    Fetched 150 new trades, from 2013-12-29 10:51:04 (UTC) to 2013-12-29 11:02:06 (UTC)
2013-12-29 13:02:10 (INFO): No history found, starting to build one now
2013-12-29 13:02:10 (DEBUG):    Creating a new daily database for day 2013-12-29
2013-12-29 13:02:11 (DEBUG):    minimum trade treshold: 2013-12-29 00:00:00
2013-12-29 13:02:11 (DEBUG):    processing 150 trades
2013-12-29 13:02:11 (DEBUG):    inserting candle 651 (10:51:00 UTC) vol: 0.4267344 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 652 (10:52:00 UTC) vol: 0.7105197 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 653 (10:53:00 UTC) vol: 1.0400067 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 654 (10:54:00 UTC) vol: 0.0920233 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 655 (10:55:00 UTC) vol: 0.458507 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 656 (10:56:00 UTC) vol: 0.1845 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 657 (10:57:00 UTC) vol: 7.978032 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 658 (10:58:00 UTC) vol: 0.56077001 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 659 (10:59:00 UTC) vol: 6.07177633 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 660 (11:00:00 UTC) vol: 6.92122311 2013-12-29
2013-12-29 13:02:11 (DEBUG):    inserting candle 661 (11:01:00 UTC) vol: 1.6634326 2013-12-29
2013-12-29 13:02:11 (DEBUG):    Leftovers: 662

I will let it run for some time and report any bugs.

I have a question on how exactly does gekko calculates the buy and sell volumes? Right now it seems that when the advise is to BUY it will buy BTC for all available money and if the advise is to SELL it will sell all BTC, is my understanding correct?

askmike commented 10 years ago

@gabbello Thanks, let me know if Gekko starts acting strange.

If anyone wants to discuss some details, has some specific questions or wants to be involved get on IRC: #gekkobot on freenode.

EDIT:

@gabbello yes it will buy/sell using all available funds. The EMA method tries to detect trends, upon detecting an uptrend the advice is to buy the asset so that you can ride the trend. The hard part lies in detecting trends (that stay a trend in the future).

dasher commented 10 years ago

The localDB version doesn't appear to be currently provide advice - I've had a quick look around to try and find out why without luck - any suggestions on what needs changing to get that to start happening? I'd like to tinker about with some trading algos

askmike commented 10 years ago

Ah I probably wasn't clear enough:

This new version is a rewrite of the whole core of Gekko, nothing works except for storing candles at CEX.io, Bitstamp and BTC-e.

So at this moment the localDB branch:

doesn't give any advice (mail or log).
doesn't do any real trading.
doesn't work on BTCChina or Mt. Gox.

But if you are already storing data right now (and we are lucky that I don't have to change anything anymore on the datastorage level which would invalidate current data) and this functionality comes out you can use the history you are building now for calculating advice.

EDIT: I could use some help to get this working ASAP. If you want to help implement these features come to #gekkobot (freenode) on IRC!

dasher commented 10 years ago

Yep - I'm on track with the rewrite - it looks cleaner and more flexible in ways that should make adding/changing stuff later. It looks good.

I'm just wondering about how best to get things moving with algos so we can test out approaches that might work better on the exchanges with unusual markets - like GHS on cex.io

askmike commented 10 years ago

We still need some things to get this working, it is a lot of work to do this on my own. I was also thinking for CEX.io:

Instead of watching for trends happening in the market, calculate how much profit one GHS is going to make you (using one of the online profit calculators for example) and buy if price < profit, sell if price > profit.

To get Gekko spitting out advice we need to have the following implemented:

Glue trading methods to the candleManager. (tradingMethods were glued to realtimeCandleFetcher before, this isn't used anymore).
If the history is complete (there are checks for this on startup, though if it isn't Gekko will currently never check this again), pass the history from candleManager to tradingMethod using the new glue.
sync real candles (not the 1m based internal ones) with the available history (right now it calcs real candles based on 1m candles inserted since startup) and pass them from DBmanager -> candleManager -> trading method.

djmuk commented 10 years ago

WRT to trading methods - how about writing a skeleton trading method that just fetches the candle data but can act as a template for adding our own methods?

For CEX I reckon a simple algo - if the trend is up buy some (%age of available funds or fixed amount), if it is down then sell some (again %age or fixed amount). In both cases until we run out of cash/asset. At present the profit per GHS doesn't support the price....!

Someone else suggested a 'grid' method - setting matched buy/sell pairs either side of the market but I don't think the spreads on CEX would let this work

askmike commented 10 years ago

There was a simple script that did something similar but since the API does not provide historical data anymore (see openingspost) you either:

need something that doesn't require an history (everything related to trends does).
or keep everything in memory and build up a history there (really simple & straightforward), start with giving advice once the history has been build.

Last one does mean that every time you want to make a change you need to rebuild the history again, and you always need to wait for the history the be built. At default EMA settings Gekko needs ~3 days of history (so it can give advice after it has been on for 3 days).

My idea is that after the data store is done I am going to implement very simple trading methods (like buy if price < 100, sell if price > 200) and write documentation like the exchange documentation that will explain the API all methods can use to interact with. So it becomes extremely trivial to add your own methods.

But since almost all APIs hammered down on historical data, we first need a proper way to save it ourselves. Which is exactly what I'm trying to do right now.

djmuk commented 10 years ago

Just did a re-sync (as it crashed when I restarted after midnight) and now crashes:-

"2013-12-30 00:10:34 (INFO): Using normal settings to monitor the live market

2013-12-30 00:10:34 (INFO): NOT trading with real money 2013-12-30 00:10:34 (INFO): Starting to watch the market: CEX.io BTC/GHS 2013-12-30 00:10:34 (DEBUG): Requested trade data from CEX.io ... 2013-12-30 00:10:35 (DEBUG): Fetched 1001 new trades, from 2013-12-29 23:47:5 2 (UTC) to 2013-12-30 00:10:39 (UTC) 2013-12-30 00:10:35 (INFO): Full history available GOT FULL HISTORY, SIZE: 839 2013-12-30 00:10:35 (DEBUG): minimum trade treshold: 2013-12-29 23:58:00 UTC 2013-12-30 00:10:35 (DEBUG): processing 551 trade(s) 2013-12-30 00:10:35 (DEBUG): This batch includes trades for a new day. 2013-12-30 00:10:35 (DEBUG): TODO: test if this is working correctly.. 2013-12-30 00:10:35 (DEBUG): Creating a new daily database for day 2013-12-30

D:\Users\David\Documents\GitHub\gekko\databaseManager.js:497 throw 'a'; ^ a

So I guess the answer to the TODO is that it isn't... The file is created but is empty... I don't program in JS but do you need to include a test for an empty set of candles?

askmike commented 10 years ago

Thanks for letting me know! Also noticed that at midnight, should be fixed now. Could you let me know if it still bothers you?

askmike commented 10 years ago

We are discussing issues related to memory usage and leaks that the localDB branch is causing. If anybody wants to chip in, now is the time to do so give your input: either comment in the issue or get on IRC: #gekkobot (freenode)!

paraghardas commented 10 years ago

Is it not possible that we accumulate all history on our own server space and Gekko get the history from one place instead of everyone storing it on their individual machine? ... If required ... I can give you some space on my vps where may be you can install a script which will keep pulling live data from CEX ... then all machines can pull data from that one place?

askmike commented 10 years ago

I was just thinking about that and the main reason I want everybody to build their own history is that you have to trust the dataprovider (I got a lot of flaming about people not wanting to trust bitcoincharts before). If gekko is open source and only uses the APIs of the exchanges everybody can verify that nobody is mocking with anything. Trusting others is really hard when there is real money at stake, you could use these points to basically manipulate the market (in theory, it would probably take a while to manipulate bitstamp BTC/USD though if you can figure out that some altcoin almost no one is trading is being traded by a Gekko...).

Though in the far, far future we might do:

I offer the history through wizbit. Unless you read all the code you are putting your trust on me already. (I hold the keys to stuff something bad in here and I am sure it won't get noticed for at least a little while, I also hold the keys for this library which has access to people's funds on Bitstamp that's getting downloaded over 1200 times a month).
If the data is getting to big: I can offer checksums on wizbit for each day and others can setup free or paid dataproviders. Gekko could download the candles from each dataprovider (maybe for a fee) and Gekko would verify that those are valid using checksums stored on Wizbit.

gabbello commented 10 years ago

I would prefer to also have the ability to use my local data, since any solution of storing the data remotely will induce some costs and will be prone to service unavailability. As a remote storage solution maybe Amazon S3 services could be of interested?

Farmerted commented 10 years ago

Hi has there been a fix for issue #90 yet?

askmike commented 10 years ago

Exciting news guys:

I glued together the EMA method to the localDB, which means you now start seeing EMA advice!

Please be aware that there will probably be bugs and I highly advice to ~~double~~ triple check the advice before even considering it.

What should work:

Fetch trades at: CEX.io, Bitstamp & BTC-e (altcoin markets included!)
Store the history locally (inside the history folder).
If on start your local history is big enough, start giving EMA advice right away.
If after X time your local history is big enough, start giving EMA advice (this is untested, if it doesn't work let me know and restart Gekko).

What is known to not work:

Auto trading
Mailing advice
Fetching at Mt. Gox

bppsp1 commented 10 years ago

Nice, was wondering if it would be possible to add cryptsy support (and their various altcoins) in this new release.

yin commented 10 years ago

It gets into "should work" to often. Looks we need automated test, but first let's settle on an interface. I can volunteer writing tests.

Exciting news guys:

I glued together the EMA method to the localDB, which means you now start seeing EMA advice!

Please be aware that there will probably be bugs and I highly advice to ~double~ triple check the advice before even considering it.

What should work:

Fetch trades at: CEX.io, Bitstamp & BTC-e (altcoin markets included!) Store the history locally (inside the history folder). If on start your local history is big enough, start giving EMA advice right away. If after X time your local history is big enough, start giving EMA advice (this is untested, if it doesn't work let me know and restart Gekko).

What is known to not work:

Auto trading Mailing advice Fetching at Mt. Gox

— Reply to this email directly or view it on GitHub.

GoodDragon commented 10 years ago

I started running localDB branch yesterday on two instances. Both of them was crached on last night.

---------------------------------------- First instance ---------------------------------------- Config: exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio' currency: 'USD', asset: 'BTC',

Last lines from log:

2013-12-31 02:00:35 (DEBUG):    Creating a new daily database for day 2013-12-31
2013-12-31 02:00:35 (DEBUG):    inserting candle 1439 (2013-12-30 23:59:00 UTC) volume: 7.035317699999999
2013-12-31 02:00:35 (DEBUG):    Leftovers: 0
2013-12-31 02:00:35 (INFO):     Processed trades, sleeping for a minute ...
2013-12-31 02:01:47 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:47 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:55:13 (UTC) to 2013-12-31 00:01:27 (UTC)
2013-12-31 02:01:47 (DEBUG):    minimum trade treshold: 2013-12-31 00:00:28 UTC
2013-12-31 02:01:47 (DEBUG):    processing 44 trade(s)

Weird error

When I started it again:

2013-12-31 15:44:00 (INFO):     Using normal settings to monitor the live market
2013-12-31 15:44:00 (INFO):     NOT trading with real money
2013-12-31 15:44:00 (INFO):     Starting to watch the market: BTC-e USD/BTC
2013-12-31 15:44:00 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 15:44:01 (DEBUG):    Fetched 150 new trades, from 2013-12-31 13:43:41 (UTC) to 2013-12-31 13:43:54 (UTC)
2013-12-31 15:44:01 (WARN):     Found a corrupted database ( 2013-12-31 ), going to clean it up
2013-12-31 15:44:01 (DEBUG):    This should not happen, please post details here: https://github.com/askmike/gekko/issues/90
BAIL
2013-12-31 15:44:01 (INFO):     No history found, starting to build one now

---------------------------------------- Second instance ----------------------------------------

Config:
  exchange: 'BTCe', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'LTC',

Last lines from log:

2013-12-31 01:59:22 (DEBUG):    NEW REAL CANDLE
2013-12-31 01:59:22 (DEBUG):    inserting candle 1437 (2013-12-30 23:57:00 UTC) volume: 0
2013-12-31 01:59:22 (DEBUG):    Leftovers: 1438
2013-12-31 01:59:22 (INFO):     Processed trades, sleeping for 2 minutes ...
2013-12-31 02:01:22 (DEBUG):    Requested trade data from BTC-e ...
2013-12-31 02:01:23 (DEBUG):    Fetched 150 new trades, from 2013-12-30 23:30:34 (UTC) to 2013-12-31 00:01:20 (UTC)
2013-12-31 02:01:23 (DEBUG):    minimum trade treshold: 2013-12-30 23:58:12 UTC
2013-12-31 02:01:23 (DEBUG):    processing 6 trade(s)
2013-12-31 02:01:23 (DEBUG):    This batch includes trades for a new day.
2013-12-31 02:01:23 (DEBUG):    Creating a new daily database for day 2013-12-31

TypeError: Cannot set property 's' of undefined.  at Manager.processTrades (/home/user11/gekko/databaseManager.js:372:19)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at EventEmitter.emit (events.js:117:20)
    at bound [as emit] (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Fetcher.processTrades (/home/user11/gekko/tradeFetcher.js:163:8)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at process (/home/user11/gekko/exchanges/btce.js:135:7)
    at bound (/home/user11/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at Request._callback (/home/user11/gekko/node_modules/btc-e/btc-e.js:76:7)
    at Request.self.callback (/home/user11/gekko/node_modules/btc-e/node_modules/request/index.js:142:22)

Works after restart but probably will crash on next night again.

Anyway. I will now update latest version and try again.

askmike commented 10 years ago

@GoodDragon that should have been resolved by this commit. Let me know what happens next time.

@yin Totally agree, though because of the huge todo I have not written tests yet. Also because of the hard to test edgecases around having parted history / receiving weird fetch data it's hard to write bullet proof testing. Though I'm very interested in building a test suite asap. Pull requests more than welcome!

yin commented 10 years ago

Already working on tests, well one is already passing - Store.read(). I mean the CSV store from your gist. Correct me, if I am on wrong track.

I used nodeunit, so we can have test framework and examples at hand. Have a look if you're interested.

https://github.com/yin/gekko/commits/localDB

I'd also copy in my timing code from yesterday as a nice module.

Regards / S pozdravom

Matej Gagyi

On Tue, Dec 31, 2013 at 3:28 PM, Mike van Rossum notifications@github.comwrote:

@GoodDragon https://github.com/GoodDragon that should have been resolved by this commithttps://github.com/askmike/gekko/commit/80c3c51e921443925d8c979c6f9d510e5923ec48. Let me know what happens next time.

@yin https://github.com/yin Totally agree, though because of the huge today I have not written tests yet. Also because of the hard to test edgecases around having parted history / receiving weird fetch data it's hard to write bullet proof testing. Though I'm very interested in building a test suite asap.

— Reply to this email directly or view it on GitHubhttps://github.com/askmike/gekko/issues/90#issuecomment-31396383 .

askmike / gekko

Unable to get historical data from CEX.io #90

Test new Gekko