search

askmike / gekko

A bitcoin trading bot written in node - https://gekko.wizb.it/
MIT License
10.07k stars 3.95k forks source link

Unable to get historical data from CEX.io #90

Closed askmike closed 10 years ago

askmike commented 10 years ago

Since a couple of days the CEX.io API does not give back old trades anymore (using the since parameter). After contacting support they stated:

Hi Mike,

We have limited our trade_history call to 1000 trades now, We will be implementing a daily trade static file in a future update I apologize for the inconvenence and we will try to get this out ASAP.

Which means that until the static file is offered we need to redesign CEX.io implementation to:

ArgyleEl1te commented 10 years ago

I've got the local database version up and running, but it won't let me change out of BTC/TRC.... 

// Monitor the live market
config.normal = {
  enabled: true,
  exchange: 'btce', // 'MtGox', 'BTCe', 'Bitstamp' or 'cexio'
  currency: 'BTC',
  asset: 'TRC',
  tradingEnabled: true,

when I change to 

  currency: 'USD',
  asset: 'BTC',

I get this:

/Users/dave/Downloads/gekko-localDB/config.js:39
  currency: ‘USD’,
            ^
SyntaxError: Unexpected token ILLEGAL
    at Module._compile (module.js:439:25)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.util.getConfig (/Users/dave/Downloads/gekko-localDB/core/util.js:14:15)
    at Object.<anonymous> (/Users/dave/Downloads/gekko-localDB/core/util.js:105:19)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)

am I missing something obvious? haven't seen anyone else mention this...

askmike commented 10 years ago

@ArgyleEl1te it seems you are using the wrong quotes, they should not be curly but normal ones. (if unsure copy quotes from somewhere else and change the USD inside).

Note that Gekko watches per market, so you do need to rebuild a history for this other one.

ArgyleEl1te commented 10 years ago

@askmike wow... not sure how that happened. Even copied and pasted out of the old config.js from the previous version... but yes, you are right. Wiped out databases, it fired up, excited to see how it goes. Thanks for the 5 minute help turnaround!

ArgyleEl1te commented 10 years ago

Ran for about 4 hours, then threw this error and crashed:

2013-12-31 17:59:53 (INFO): Processed trades, sleeping for 2 minutes ... 2013-12-31 18:01:28 (DEBUG): Requested trade data from BTC-e ... 2013-12-31 18:01:28 (DEBUG): Fetched 150 new trades, from 2013-12-31 23:57:32 (UTC) to 2014-01-01 00:01:18 (UTC) 2013-12-31 18:01:28 (DEBUG): minimum trade treshold: 2013-12-31 23:59:51 UTC 2013-12-31 18:01:28 (DEBUG): processing 47 trade(s) 2013-12-31 18:01:28 (DEBUG): This batch includes trades for a new day. 2013-12-31 18:01:28 (DEBUG): Creating a new daily database for day 2014-01-01

TypeError: Cannot set property 's' of undefined at Manager.processTrades (/Users/dave/Downloads/gekko-localDB/core/databaseManager.js:373:19) at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21) at EventEmitter.emit (events.js:117:20) at bound as emit at Fetcher.processTrades (/Users/dave/Downloads/gekko-localDB/core/tradeFetcher.js:163:8) at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21) at process (/Users/dave/Downloads/gekko-localDB/exchanges/btce.js:135:7) at bound (/Users/dave/Downloads/gekko-localDB/node_modules/lodash/dist/lodash.js:729:21) at Request._callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/btc-e.js:76:7) at Request.self.callback (/Users/dave/Downloads/gekko-localDB/node_modules/btc-e/node_modules/request/index.js:142:22)

Loaded it back up and it gave me this: 2013-12-31 19:16:42 (INFO): I'm gonna make you rich, Bud Fox.

2013-12-31 19:16:43 (INFO): Starting to watch the market: BTC-e USD/BTC 2013-12-31 19:16:43 (DEBUG): Requested trade data from BTC-e ...

2013-12-31 19:16:43 (INFO): Seting up a new price actor: 2013-12-31 19:16:43 (INFO): Trade advisor 2013-12-31 19:16:43 (INFO): Calculate trading advice based on the Exponential Moving Average Crossovers

2013-12-31 19:16:48 (DEBUG): Fetched 150 new trades, from 2014-01-01 00:48:42 (UTC) to 2014-01-01 01:16:24 (UTC) 2013-12-31 19:16:48 (WARN): Found a corrupted database ( 2014-01-01 ), going to clean it up 2013-12-31 19:16:48 (DEBUG): This should not happen, please post details here: https://github.com/askmike/gekko/issues/90 2013-12-31 19:16:48 (INFO): No history found, starting to build one now 2013-12-31 19:16:48 (INFO): Expected to start giving advice in 3 hours (2014-01-01 04:36:48 UTC) 2013-12-31 19:16:48 (DEBUG): Creating a new daily database for day 2014-01-01 2013-12-31 19:16:48 (DEBUG): minimum trade treshold: 2014-01-01 00:00:00 UTC 2013-12-31 19:16:48 (DEBUG): processing 150 trade(s) 2013-12-31 19:16:48 (DEBUG): inserting candle 48 (2014-01-01 00:48:00 UTC) volume: 0.207574 2013-12-31 19:16:48 (DEBUG): inserting candle 49 (2014-01-01 00:49:00 UTC) volume: 0.128544 2013-12-31 19:16:48 (DEBUG): inserting candle 50 (2014-01-01 00:50:00 UTC) volume: 0 2013-12-31 19:16:48 (DEBUG): inserting candle 51 (2014-01-01 00:51:00 UTC) volume: 1.4048787200000001 2013-12-31 19:16:48 (DEBUG): inserting candle 52 (2014-01-01 00:52:00 UTC) volume: 0 2013-12-31 19:16:48 (DEBUG): inserting candle 53 (2014-01-01 00:53:00 UTC) volume: 0

So, corrupted database somehow.... FYI...

askmike commented 10 years ago

@ArgyleEl1te hmm the first error is weird, let me know if it still happens...

For the second one I am going to write a document explaining the issue, but it boils down to: it's not really corrupt but just incomplete:

You started it 75 minutes after the crash, but BTC-e most likely won't tell you what happened before a couple of minutes.

you should think of it like this: you are trying to count the cars on the road for 24 hours if you are watching for 23 hours and counted how many cars passed by you know a lot. But if you take a break for the last hour you don't know the total and you can never make assumptions about what happened during breaks if you want to make financial decisions. (or give advice)

So I created a new issue that hosts the TODO before I consider all the localDB stuff stable (and will merge this into master):

114

kingcrunch commented 10 years ago

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you :confused:

If you see this please let me know the following:

~Sat Jan 04 2014 18:38:56 GMT+0000 ~Sat Jan 04 2014 19:08:00 GMT+0000

Btw: Local time was 20:08 (MEZ/GMT+1)

askmike commented 10 years ago

Hi. Didn't follow up the whole discussion, but the app told me, I should contact you

Awesome thanks! So my question is: did it crash it did it work?

kingcrunch commented 10 years ago

@askmike Actually it looks like it works. At least no crash. :wink:

trader-cheat-codes commented 10 years ago

May I recommend removing the "If you see this please let me know the following:" I've been running this in two different instances (mtgox btc-e)since the midnight commit. No issues yet and the history looks good. Gotta say I'm impressed at this from looking over localDB and look forward to working on this project in the future.

askmike commented 10 years ago

@trader-cheat-codes I'm still working on that code and there still is one situation where it crashes after that. Gonna be out soon!

ejko commented 10 years ago

i have this error :

user@host:~/gekko$ node gekko 2014-01-18 03:39:06 (INFO): I'm gonna make you rich, Bud Fox. 2014-01-18 03:39:06 (INFO): Let me show you some Exponential Moving Averages. 2014-01-18 03:39:06 (INFO): Using normal settings to monitor the live market 2014-01-18 03:39:06 (INFO): NOT trading with real money /home/user/gekko/gekko.js:60 throw invalid; ^ At this moment Gekko can't monitor CEX.io, find out more info here: https://github.com/askmike/gekko/issues/90 user@host:~/gekko$

do I have a mistake or a problem with the source?

askmike commented 10 years ago

@ejko it depends a little on what you want:

w1ckd commented 10 years ago

Hey,

Ran the Gekko-LocalDB for 48 hours to monitor LTC-USD with no isseus. A few Trade propositions were made.

Not sure if you need a log from a working one?

Cant wait to test the Auto Trade future for Altcoin! :)

With friendly Regards, R

ejko commented 10 years ago

I apologize again, I'm new in this system. how to install unstable Gekko ?

askmike commented 10 years ago

Because the unstable and unverified state of the new version I've disabled real trading for now. All code is located in another branch (like a nightly build), to get it download and install git and run:

git clone -b localDB --single-branch git://github.com/askmike/gekko.git
cd gekko
npm install

and follow instructions in the readme.

gr225607 commented 10 years ago

hope this helps

2014-01-27 04:53:45 (INFO): I'm gonna make you rich, Bud Fox.

2014-01-27 04:53:45 (INFO): Setting up Gekko in realtime mode

2014-01-27 04:53:45 (INFO): Setting up: 2014-01-27 04:53:45 (INFO): Trading Advisor 2014-01-27 04:53:45 (INFO): Calculate trading advice 2014-01-27 04:53:45 (INFO): Using the trading method: MACD

2014-01-27 04:53:45 (INFO): Setting up: 2014-01-27 04:53:45 (INFO): Profit Simulator 2014-01-27 04:53:45 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:46 (INFO): Setting up: 2014-01-27 04:53:46 (INFO): Profit Simulator 2014-01-27 04:53:46 (INFO): Paper trader that logs fake profits.

2014-01-27 04:53:48 (INFO): Starting to watch the market: BTC-e BTC/USD 2014-01-27 04:53:49 (INFO): We don't have enough history yet to start giving advice, I'll need to gather more data first. 2014-01-27 04:53:49 (INFO): I will start giving advice around 2014-02-19 17:54:00 (local time, that's in 24 days). 2014-01-27 20:21:10 (WARN): DOUBLE UNIQUE INSERT, this should never happen. Please post details here: https://github.com/askmike/gekko/issues/90 2014-01-27 undefined { [Error: ENOSPC, write] errno: 54, code: 'ENOSPC' }

/home/pi/gekko-localDB/core/candleManager.js:898 throw err; ^ Error: ENOSPC, write

askmike commented 10 years ago

Excited to announce that the new localDB branch is out. Trading is CEX.io is working and stable.