Importing data from Poloniex gets stuck

gmaijoe commented 7 years ago

So I'm running node gekko --import and every time it gets stuck around 5 - 10 days later. I'm not sure what the issue is.

Any ideas thanks.

Here is a screenshot. For reference it has been waiting for 13 minutes now.

askmike commented 7 years ago

Very strange, some questions:

Are doing anything else on poloniex from the same IP address?
Is Gekko using a lot of CPU when this happens?

If it is neither it seems the connection to poloniex hangs for some reason.

gmaijoe commented 7 years ago

I'm not doing anything on poloniex from the same IP address.

The cpu usage ranges from 10% - $45%.

Also, if I re-import data, how does that impact the data set? (assuming it is a file inside the "history" folder)

askmike commented 7 years ago

Also, if I re-import data, how does that impact the data set?

You can just reimport, Gekko will only insert missing data. You can set the import date to wherever it stopped working as well (a few minutes before to be safe).

assuming it is a file inside the "history" folder

Gekko uses a database called sqlite, it is indeed stored in a file in the history folder. You can at any point delete that to reset Gekko and remove all imported data.

askmike commented 7 years ago

Some other questions to help me debug:

What OS are you running?
How stable is your internet?

gmaijoe commented 7 years ago

Internet is very stable. I think you might be right about the Poloniex open on another tab (I might have had it open and it refreshed by itself). I've since deleted the tab and it is running smoothly. I'll let you know if it stops again.

askmike commented 7 years ago

Great! Just so you know, the latest version of Gekko comes with an UI which also supports importing. It should be just as stable (if it doesn't work on commandline, it also should work there). But it might be more handy to use.

gmaijoe commented 7 years ago

Sweet - for the UI Gekko, how do I install? I pulled from this Repo around one week ago so not sure if that is the most updated. I think it is Gekko V0.3.5 right now.

askmike commented 7 years ago

Ah yes, you would need to update (if you use git just do git pull && npm install). After that you can run it with node gekko --ui. With the UI you also don't need to worry about the config file anymore.

askmike commented 7 years ago

About the original issue: Gekko uses a library that connects to poloniex, I submitted a fix that will timeout the request after 10 seconds. But we have to wait until that get's merged (or I can fork it if that never happens).

askmike commented 7 years ago

Next time it gets stuck: could you let me know how long you have been waiting for? Does it get stuck for longer than 30 minutes?

dutu commented 7 years ago

@askmike some points regarding Poloniex:

"doing anything on poloniex from the same IP address" -> see https://github.com/dutu/poloLender/issues/6 on how Poloniex implements API limits (it is per IP and per account including linked accounts)
"Internet is very stable" -> see https://github.com/dutu/poloniex-api-node/issues/4 (Poloniex is suffering from frequent ESOCKETTIMEDOUT and ETIMEDOUT errors (when error handling implemented correctly)

askmike commented 7 years ago

@dutu thanks for that!

it is per IP and per account including linked accounts

For importing Gekko uses the public API (no account credentials sent)
When Gekko is "importing" it waits at least a second between API calls. (global for al exchanges right now, could be optimised). So it is not likely we are hitting the 6 calls a second limit*.

*Though looking closer at the code, it might be the case that Gekko swallows errors and keeps retrying more than 6 times a second, resulting in exactly this issue.

I'll add some debug logging when poloniex is retrying, to see if this is related to the retrying mechanism.

EDIT: it can be an error being swallowed, but it won't be the 6 calls a second error. The retry mechanism will first wait 10 seconds.

EDIT2: It cannot be the retry mechanism, there is logging for this already which is not visible in the picture. I am too tired for this, need to sleep...

dutu commented 7 years ago

@askmike, a note first: I've forked poloniex module you are using (poloniex.js) and I'm currently maintaining https://github.com/dutu/poloniex-api-node (for a separate read, see notable differences and why I'm maintaining a separate module).

I see in your code you are checking Poloniex response for errors (err returned by poloniex module).

It looks that poloniex module you are using is not handling all cases of errors returned by Poloniex. See here, comparing with error handling in my module, specifically when !err && response.statusCode !== 200. (As an example error Poloniex error 429: Too Many Requests won't be picked up by the poloniex module you are using since Poloniex returns err as null and instead the error is set by response.statusCode = 429.)

I have no idea if your issue is caused by poloniex API errors, but certainly API errors are not reported correctly to your code and you won't see them in your debugging either. This can also be one of the causes for https://github.com/askmike/gekko/issues/479.

(You are welcome to use my poloniex module, if you consider it appropriate, however a few changes would be needed when passing the parameters)

EDIT: Also the timeout request you mentioned is already implemented

cloudwitch commented 7 years ago

I too am getting this issue with the latest version of gekko. Cannot import poloniex data. Seems to be any coin. I can import from other exchanges fine.

I'm on OpenSUSE 42.3, it's a fresh VM.

sheerlox commented 7 years ago

@askmike I'm getting this error too .... a lot really, not stoping after 30 minutes, it's about 1 hour right now and pm2 restarted Gekko, so I have to assume that it crashed:

2017-08-13 16:58:19 (INFO): We have new trading advice!
2017-08-13 16:58:19 (INFO):      Position: short
2017-08-13 16:58:19 (INFO):      Market price: 0.00053639
2017-08-13 16:58:19 (INFO):      Based on market time: 2017-08-13 16:57:00

2017-08-13 16:58:19 (INFO): Trader Received advice to go short. Selling  LSK
2017-08-13 16:58:20 (INFO): Send advice via email.
2017-08-13 16:58:25 (DEBUG):    Poloniex returned an error, retrying..
2017-08-13 16:58:34 (INFO): SELL was successfull
2017-08-13 16:58:34 (INFO): (PROFIT REPORT) original simulated balance:  0.01487964 BTC
2017-08-13 16:58:34 (INFO): (PROFIT REPORT) current simulated balance:   0.01489550 BTC
2017-08-13 16:58:34 (INFO): (PROFIT REPORT) simulated profit:        0.00001586 BTC (0.10660497%)
2017-08-13 16:58:34 (INFO): (ROUNDTRIP) entry date      exit date       exposed duration P&L            profit

    ______   ________  __    __  __    __   ______
   /      \ /        |/  |  /  |/  |  /  | /      \
  /$$$$$$  |$$$$$$$$/ $$ | /$$/ $$ | /$$/ /$$$$$$  |
  $$ | _$$/ $$ |__    $$ |/$$/  $$ |/$$/  $$ |  $$ |
  $$ |/    |$$    |   $$  $$<   $$  $$<   $$ |  $$ |
  $$ |$$$$ |$$$$$/    $$$$$  \  $$$$$  \  $$ |  $$ |
  $$ \__$$ |$$ |_____ $$ |$$  \ $$ |$$  \ $$ \__$$ |
  $$    $$/ $$       |$$ | $$  |$$ | $$  |$$    $$/ 
   $$$$$$/  $$$$$$$$/ $$/   $$/ $$/   $$/  $$$$$$/

    Gekko v0.5.2
    I'm gonna make you rich, Bud Fox. 

2017-08-13 16:58:35 (INFO): Setting up Gekko in realtime mode
2017-08-13 16:58:35 (INFO): 
2017-08-13 16:58:35 (INFO): Setting up:
2017-08-13 16:58:35 (INFO):      Candle writer
2017-08-13 16:58:35 (INFO):      Store candles in a database
2017-08-13 16:58:35 (DEBUG):    Enabling adviceWriter.
2017-08-13 16:58:35 (DEBUG):    Enabling candleWriter.

sheerlox commented 7 years ago

Seems there is a bug after this line, he don't stop restarting after it without giving stacktrace: 2017-08-13 16:58:34 (INFO): (ROUNDTRIP) entry date exit date exposed duration P&L profit I'll take a look asap

askmike commented 7 years ago

if you use PM2 you can access error logs, please post those @SherlockStd!

sheerlox commented 7 years ago

My bad ahah

TypeError: Cannot read property 'format' of undefined
    at Logger.logRoundtrip (/home/sherlock/projects/gekko/plugins/performanceAnalyzer/logger.js:52:15)
    at Logger.handleRoundtrip (/home/sherlock/projects/gekko/plugins/performanceAnalyzer/logger.js:131:10)
    at PerformanceAnalyzer.handleRoundtrip (/home/sherlock/projects/gekko/plugins/performanceAnalyzer/performanceAnalyzer.js:119:16)
    at PerformanceAnalyzer.bound [as handleRoundtrip] (/home/sherlock/projects/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at PerformanceAnalyzer.logRoundtripPart (/home/sherlock/projects/gekko/plugins/performanceAnalyzer/performanceAnalyzer.js:94:10)
    at PerformanceAnalyzer.bound [as logRoundtripPart] (/home/sherlock/projects/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at PerformanceAnalyzer.processTrade (/home/sherlock/projects/gekko/plugins/performanceAnalyzer/performanceAnalyzer.js:72:8)
    at Trader.bound (/home/sherlock/projects/gekko/node_modules/lodash/dist/lodash.js:729:21)
    at emitOne (events.js:96:13)
    at Trader.emit (events.js:191:7)

sheerlox commented 7 years ago

@askmike ever seen this ?

askmike commented 6 years ago

This has since been fixed!

sheerlox commented 6 years ago

Long time no see ! Good to know you're still improving the software it's a great job I didn't had the chance to check out the last versions since a while ^^ Good continuation anyway

askmike / gekko

Importing data from Poloniex gets stuck #663