search

ctubio / Krypto-trading-bot

Self-hosted crypto trading bot (automated high frequency market making) written in C++
https://127.0.0.1:3000
Other
3.37k stars 823 forks source link

Forever not always restating the app ? UPDATE WITH LOGS #95

Closed Camille92 closed 7 years ago

Camille92 commented 7 years ago

Hi Carles,

It seems that some instances are not restarted by forever (or not always). I don't have more info about it for now but it has been happening in all 3 of my VPS.

I don't know where would be the issue but the version I ran (this commit) before updating today did not have it. https://github.com/ctubio/Krypto-trading-bot/commit/68ae21c3d562ec0158c6f1009dcad1cf4409003a and did not have the issue. And then here

capture d ecran 2017-07-24 a 22 46 42

Tell me if you have an idea where I could come from!

Ps: Great work I think K is way lighter no? Or it's just a feeling?

ctubio commented 7 years ago

this STOPPED instances may have some error at the very first second (forever only restart process if the process doesnt exit as soon as it starts)

i believe some log message may be generated, mostly possible about failing to get the ticker on boot. the instances may had been restarted because some error, and later coulnt be restarted because another error also happened while initialy reading the ticker; i bet bitfinex was innaccessible at that time? and forever give up restarting :(

no solution without log msg (and meanwhile exchanges insisit to fall offline)

ctubio commented 7 years ago

just saying, the app will get much much much more lighter once javascript is totally removed from the server; currently even if is c++ is still using javascript engine

ctubio commented 7 years ago

a possible solution could be to retry on ticker fail at boot; at least this way instances will not be left STOPPED until a manual restart (hopefully)

Camille92 commented 7 years ago

Hey Carles,

The thing is it works on boot it's just that after some time it doesn't restart. I restarted all of them and they seem to work fine for now.

I'll tell you in the morning if some of them are stopped or not !

Looking forward for the very fast and light version !

Le 24 juil. 2017 23:52, "Carles Tubio" notifications@github.com a écrit :

a possible solution could be to retry on ticker fail at boot; at least this way instances will not be left STOPPED until a manual restart (hopefully)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ctubio/Krypto-trading-bot/issues/95#issuecomment-317565914, or mute the thread https://github.com/notifications/unsubscribe-auth/AGizFs8zJVv52lvyMQGSdvAEDY4T1WQVks5sRRIqgaJpZM4Ohubu .

ctubio commented 7 years ago

yea when restarting manyally may work cos the exchange is online, but at some point in the past it was offline and thats why the boot failed and left the instance STOPPED; if you try to manually restart when the excange is not responding will left the instance also STOPPED, hope we are talking about the same; otherwise i donno why an instance could be left STOPPED

Camille92 commented 7 years ago

So after one night running, all my instances are off (which never happened). I'll try to investigate.

Tell me if you have any idea!

Is there a way I can start all of them with forever and logs? capture d ecran 2017-07-25 a 11 15 14

Camille92 commented 7 years ago

Hey Carles,

I don't know how to manually start forever with a custom config to get the logs. Please tell me how to so I'll run the instances like that and get what it's happening and what forever says :)

Camille92 commented 7 years ago

GOT THE LOGS!!

terminate called after throwing an instance of 'std::invalid_argument'
  what():  parse error - unexpected '<'
error: Forever detected script was killed by signal: SIGABRT
error: Script restart attempt #1
K.linux.51
10:54:46.723871 CF settings loaded from JSON file btcusd OK.
10:54:46.7258 DB /data/db/K.4.0.82.db loaded OK.
10:54:46.725499 UI ready over HTTP on external port 3000.
terminate called after throwing an instance of 'std::invalid_argument'
  what():  parse error - unexpected '<'
error: Forever detected script was killed by signal: SIGABRT
ctubio commented 7 years ago

just for the record, even if you already know, to make forever print logs, replace /dev/null by /path/to/file in package.json lines of forever start

ctubio commented 7 years ago

this error means that Bitfinex API didnt respond with a JSON response, but with a HTML respose (thats why the unexpected '<' because of the <html> tag.

There is no solution i think, because continue trading while the exchange is responding garbage is simply not possible.

A workaround to the application crash could be to stop trading and wait for the exchange to come back; but this is aaaaalmost what is already happening; just that the app is not notified when the exchange is coming back so is left STOPPED and needs a manual restart, but the important think if you ask me is to stop trading, and this is already happening.

and yea, thats why forever doesnt restart the app, because first the app crashed while trading and bitfinex resonding grabage, so it crashed and forever restarted it, but just after restart it crashed again because bitfinex was still responding garbage, so it didnt restarted it again.

therefore another workarround is to not restart the app immediately, but to wait a bit to let bitfinex come back.

also another workarround is to let forever restart the app immediately but put a sleep of +1second at the very begin of the application boot, this way forever will always last more than 1 second and therefore will always try to restart the app no matter if exchanges make it crash.

seen all options i think the last one is the best (easy +1 sleep on boot and keep all other as-is).

how feels in your body?

Camille92 commented 7 years ago

Hey the third option looks like the best!

Looking at the logs again it looks like when it happened before but bot did not crash put log something like:

18:28:28.789 bitfinex Got message on unknown topic [ 26724, [ 41.426, 2, -148.963 ] ]

Camille92 commented 7 years ago

Btw I found the easiest option for logs was to just suppress -l dev/null and forever write them in the default location: :) /home/user/.forever/NAMECONFIG.log

Camille92 commented 7 years ago

The thing is, if the bot is off, it doesn't update the ewma value. Is there is a solution to ignore weird messages from bitfinex but still be connected to the book so API is updated ?

ctubio commented 7 years ago

to keep EWMA it depends if the garabage response is in the wallet data or the order data or the market data; if is not the market data, yes, the application would then be able to maintain EWMA values.

ok so lets add +1 sleep in boot or something similar, thanks for insist in this issue'¡!

ctubio commented 7 years ago

will make use of forever args:

    --minUptime      Minimum uptime (millis) for a script to not be considered "spinning"
    --spinSleepTime  Time to wait (millis) between launches of a spinning script.
Camille92 commented 7 years ago

Perfect! It seems to be the best / easiest way to do it :)

so like --minUptime 0 and --spinSleepTime 5000 ?

ctubio commented 7 years ago

almost xD uptime 1 millisecond, and sleep to 21 seconds

if still 1 millisecond is too little, we can then add a small sleep on boot to the app so it last longer always for sure (currently there is no sleep on boot [but dont think so xD i wish exchanges were so fast xD])

Camille92 commented 7 years ago

Perfect! :) I try that now :)