Making Tribeca more 24/7 environnement compatible

[Camille92]

Hello Carles and everyone,

I think a new way of improvement for Tribeca is to look on how to make it more compatible in working 24/7 reliably.

For instance, for now, I'm not confident in letting Tribeca run for a week without checking before every so often something goes wrong.

Here is a list that in my opinion would make Tribeca work better 24/7.

1) Saving in MongoDB the price from the last 200mn. [FIXED]

The goal here is to no lose the values of the EWMA when we restart Tribeca. The problem now it that even for a small restart to kill a ghost order or apply an update, or testing something in the code, you lose your EWMA value and thus the APR position coming from it.

By saving it we would be able to do a restart without having any consequence. (As long as we don't put Tribeca off fo too long).

Another solution to this issue is to retrieve those data from the exchange so we will not even need to wait 3 hours warm up for the EWMA to be most efficient. (But I think it's more work than the first solution).

2) Saving default settings in the conf file. [FIXED]

From time to time, after a restart Tribeca loses its setting and go back to default settings of main.js. Maybe allowing to put those settings in the conf file of the instance will allow retrieving those settings in case Tribeca lose them. It is also convenient when you have numerous instances and don't want to connect to each of them to check if the configuration is good :)

A very cool solution would be to keep the settings in main.js but also to give the priority of the settings in the conf file. So that you can have a base for all instances and just need to change what would be peculiar for each market (ie. width).

3) Better management when we lose connection from the exchange [FIXED]

Correct me if I'm wrong but I feel like that when we lose the connection from the exchange, Tribeca can never get it back and you need a restart it to work again.

Maybe a better handling is possible here, by trying to reconnect to the exchange every 5 minutes and after 2 or 3 fails to quit Tribeca etc.

I hope that you share my view on that but I think this will greatly improve the ability of the bot to work for a long time without checking it every so often. :)

[ctubio]

ima leave this here https://github.com/coinbase/gdax-node/issues/24

[Camille92]

I can put a quote from the book I'm reading here Flash Boys by Michael Lewis:

« A stock market was a complex system. One definition of a complex system was a place where, as Zoran put it, “Shit will break and there is nothing you can do about it.” The person whose job it was to make sure shit didn’t break ran two kinds of career risks: the risk of shit breaking that was within his control, and the risk of shit breaking over which he had no control. »

I think that will all know "shit will break" but we have to make sure Tribeca can handle it most of the time :)

[Camille92]

Hello Carles,

Thank you for the update!

Can you please explain to me how fv/ewma and stdev are saved in the database ?

I don't really understand under which situations it works and how.

For instance, if the bot runs for 5 hours and stops 10 min will it get the data back? And if it stops 2h?

Or if it stops every 10mn for a few seconds will it manage to get ewma backed?

Thank you very much for the infos !

[ctubio]

fv and top bid and top ask are saved in the database every second every second too database is cleanup from old records based on factor on boot stdev/ewma is calculated as usual using the last data in database if data is not outdated by factor

is the very same as in upstream, but we only do it with fv/bid/ask, not with all the market levels xD

if factor is 5h, and app is stopped for 3h, on boot only the remaining 2h not yet outdated will be used to calculate initial stdev/ewma (the usefulness of this is inverse to the amount of time that the application is down)

btw i hate to write to the database every second, will update that someday (well, everything eventually :D)

[Camille92]

Ok thank you very much! It sounds perfect :)

I don't know that the issue about writing to the database every second but I trust you :p

[Camille92]

Hello Carles,

Is the "save in database" feature working for you?

I've been trying to restart some Tribeca for testing in Ok, Gdax and Bitfinex and it doesn't seem to perform as expected. (TBP goes back to its medium position)

[ctubio]

it can be tested with:

 $ mongo
MongoDB shell version: 3.0.14
> show databases
bitfinex          0.078GB
local             0.078GB
tribeca           0.078GB
tribeca_coinbase  0.078GB
> use tribeca
switched to db tribeca
> show tables
f
mkt
rfv
system.indexes
trades
> db.mkt.count()
493
> db.mkt.find()
...
> exit
bye

hope it helps, let me know if you need other info

[Camille92]

Hello Carles,

I'm sorry I did not reply earlier.

I found the issue! There is no problem with the integration of the data in mongo and it works very well! The only issue (minor one) is that Tribeca will look at this data only after 10mn (when Tribeca refreshes TBP).

Is it possible to make it look at it right after a boot/ restart ?

[ctubio]

errr sure, let me review it

[ctubio]

ah yea i remember, the values are loaded on boot at https://github.com/ctubio/tribeca/blob/master/src/server/statistics.ts#L26 and at https://github.com/ctubio/tribeca/blob/master/src/server/statistics.ts#L127, but are not published to the UI until next interval as you said, let me try to push to UI sooner

[ctubio]

values are pushed to the UI 1 minute after boot at https://github.com/ctubio/tribeca/blob/master/src/server/statistics.ts#L59, so i believe all is OK here :S can't reproduce your issue

[Camille92]

I'll check that (and edit this comment) tomorrow maybe it's just me or I've been unlucky!

EDIT: Ok I think I get what's wrong, even if the values are pushed after 1mn , the EWMA itself is updated only every 10mn, so before the first 10 mn TBP is neutral (and Tribeca buys/sells a lot in the wrong direction).

See the screenshot for more details, Short and long EWMA appeared only after 10mn (lines on the right) in stats and that's also when TBP is updated to its "normal value" (0 in the present case).

[Camille92]

A quick fix (but not the best) is to change the update time of EWMA in position management from 10 to 1mn :)

[ctubio]

lets make ewma customizable at https://github.com/ctubio/tribeca/issues/114

here the last pending thing if im not wrong is to fix the eventually missing configs after some restarts

[ctubio]

thanks for the idea of the optional quoting parameters in the config file!