Threebot server fails after unsuccessful shutdown [development_threebotstart]

[waleedhammam]

How to reproduce

• start 3bot server
• kill it using pkill or any ungraceful way
• try to start again

the problem here is 3bot server sets a key in redis called "threebot.starting" but if this key is not well cleaned it will cause the server to stuck in loading

[zaibon]

The key added in redis has a timeout. So it will eventually go away.

[waleedhammam]

@zaibon I see the redis key timeout is set to 120 sec, we only wait for 75 sec so we at least we should make them match,no ?

[xmonader]

-m flag when you start kosmos to overcome that.

[waleedhammam]

@xmonader the same with me, still have to wait a 120 seconds

[waleedhammam]

Also if the master process died..

we will get an error during retrieving the data cause bcdb redis server dead. it could be in myjobs, .. etc

[waleedhammam]

well it turns out the problem only happens when background=True

When we try to start 3bot server again when the key is still in redis, it thinks that the server is still running, so configmanager : startupcmds uses bcdb model client where bcdb redis socket is not up so leads to fail.

so suggesting running bcdb socket at early stage, and before each bcdb usage we check it's up to determine master state or not

• Still investigating

[waleedhammam]

PR at: https://github.com/threefoldtech/jumpscaleX_core/pull/609

[john-kheir]

Verified