kuzzleio / kuzzle

Open-source Back-end, self-hostable & ready to use - Real-time, storage, advanced search - Web, Apps, Mobile, IoT -
https://kuzzle.io
Apache License 2.0
1.44k stars 124 forks source link

Plugin kuzzle-plugin-auth-passport-local bootstrap - lock wait timeout exceeded #1050

Closed Tyki closed 4 years ago

Tyki commented 6 years ago

When there is an issue with bootstrap lock, kuzzle keep restarting and is draining every %CPU available until locks are removed from ES.

Is that possible to avoid the auto-restart when there is bootstrap locks after a specified amount of retrys? I think this feature is already somewhere else @ the loading of Kuzzle, can't remember where

benoitvidis commented 6 years ago

I am sorry, I do not manage to reproduce the issue.

The current behavior when bootstraping plugins is to try 10 times before raising an exception. PM2 then catches the exception and restarts Kuzzle but after a given amount a time, the lock is considered to be obsolete and is forcibly refreshed.

The process itself, apart from Kuzzle start, is not extremely CPU consuming and I cannot observe it either, even when forcing Kuzzle not to erase the current lock and injecting an invalid one.

Could you provide a dump of the lock db when you encounter the issue again so we can try to investigate it further?

Tyki commented 6 years ago

I just got the error (still using Kuzzle 1.2.4). Here is the content of the %25kuzzle/config:

POST http://localhost:9200/%25kuzzle/config/_search

{
    "took": 179,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 4,
        "max_score": 1,
        "hits": [
            {
                "_index": "%kuzzle",
                "_type": "config",
                "_id": "bootstrap-lock-955d2bb10d32bc42",
                "_score": 1,
                "_source": {
                    "timestamp": 1520246124822
                }
            },
            {
                "_index": "%kuzzle",
                "_type": "config",
                "_id": "bootstrap-lock",
                "_score": 1,
                "_source": {
                    "timestamp": 1520343718515
                }
            },
            {
                "_index": "%kuzzle",
                "_type": "config",
                "_id": "security.jwt.secret",
                "_score": 1,
                "_source": {
                    "seed": "721c42227eb492e2deb2a22825eb0c8039f597680a8fd2b874647e6a7cae0500dff950c56f47c40026d9e8e1de5eeec50242e5ba363d492c011a5d1ce6056bdef195eca0aa7d4c9e2e942692725042d19019597b78d90b99eb1c66e0c20ad06be640a62f3c92b7b1bc9a9d155614c202997b78619cdce518631a7224757fcfdd7826aa83566357292126d5aa63401bb1a6ef8c87a4181fa6dcdc3f0b2ad1727e9b83d70a3ea7ad6fa5446c751a7c177e6a2a66125575524ab8b4df816669e6687da9616cbce4ba860ccd25108f508a9022e7687d117254c1c47c1cae00d7a1f5596dcab692fd055898c08c9480d7a7802b2c1fa86e1e3132e4847757d2ddec15afe4f2e17abc0574a21c634f06f9e53447e86d1c6d6a775935d132d0b38ff230ac52a6a861849f628adc9efaf67ca5f87f7c4660b71a51dbdb12fdfb42e04072ffa9d8dd25c65f2cc24233cd1b6b687a9baca19806a441a2ff405b4bb41bd31bfb91ff981558e47e0deb839512721456f91aae5fa0430ac2ebc46c26d4c4c8a23dab6cc299917863fad08d3847596005b7b1dc1d93197d48fa1309a971b34d069e73eb74da430361e21940c2969f20f937caf26663e6bc770cd7e87ca7c3dd0da27706505b67cd7edda5319ac356c5710ae94771c2f20b4b3914622c1d14da3346a54c92529fe59af95fa32283cee7bddbf27d37aef8a2eb89d68b4c9c01701d"
                }
            },
            {
                "_index": "%kuzzle",
                "_type": "config",
                "_id": "bootstrap-lock-2a0395a4533ea3eb",
                "_score": 1,
                "_source": {
                    "timestamp": 1520246124823
                }
            }
        ]
    }
}

I can provide a .json with all the %25kuzzle data, but it comes with 300K users

Aschen commented 4 years ago

Fixed in v2