RocketChat / helm-charts

Repository for RocketChat helm charts
37 stars 67 forks source link

liveness probe is wrong #49

Closed smulikHakipod closed 2 years ago

smulikHakipod commented 2 years ago

Hey, It seems the liveness probes are wrong, because of this issue: https://github.com/RocketChat/Rocket.Chat/issues/24795 It seems rocketchat is dead after a mongo disconnection, yet liveness probe in my case say the rocket chat is alive and well. We need to improve the liveness probe, or either fix upstream rocketchat? Any idea how do to that?

For now, I patched my local k8s rocketchat with that:

        livenessProbe:
          failureThreshold: 1
          exec:
            command:
              - /bin/sh
              - -c 
              - node -e "var net = require('net'); var s = new net.Socket(); s.connect(3000, '127.0.0.1', (e) => console.log('connected') && s.end()); s.end()" && node -e "var net = require('net'); var s = new net.Socket(); s.connect(27017, 'rocketchat-mongodb-headless', (e) => console.log('connected') && s.end()); s.end()"
          initialDelaySeconds: 60
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 5

which is probably far from ideal.

Thanks

prppedro commented 2 years ago

hmmm... Yes, I tend to think it's better to wait on the upstream fix, though it may take a while. The way I see it, liveness probe keeps working probably because it's not directly affected by the failing components in an event of MongoDB disconnection. Either way we'll keep tracking the status of that upstream issue and take the appropriate actions, should we, for example, improve our default probing parameters.

prppedro commented 2 years ago

The development team fixed that issue, already. Closing, here.

smulikHakipod commented 2 years ago

Thanks! Is there a link to the changelog indicating that? Because the upatream issue is still open

prppedro commented 2 years ago

@smulikHakipod, maybe they forgot to close the issue? But one of the developers told me they fixed it. Also, I've tested it afterwards and it worked. The probes now report failure whenever the software fails to connect to MongoDB.

smulikHakipod commented 2 years ago

@prppedro Hi! tested on 4.7.4, its not solved :(