search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.03k stars 174 forks source link

etcd slowly increase in size, sessions cleanup missing #5063

Open elfranne opened 3 months ago

elfranne commented 3 months ago

Sensu store sessions inside etcd but does not clean up. This will slowly increase the size of etcd, potentially stopping etcd as there is a hard limit at 2GB (default).

Expected Behavior

Clean up old/expired sessions

Current Behavior

Old sessions are kept under /sensu.io/user_sessions/admin/[uuid] inside etcd.

You can see the number of sessions kept in etcd: etcdctl [insert authentication and endpoint info] --command-timeout=200s --dial-timeout=200s get "/sensu.io/user_sessions/admin" --prefix=true --keys-only | grep -c -v -e '^\s*$'. After a few month we were getting close to 1.7GB, with nearly 5.000.000 keys in the etcd, and over 4.9m just being sessions.

Possible Solution

Quick fix is to delete all sessions: etcdctl [insert authentication and endpoint info] --command-timeout=200s --dial-timeout=200s del "/sensu.io/user_sessions/admin" --prefix=true This reduced etcd from 1.7GB to 392MB.

Proper fix would be to clean expired sessions.

Steps to Reproduce (for bugs)

The scale of the issue would probably hard to reproduce but we can see that the sessions are kept in etcd and keeps increasing.

Your Environment

Sensu 6.10 OSS Ubuntu 22.04 LTS

jhenderson-pro commented 3 months ago

Hey @elfranne. 👋

Thank you for the well-written issue and testing. I'll have our Engineering team take a look at this.

elfranne commented 3 months ago

not sure it's related but it am now getting new errors on the backend, every 1 to 2 minute:

{"component":"apid.graphql","error":"no claims found in the request context","level":"warning","msg":"couldn't access resource","time":"2024-08-22T10:56:03Z"}
valsharess commented 2 months ago

This is a major bug, is there any update from Engineering? We had 10.5 million user session keys that we had to cleanup, this has brought memory usage down by 9GB and reduced the database from 1.8GB to 20MB.

I don't want to have to check on this every few months as we use the api quite a lot so the sessions start piling up.

ManishaKumari295 commented 2 months ago

This is a major bug, is there any update from Engineering? We had 10.5 million user session keys that we had to cleanup, this has brought memory usage down by 9GB and reduced the database from 1.8GB to 20MB.

I don't want to have to check on this every few months as we use the api quite a lot so the sessions start piling up.

Hi , the issue is being worked upon with code fixes . We will update you about the fix once it is integrated and available. Trying our best to make it soon.

sensu-discourse commented 2 months ago

This issue has been mentioned on Sensu Community. There might be relevant details there:

https://discourse.sensu.io/t/official-install-not-working-for-ubuntu-24-04-lts/3246/9

ManishaKumari295 commented 1 month ago

PR raised - https://github.com/sensu/sensu-go/pull/5065