search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.03k stars 174 forks source link

Etcd cleanup #5064

Closed ManishaKumari295 closed 1 month ago

ManishaKumari295 commented 2 months ago

Closed https://github.com/sensu/sensu-go/issues/5063

Description

The user sessions gets piled up in etcd occupying upto 2bg space .

Change in behavior

Deleting the etcd user_sesssions them after 6 minutes of creation ( Access token of session gets expired in 5 mnts) This change will make sure the memory occupancy uptil 1.8 GB will be eradicated

Added

Added TTL time of 6 minutes in session.go file for each entry regarding user-sessions.

Fixed

This fixes the etcd space occupancy issue .

Change verification

elfranne commented 2 months ago

Thanks for the fix @ManishaKumari295 but it seems there is a issue with the branch...

ManishaKumari295 commented 2 months ago

Thanks for the fix @ManishaKumari295 but it seems there is a issue with the branch...

Hi @elfranne , the branch seems to be fine now. Please verify the commited changes. Thanks

fguimond commented 2 months ago

Great approach @ManishaKumari295. Just a couple of minor things and it should be good to go!

elfranne commented 1 month ago

I have limited knowledge in the inner workings of the Sensu, please take my findings with a grain of salt. After some research I can see that Etcd has a feature to create a lease grant:

LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.

But it seems we are now creating grants based on username and sessionID and usuing new grants at every UpdateSession. Will this not result of a similar situation where we now have grants filling up etcd?

Another approach would be to create a lease grant per user at the token creation and do a LeaseKeepAlive. There might also be necessary to create a cleanup of lease grants for deleted users.

@fguimond, out of curiosity, why would you keep expired token?

(token expiration * 1.5)