search

hubotio / hubot-redis-brain

A hubot script to persist hubot's brain using redis
69 stars 55 forks source link

Too much user data #39

Closed daniel-beard closed 12 months ago

daniel-beard commented 3 years ago

I am filing this since I have the same issue as #37 which was closed by its reporter:

My work's slack workspace has over 7000 users. As the bot runs for a while, it caches all of those users in the brain, which gets written to and read from Redis. The JSON blob gets up over 10MB, which isn't a lot, but moving that back and forth so much between the bot process and Redis starts taking a lot of time, causing the bot to be sluggish. Is there a way to disable caching the user info in Redis? My current solution is to wipe the brain user cache ever 15 min, but it feels hacky.

kieran-barry-disney commented 2 years ago

I am having the same issue, except it is causing my app to crash because I am using all my allotted memory (30MB). Is there any progress on this issue?

@daniel-beard What did you do to wipe the user cache every 15 minutes?

joeyguerra commented 1 year ago

I suspect this has more to do with the Slack adapter's strategy of "getting all the users" than hubot-redis-brain's design to store the brain as JSON. I'm working on a new hubot-slack adapter so I'll add this to my design constraints.

joeyguerra commented 1 year ago

btw, I found some environment variables that might help you manage the problem. DISABLE_USER_SYNC and API_PAGE_SIZE.

joeyguerra commented 12 months ago

I haven't seen any more responses to this issue so I'm going to close it for now.