Protect against login session outage during memcached restart

[robrap]

At this time, when we lose our caches in Production (outage or restart), we lose all user sessions and everyone is logged out. This is probably a frustrating experience for users, and also hammers our servers with logins until the event is over.

There are a number of ways this might be solved.

1. We could use a db-backed cache. (There might be performance implications.)
2. We might switch to Redis cache with persistence. (There still might be situations where the cache gets lost, but fewer?)
3. We might look at https://github.com/martinrusev/django-redis-sessions. (Discovery is needed to understand if this provides any or enough benefit beyond just switching to Redis with persistence.)

For more details on using a db-backed cache: See Using database-backed sessions and see cached_db details in Using cached sessions.

Regarding timing:

• This ticket might be a blocker to moving to Redis if we don't want to lose the sessions even once, or we might incur a planned event of losing all sessions in order to move to Redis.

[robrap]

Questions:

• Are there any security issue with these in the DB? Could anyone with DB access take over a session?
• For GUID Auth, does this become even more important, or do other services not care if edX loses its session after their session has begun?

[robrap]

This is mostly a duplicate of the new issue in edx-platform: https://github.com/openedx/edx-platform/issues/32984

However, this ticket could address 2U specific issues, like Memcached vs Redis, or other environment specific questions.

[robrap]

Update:

• We had a Memcached related incident on March 27, and April 1 of 2024.
  • Running into network bandwidth issues in AWS, dropped packed on ElastiCache node 3 made memcache spiral out of control, which made LMS performance spin out of control.
  • Some memcache node restarts invalidated sessions as well.
  • Adding a memcached node may have helped relieve network pressure, but may have caused more session invalidation.
  • Many exams and proctoring related auth issues that were likely related.
  • Can add RCA link once that occurs.

[robrap]

On May 29, 2024, the question arose about flushing parts of a cache, and it was determined that memcached only has flush_all. Another reason to switch to Redis.