Move away from redis

[mfocko]

In the light of redis/redis#13157 and sudden cut-off of sclorg/redis-7 that caused a disruption of our production deployment on Thursday, it's time to move away from the redis

TODO:

• [ ] research possible alternatives (list of them, given by Mirek, follows):
  • [ ] Valkey - fork of Redis 7. Under Linux Foundation. Continue under BSD license.
  • [ ] Redict - fork of Redis 7. Hey, never again such problem with license. Let's change license to LGPL-3.0-only.
  • [ ] KeyDB - forked long time ago, when there was past issue with license. Based on Redis 6. Added multithreading, which add level of complexity on the simplicity for which was Redis heavily used.
• [ ] look around for images customized for OpenShift (such as https://github.com/sclorg/redis-container that we used)
• [ ] check if it's possible to swap 1:1 without »much« additional overhead
• [ ] “swish swish”[^1]

[^1]: switch to the alternative…

[mfocko]

• There's redict currently deployed (since Tuesday; Apr 23) on both production and stage
• There's been an issue hit with the flower not being reconfigured to monitor the redict instead of redis; discovered by @lbarcziova on Wednesday morning
  • However as I'm noting this down, I realized that redis was set up as a redict replica (read-only, same contents as redict), therefore I don't really understand how did it manage to set off the alerts…
• Smallish issue has been hit on the Thursday afternoon, first-response and fix by @nforro; appears to be caused by network flakes
  • Redict logs: Redis requested multiple times resync, failed on timeout and Connection reset; cause of restarts is unknown
  • Sentry events: related to the Redis sync and also failed connections to the Redict
  • @mfocko scaled down Redis on Monday as it doesn't appear we're hitting any blockers and will need to quick swap