googleforgames / open-match

Flexible, extensible, and scalable video game matchmaking.
http://open-match.dev
Apache License 2.0
3.18k stars 335 forks source link

OpenMatch is incompatible with Redis Cluster (e.g., Redis Enterprise) #1770

Closed kt81 closed 2 months ago

kt81 commented 2 months ago

What happened:

We are developing a large-scale matchmaking game application. Due to issues with high CPU load on Redis, we decided to switch to Redis Enterprise, following the guidance from this blog post: Power matchmaking with Redis Enterprise on Google Cloud Marketplace. To handle the significant load, clustering is enabled.

However, with this setup, OpenMatch does not function as expected.

error executing assignment set: CROSSSLOT Keys in request don't hash to the same slot (context='within MULTI', command='set', original-slot='8962', wrong-slot='9154', first-key='cqtd1m1vcuss********', violating-key='cqtd1o16uno********')

We encountered the above CROSSSLOT error. Upon inspecting the code, it appears that the MULTI command is used in the backend for processing AssignmentGroup.

While it might be possible to mitigate this by breaking down the AssignmentGroup into individual tickets and issuing AssignTicketsRequest per ticket, we are uncertain about the potential outcomes of this approach.

Could you advise on potential solutions? Additionally, is there a future plan for OpenMatch to support Redis Cluster?

What you expected to happen:

I expected OpenMatch to work seamlessly with Redis Enterprise when clustering is enabled, without encountering CROSSSLOT errors.

How to reproduce it (as minimally and precisely as possible):

  1. Set up a Redis Enterprise cluster with clustering enabled.
  2. Configure OpenMatch to use this Redis cluster as its backend.
  3. Attempt to process matchmaking requests, specifically those that involve the AssignmentGroup in the backend.
  4. Observe the error that occurs during the execution of the MULTI command due to keys being hashed to different slots.

Anything else we need to know?:

The issue appears to be related to the use of the MULTI command in the backend, which is not compatible with Redis Cluster when keys are not hashed to the same slot.

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.12", GitCommit:"c6939792865ef0f70f92006081690d77411c8ed5", GitTreeState:"clean", BuildDate:"2022-09-21T12:20:29Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.6-gke.1254000", GitCommit:"7466e0bc7654b17e7dc1eaa541a338c05988c9fd", GitTreeState:"clean", BuildDate:"2024-06-28T09:15:46Z", GoVersion:"go1.21.11 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.29) exceeds the supported minor version skew of +/-1

Cloud Provider/Platform (AKS, GKE, Minikube etc.):

GKE

Open Match Release Version:

1.7.0

Install Method(yaml/helm):

Helm

kt81 commented 2 months ago

I am aware that a similar issue has already been raised here: #1377. However, any information regarding potential workarounds would be greatly appreciated.

kt81 commented 2 months ago

This is no longer an urgent issue for me now. It’s sufficient for it to be tracked under the related issue.