Return labels and annotations with GameServerAllocation results.

[freegroup]

I'm using the multi-cluster setup to allocate new GameServer. Because my nodes do not have any external IP's, I must create an ISTIO routing to communicate with my GameServer. This happens with an operator. Works perfect. My GameServer can receive the traffic if I know the endpoint.

The problem is, that I'm unable to determine which cluster has create the new GameServer because the response of the AllocationService do not provide this information:

{
  "gameServerName": "demo-server-77q8b-s6wb9",
  "ports": [
    {
      "name": "default",
      "port": 7534
    }
  ],
  "address": "ip-10-250-9-35.eu-west-1.compute.internal",
  "nodeName": "ip-10-250-9-35.eu-west-1.compute.internal"
}

As mention before, I can't use the nodeName because this is just the k8s internal hostname which has always the same pattern for all clusters. To implement my mapping I need the cluster name or the GameServerAllocationPolicy name which has affected.

I can name my Fleet in that way, that I can recognize the cluster by the name of the GameServer - but I find this solution really ugly.

Question: is it possible to add the cluster name to the AllocationService response?

[freegroup]

..or even better - return the annotations or labels of the selected GameServer. With this feature we can add additional information to the client response.

[markmandel]

Interesting problem. The underlying issue is that Kubernetes has no notion of a "name" or "id" on a cluster.

Two suggestions:

1. Any reason the Istio operator couldn't exist on each cluster and watch for GameServers that move to Allocation, and create routes accordingly?
2. Could also use any of the tools from https://agones.dev/site/docs/third-party-content/libraries-tools/#messaging to provide multi-cluster visibility into GameServers.

[freegroup]

Agones in the multi-cluster setup has something like a cluster name - maybe the name in the GameServerAllocationPolicy CRD.

But much better is to expose the Annotations or Labels of the selected/allocated GameServer. This would give us the possibility to transport gameserver specific information to the client. Like version of the GameServer or one-time-tokens for authentication. Endless possibilities :-) . You can hide this information with an featureFlag to avoid breaking clients.

Using ISTIO with a multicluster setup is possible as well. But to use this we must expose a public, single entry point and ISTIO routes the traffic to the dedicated cluster. If I go this way, I have a single point of failure and additional components in between which slow down my traffic. e.g. Envoy + TLS communication between the istio clusters.

The nice thing about the Agones setup is that you get as close as possible to the node and leave out as many components as possible.

[freegroup]

I try to override the status.adddress field of the GameServer with my own controller. But I think this is reconciled by the agones controller -isn't it?

[roberthbailey]

/cc @pooneh-m in case she has any good ideas.

[freegroup]

I try out the named solutions on https://agones.dev/site/docs/third-party-content/libraries-tools/#messaging but all of them brings a lot of network traffic on top if the cluster is already on allocation load. I try to avoid additional load if we scale massiv. It would be great if I can get the annotations of a GameServer, This solves almost all my problems with authentication, custom endpoints, versioning,...I can very well imagine that this can positively expand the scope of use cases of Agones.

If you could add this to the backlog I would be super grateful. My PoC is running so promising - it would be a shame if I had to add some workarounds to my setup. :-)

I have tried to implement it myself - but honestly, my Golang knowledge is still very limited and I would not take into account all the aspects that are important and necessary to you when making such a change.

[roberthbailey]

Looking back over this discussion, I think when doing multi-cluster allocation having some notion of which cluster the allocation came from could be useful - especially if it's being used to for metrics to track where allocations are occurring.

We should think hard about how we want to pass data around the system during allocations. We already have a mechanism to push data into the game server during allocation (you can set a meta patch to add labels or annotations). I think before we add the ability to extract data from the game server as well we should think hard about what the use cases are that would need it.

[markmandel]

I did just have a thought on where returning labels and annotations from the GameServer might also be very useful - high density game server allocation.

If you have a multi-tenant game server setup, with each game server tenant receiving data on a different port, if you are going through the High Density Game Server flow - you have no idea which port your session is going to be set on.

If you returned the set of labels and annotations with a GameServer on allocation, you could provide which is the next port to be utilised as an annotation (or a label) when you doing the "label unlock" step, and it would then get passed back to the allocation.

It took a while for this to percolate in my head, but I'm warming up to this idea, it opens up several opportunities to be able to communicate specific information one might need to add that is specific to their integration, without having to build it into Agones itself. For example someone was requesting the Pod internal IP with the allocation data. With this in place, one could write a sidecar for the GameServer (or a simple controller) that grabbed that data and applied an annotation to the GameServer such that it could be passed back to the allocation endpoint.

The only concern I have - if someone adds a large chunk of information in a label/annotation - does that matter.

[markmandel]

A question raised by @danieloliveira079 in chat - could there be sensitive information in a label / annotation that a user wouldn't want to pass through?

I personally can't think of anything (and it's all internal APIs anyway), so I don't think it's an issue - but it is a question worth asking just in case.

[austin-space]

It seems like it would be possible to set the allocator to return the labels and annotations unless some flag is set(configurable in the helm charts), or alternatively return nothing unless the flag is set, then return the labels and annotations(if you're afraid that someone may not want to expose the labels or annotations for some reason).

Regarding annotation length concerns: there is a limit to the overall size of all annotations(256Kb) which isn't exactly small, but shouldn't cause any issues(to my knowledge) with either the REST or gRPC api.

[ZakShearman]

Just adding that I too would like to see annotations and labels passed through for an allocation. I need to pass information about a server's version to my matchmaker which would be simplified if it could be stored in an annotation.

[freegroup]

cool - thanks!