search

googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.12k stars 818 forks source link

Dedicated server is shut down immediately after starting #3274

Closed laivu266 closed 1 year ago

laivu266 commented 1 year ago

What happened: Dedicated server is shut down immediately after starting on some seconds The below is log of failed Dedicated server Screenshot 2023-07-19 at 16 58 18

and the below is log on normal Dedicated server: Screenshot 2023-07-19 at 17 03 10

Please help me to investigate. Thanks

What you expected to happen: Dedicated server have to Ready for match making

How to reproduce it (as minimally and precisely as possible): This case is randomly occur

Anything else we need to know?:

Environment:

laivu266 commented 1 year ago

@markmandel any idea for this?

markmandel commented 1 year ago

Please go through https://agones.dev/site/docs/guides/troubleshooting/#something-went-wrong-with-my-gameserver and post your results.

laivu266 commented 1 year ago

Hi @markmandel , I can't see expected information I want from your link. There are no abnormal logs in agones controller. Today, I have still faced this issue, our gs pod was shutdown immediately after starting.

Screenshot 2023-08-15 at 14 25 50

markmandel commented 1 year ago

I'll ask again for the information provided in: https://agones.dev/site/docs/guides/troubleshooting/#something-went-wrong-with-my-gameserver

Particularly:

ashutosji commented 1 year ago

Hi @laivu266, Were you able to get the root cause of why gameserver is shut down immediately? Follow the troubleshooting guidelines and let us know if you are unable to resolve this issue.

laivu266 commented 1 year ago

Hi @markmandel and @ashutosji , The below is event log from our Pod and GameServer:

kubectl get events | grep hmqnp-bjtd5
43m         Normal    PortAllocation           gameserver/game-0-3-4-mygs-hmqnp-bjtd5       Port allocated
43m         Normal    Creating                 gameserver/game-0-3-4-mygs-hmqnp-bjtd5       Pod game-0-3-4-mygs-hmqnp-bjtd5 created
43m         Normal    Scheduled                pod/game-0-3-4-mygs-hmqnp-bjtd5              Successfully assigned default/game-0-3-4-mygs-hmqnp-bjtd5 to ip-10-10-1-91.ap-southeast-1.compute.internal
43m         Normal    Scheduled                gameserver/game-0-3-4-mygs-hmqnp-bjtd5       Address and port populated
43m         Normal    Pulled                   pod/game-0-3-4-mygs-hmqnp-bjtd5              Container image "us-docker.pkg.dev/agones-images/release/agones-sdk:1.31.0" already present on machine
43m         Normal    Created                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Created container agones-gameserver-sidecar
43m         Normal    Started                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Started container agones-gameserver-sidecar
43m         Normal    Pulling                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Pulling image "AWSID.dkr.ecr.ap-southeast-1.amazonaws.com/game/mygs:0.3.4-hash"
43m         Normal    Pulled                   pod/game-0-3-4-mygs-hmqnp-bjtd5              Successfully pulled image "AWSID.dkr.ecr.ap-southeast-1.amazonaws.com/game/mygs:0.3.4-hash" in 166.238103ms
43m         Normal    Created                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Created container game-dedicated-server
43m         Normal    Started                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Started container game-dedicated-server
42m         Normal    Shutdown                 gameserver/game-0-3-4-mygs-hmqnp-bjtd5       Deletion started
42m         Normal    Shutdown                 gameserver/game-0-3-4-mygs-hmqnp-bjtd5       Deleting Pod game-0-3-4-mygs-hmqnp-bjtd5
42m         Normal    Killing                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Stopping container agones-gameserver-sidecar
42m         Normal    Killing                  pod/game-0-3-4-mygs-hmqnp-bjtd5              Stopping container game-dedicated-server
43m         Normal    SuccessfulCreate         gameserverset/game-0-3-4-mygs-hmqnp          Created gameserver: game-0-3-4-mygs-hmqnp-bjtd5
42m         Normal    SuccessfulDelete         gameserverset/game-0-3-4-mygs-hmqnp          Deleted gameserver in state Scheduled: game-0-3-4-mygs-hmqnp-bjtd5

and log of game server gs-log.txt (some sensitive info was removed) Please take a look on it. Thanks.

markmandel commented 1 year ago

Looking at the logs and your event stream, the game server process never reports as Ready - in fact, it looks like there is something in your code that is moving the GameServer directly to Shutdown.

So two thoughts:

1) There is something in your game server code that is moving things to Shutdown on startup 2) You have a Fleet autoscaler in effect, which will occasionally overshoot and then self heal (so it can spin up extra servers and bring them back down).

laivu266 commented 1 year ago

Thanks @markmandel We'll deep dive to detect the root cause from our GameSever.

dmitryrn commented 1 year ago

I also have this problem, seems like the game server pod gets shutdown by something when it shouldn't.

Few logs of game server sidecars:

Defaulted container "agones-gameserver-sidecar" out of: agones-gameserver-sidecar, gameserver
{"ctlConf":{"Address":"localhost","IsLocal":false,"LocalFile":"","Delay":0,"Timeout":0,"Test":"","TestSdkName":"","GRPCPort":9357,"HTTPPort":9358},"featureGates":"CustomFasSyncInterval=true\u0026Example=false\u0026LifecycleContract=false\u0026PlayerAllocationFilter=false\u0026PlayerTracking=false\u0026ResetMetricsOnDelete=false\u0026SDKGracefulTermination=false\u0026StateAllocationFilter=true","message":"Starting sdk sidecar","severity":"info","source":"main","time":"2023-08-23T14:33:51.731016086Z","version":"1.28.0"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Created GameServer sidecar","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.732013342Z"}
{"grpcEndpoint":"localhost:9357","message":"Starting SDKServer grpc service...","severity":"info","source":"main","time":"2023-08-23T14:33:51.732955618Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","health":{"disabled":true},"message":"Setting health configuration","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.833995735Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Starting SDKServer http health check...","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.834221979Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Starting workers...","queue":"agones.dev.dev-server.fleet1-h2pmt-7j796","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.8342461Z","workers":1}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Received Ready request, adding to queue","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.906111647Z"}
{"gsKey":"updateState","message":"Enqueuing","queue":"agones.dev.dev-server.fleet1-h2pmt-7j796","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.906230679Z"}
{"gsKey":"updateState","message":"Processing","queue":"agones.dev.dev-server.fleet1-h2pmt-7j796","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.911930585Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Updating state","severity":"debug","source":"*sdkserver.SDKServer","state":"RequestReady","time":"2023-08-23T14:33:51.91220117Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.92889055Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Event(v1.ObjectReference{Kind:\"GameServer\", Namespace:\"dev-server\", Name:\"fleet1-h2pmt-7j796\", UID:\"d26165d9-46fc-4543-8d5a-29b5e822267f\", APIVersion:\"agones.dev/v1\", ResourceVersion:\"5958644\", FieldPath:\"\"}): type: 'Normal' reason: 'RequestReady' SDK state change","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:51.929445839Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:52.566053464Z"}
{"httpEndpoint":"localhost:9358","message":"Starting SDKServer grpc-gateway...","severity":"info","source":"main","time":"2023-08-23T14:33:52.734816389Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:55.502848293Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:55.544680975Z"}
{"message":"shutting down sdk server","severity":"info","source":"main","time":"2023-08-23T14:33:55.604856616Z"}
{"gsKey":"dev-server/fleet1-h2pmt-7j796","message":"...shutting down workers","queue":"agones.dev.dev-server.fleet1-h2pmt-7j796","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:55.604895696Z"}
{"error":"http: Server closed","message":"http server closed","severity":"info","source":"main","time":"2023-08-23T14:33:55.60507128Z"}
{"error":"http: Server closed","gsKey":"dev-server/fleet1-h2pmt-7j796","message":"Health check: http server closed","severity":"error","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:55.60510191Z"}
rpc error: code = NotFound desc = an error occurred when try to find container "74f4d94b012fe051c92e3d3e3d8d037e9c42ad573f0ddbe789c4cc5b89244a14": not found%                                       ➜  ci-scripts git:(main)
Defaulted container "agones-gameserver-sidecar" out of: agones-gameserver-sidecar, gameserver
{"ctlConf":{"Address":"localhost","IsLocal":false,"LocalFile":"","Delay":0,"Timeout":0,"Test":"","TestSdkName":"","GRPCPort":9357,"HTTPPort":9358},"featureGates":"CustomFasSyncInterval=true\u0026Example=false\u0026LifecycleContract=false\u0026PlayerAllocationFilter=false\u0026PlayerTracking=false\u0026ResetMetricsOnDelete=false\u0026SDKGracefulTermination=false\u0026StateAllocationFilter=true","message":"Starting sdk sidecar","severity":"info","source":"main","time":"2023-08-23T14:33:41.7573892Z","version":"1.28.0"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Created GameServer sidecar","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.758941605Z"}
{"grpcEndpoint":"localhost:9357","message":"Starting SDKServer grpc service...","severity":"info","source":"main","time":"2023-08-23T14:33:41.760003914Z"}
{"httpEndpoint":"localhost:9358","message":"Starting SDKServer grpc-gateway...","severity":"info","source":"main","time":"2023-08-23T14:33:41.760628434Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","health":{"disabled":true},"message":"Setting health configuration","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.860149495Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Starting SDKServer http health check...","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.860287387Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Starting workers...","queue":"agones.dev.dev-server.fleet1-h2pmt-2fmlb","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.860311367Z","workers":1}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Received Ready request, adding to queue","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.947629322Z"}
{"gsKey":"updateState","message":"Enqueuing","queue":"agones.dev.dev-server.fleet1-h2pmt-2fmlb","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.947747625Z"}
{"gsKey":"updateState","message":"Processing","queue":"agones.dev.dev-server.fleet1-h2pmt-2fmlb","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.953517022Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Updating state","severity":"debug","source":"*sdkserver.SDKServer","state":"RequestReady","time":"2023-08-23T14:33:41.953720185Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Event(v1.ObjectReference{Kind:\"GameServer\", Namespace:\"dev-server\", Name:\"fleet1-h2pmt-2fmlb\", UID:\"75587f93-4714-47b6-ad53-1bc8ee5b4a52\", APIVersion:\"agones.dev/v1\", ResourceVersion:\"5958326\", FieldPath:\"\"}): type: 'Normal' reason: 'RequestReady' SDK state change","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.967624388Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:41.967893632Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:42.601671052Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:45.912538245Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Sending GameServer Event to connectedStreams","severity":"debug","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:45.943260741Z"}
{"gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"...shutting down workers","queue":"agones.dev.dev-server.fleet1-h2pmt-2fmlb","severity":"info","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:45.996972974Z"}
{"message":"shutting down sdk server","severity":"info","source":"main","time":"2023-08-23T14:33:45.996947133Z"}
{"error":"http: Server closed","gsKey":"dev-server/fleet1-h2pmt-2fmlb","message":"Health check: http server closed","severity":"error","source":"*sdkserver.SDKServer","time":"2023-08-23T14:33:45.997092016Z"}
{"error":"http: Server closed","message":"http server closed","severity":"info","source":"main","time":"2023-08-23T14:33:45.997105736Z"}
rpc error: code = NotFound desc = an error occurred when try to find container "845af8203bfb8f34dec7b780a9a785672a59ad5f588478b1feb9dc88a2ac68b3": not found

the configuration is following:

Agones version: 1.33.0

---
apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: fleet1
  namespace: dev-server
  labels:
    fleetset: "1"
spec:
  replicas: 1
  scheduling: Distributed
  strategy:
    type: Recreate
  template:
    spec:
      ports:
      - name: default
        containerPort: 7777
        portPolicy: Dynamic
        protocol: UDP
      health:
        disabled: true
      sdkServer:
        logLevel: Debug
      template:
        metadata:
          labels:
            gameserver: gameserver
        spec:
          containers:
          - name: gameserver
            imagePullPolicy: Always
            image: gs
            env:
            - name: NATS_URL
              value: "nats:4222"
            resources:
              requests:
                memory: 500Mi
                cpu: "200m"
              limits:
                memory: 1000Mi
                cpu: "800m"

---
apiVersion: "autoscaling.agones.dev/v1"
kind: FleetAutoscaler
metadata:
  name: fleet1-autoscaler
  namespace: dev-server  
spec:
  fleetName: fleet1
  policy:
    type: Buffer
    buffer:
      bufferSize: 2
      minReplicas: 0
      maxReplicas: 10
  sync:
    type: FixedInterval
    fixedInterval:
      seconds: 5

autoscaler describe:

Name:         fleet1-autoscaler
Namespace:    dev-server
Labels:       app.kubernetes.io/instance=dev-server-redacted
Annotations:  <none>
API Version:  autoscaling.agones.dev/v1
Kind:         FleetAutoscaler
Metadata:
  Creation Timestamp:  2023-08-23T13:57:10Z
  Generation:          3
  Resource Version:    5963910
  UID:                 889645da-7039-4f3b-9316-0041aae7fd1e
Spec:
  Fleet Name:  fleet1
  Policy:
    Buffer:
      Buffer Size:   2
      Max Replicas:  10
      Min Replicas:  0
    Type:            Buffer
  Sync:
    Fixed Interval:
      Seconds:  5
    Type:       FixedInterval
Status:
  Able To Scale:     true
  Current Replicas:  1
  Desired Replicas:  2
  Last Scale Time:   2023-08-23T14:37:50Z
  Scaling Limited:   false
Events:
  Type    Reason            Age                    From                        Message
  ----    ------            ----                   ----                        -------
  Normal  AutoScalingFleet  5m38s (x351 over 36m)  fleetautoscaler-controller  Scaling fleet fleet1 from 1 to 3
  Normal  AutoScalingFleet  38s (x57 over 40m)     fleetautoscaler-controller  Scaling fleet fleet1 from 1 to 2

Game servers's behavior:

➜  k -n dev-server get gs
NAME                 STATE          ADDRESS                                               PORT   NODE                                            AGE
fleet1-h2pmt-xtkkp   RequestReady   redacted   7011   redacted   2s
➜  k -n dev-server get gs
NAME                 STATE   ADDRESS                                               PORT   NODE                                            AGE
fleet1-h2pmt-xtkkp   Ready   redacted   7011   redacted   4s
➜  k -n dev-server get gs
NAME                 STATE       ADDRESS                                                 PORT   NODE                                            AGE
fleet1-h2pmt-wjjvm   Scheduled   redacted   7209   redacted     0s
fleet1-h2pmt-xtkkp   Shutdown    redacted     7011   redacted   5s
➜  k -n dev-server get gs
NAME                 STATE          ADDRESS                                                 PORT   NODE                                          AGE
fleet1-h2pmt-wjjvm   RequestReady   redacted   7209   redacted   2s
➜  k -n dev-server get gs
NAME                 STATE   ADDRESS                                                 PORT   NODE                                          AGE
fleet1-h2pmt-wjjvm   Ready   redacted   7209   redacted   3s
➜  k -n dev-server get gs

NAME                 STATE   ADDRESS                                                 PORT   NODE                                          AGE
fleet1-h2pmt-wjjvm   Ready   redacted   7209   redacted   5s
➜  ci-scripts git:(main) ✗
➜  k -n dev-server get gs
NAME                 STATE       ADDRESS                                                 PORT   NODE                                            AGE
fleet1-h2pmt-qv8pw   Scheduled   redacted     7055   redacted   1s
fleet1-h2pmt-wjjvm   Shutdown    redacted   7209   redacted     6s
➜  k -n dev-server get gs
NAME                 STATE          ADDRESS                                               PORT   NODE                                            AGE
fleet1-h2pmt-qv8pw   RequestReady   redacted   7055   redacted   2s
➜  k -n dev-server get gs
NAME                 STATE   ADDRESS                                               PORT   NODE                                            AGE
fleet1-h2pmt-qv8pw   Ready   redacted   7055   redacted   4s
➜  k -n dev-server get gs
NAME                 STATE       ADDRESS                                                 PORT   NODE                                            AGE
fleet1-h2pmt-94tx5   Scheduled   redacted   7228   redacted     1s
fleet1-h2pmt-qv8pw   Shutdown    redacted     7055   redacted   6s

The game server source code:

package main

import (
    "time"

    sdk "agones.dev/agones/sdks/go"
)

func main() {
    sdk, err := sdk.NewSDK()
    if err != nil {
        panic(err)
    }

    err = sdk.Ready()
    if err != nil {
        panic(err)
    }

    time.Sleep(time.Hour*999)
}

Controller logs:

{"error":"pod \"fleet1-6vk79-bnwr4\" not found","gsKey":"dev-gamedesign/fleet1-6vk79-bnwr4","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:36:16.354074119Z"}
{"message":"pod \"fleet1-6vk79-bnwr4\" not found","severity":"error","time":"2023-08-23T14:36:16.35417262Z"}
{"error":"pod \"fleet1-h2pmt-s7fbz\" not found","gsKey":"dev-server/fleet1-h2pmt-s7fbz","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:36:30.648925365Z"}
{"message":"pod \"fleet1-h2pmt-s7fbz\" not found","severity":"error","time":"2023-08-23T14:36:30.649023457Z"}
{"error":"pod \"fleet1-h2pmt-s7fbz\" not found","gsKey":"dev-server/fleet1-h2pmt-s7fbz","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:36:30.649087138Z"}
{"message":"pod \"fleet1-h2pmt-s7fbz\" not found","severity":"error","time":"2023-08-23T14:36:30.649137239Z"}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:39:40.682807447Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"error":"pod \"fleet1-pw6bq-cms8b\" not found","gsKey":"dev-client/fleet1-pw6bq-cms8b","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:39:46.64083671Z"}
{"message":"pod \"fleet1-pw6bq-cms8b\" not found","severity":"error","time":"2023-08-23T14:39:46.640947613Z"}
{"error":"pod \"fleet1-pw6bq-cms8b\" not found","gsKey":"dev-client/fleet1-pw6bq-cms8b","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:39:46.64975522Z"}
{"message":"pod \"fleet1-pw6bq-cms8b\" not found","severity":"error","time":"2023-08-23T14:39:46.649861482Z"}
{"error":"pod \"fleet1-pw6bq-cms8b\" not found","gsKey":"dev-client/fleet1-pw6bq-cms8b","message":"","queue":"agones.dev.GameServerController","severity":"error","source":"*gameservers.Controller","time":"2023-08-23T14:39:46.666411559Z"}
{"message":"pod \"fleet1-pw6bq-cms8b\" not found","severity":"error","time":"2023-08-23T14:39:46.684759277Z"}
{"error":"pod \"fleet1-pw6bq-cms8b\" not found","gsKey":"dev-client/fleet1-pw6bq-cms8b","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:39:46.666846877Z"}
{"message":"pod \"fleet1-pw6bq-cms8b\" not found","severity":"error","time":"2023-08-23T14:39:46.688035692Z"}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:41:09.852867341Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:44:40.702955403Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:46:09.875153238Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"message":"unable to calculate 'Creating' state duration of 'gds-example' GameServer","severity":"warning","source":"*metrics.Controller","time":"2023-08-23T14:47:30.999892883Z"}
{"message":"unable to calculate 'Starting' state duration of 'gds-example' GameServer","severity":"warning","source":"*metrics.Controller","time":"2023-08-23T14:47:31.012922689Z"}
{"message":"unable to calculate 'Scheduled' state duration of 'gds-example' GameServer","severity":"warning","source":"*metrics.Controller","time":"2023-08-23T14:47:32.549997406Z"}
{"message":"unable to calculate 'RequestReady' state duration of 'gds-example' GameServer","severity":"warning","source":"*metrics.Controller","time":"2023-08-23T14:47:33.182159367Z"}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:49:40.720649934Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:51:09.89248785Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}
2023/08/23 14:51:09 http: TLS handshake error from 172.31.35.176:37168: EOF
{"error":"pod \"fleet1-h2pmt-q2plv\" not found","gsKey":"dev-server/fleet1-h2pmt-q2plv","message":"","queue":"agones.dev.GameServerControllerCreation","severity":"error","source":"*gameservers.Controller","subqueue":"creation","time":"2023-08-23T14:51:55.795073619Z"}
{"message":"pod \"fleet1-h2pmt-q2plv\" not found","severity":"error","time":"2023-08-23T14:51:55.795191251Z"}
{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:54:40.739622909Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}

^[{"body":"","headers":{"Accept-Encoding":["gzip"],"X-Forwarded-Host":["172.31.47.218:8081"],"X-Forwarded-Proto":["https"],"X-Forwarded-Uri":["/openapi/v3"],"X-Remote-User":["system:aggregator"]},"host":"172.31.47.218:8081","message":"404","method":"GET","requestURI":"/openapi/v3","severity":"warning","source":"*https.Server","time":"2023-08-23T14:56:09.920225154Z","url":{"Scheme":"","Opaque":"","User":null,"Host":"","Path":"/openapi/v3","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""}}

Troubleshooting:

Any help will be appreciated.