gameserveer unexpectedly restarts all services

[aimuz]

What happened:

Without doing any updates, all my gs are restarted and the age is the same

➜  ~ kubectl -n pg-core-test get gs
NAME                        STATE   ADDRESS       PORT   NODE       AGE
battle-server-8kq7r-47wqv   Ready   10.77.0.215   7749   worker03   4m48s
battle-server-8kq7r-487dd   Ready   10.77.0.215   7158   worker03   5m17s
battle-server-8kq7r-9xzch   Ready   10.77.0.215   7173   worker03   5m17s
battle-server-8kq7r-qp2xr   Ready   10.77.0.215   7091   worker03   5m17s
battle-server-8kq7r-wp229   Ready   10.77.0.215   7524   worker03   5m17s
➜  ~ kubectl -n pg-core get gs
NAME                        STATE   ADDRESS       PORT   NODE       AGE
battle-server-mlgmp-c7m7f   Ready   10.77.0.215   7933   worker03   5m12s
battle-server-mlgmp-f5f2n   Ready   10.77.0.215   7049   worker03   5m20s
battle-server-mlgmp-tg4wl   Ready   10.77.0.215   7824   worker03   5m20s
battle-server-mlgmp-vcsqq   Ready   10.77.0.215   7720   worker03   5m20s
battle-server-mlgmp-xfwss   Ready   10.77.0.215   7196   worker03   5m20s
➜  ~

➜  ~ kubectl -n pg-core get events | grep Shutdown
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-25sdv   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-25sdv   Deleting Pod battle-server-mlgmp-25sdv
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-57c97   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-57c97   Deleting Pod battle-server-mlgmp-57c97
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-77dbs   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-77dbs   Deleting Pod battle-server-mlgmp-77dbs
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-bwtrq   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-bwtrq   Deleting Pod battle-server-mlgmp-bwtrq
4m54s       Normal    Shutdown           gameserver/battle-server-mlgmp-cj9h6   Deletion started
4m54s       Normal    Shutdown           gameserver/battle-server-mlgmp-cj9h6   Deleting Pod battle-server-mlgmp-cj9h6
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-cptvf   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-cptvf   Deleting Pod battle-server-mlgmp-cptvf
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f2gqt   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f2gqt   Deleting Pod battle-server-mlgmp-f2gqt
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f6wnw   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f6wnw   Deleting Pod battle-server-mlgmp-f6wnw
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-flh8q   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-flh8q   Deleting Pod battle-server-mlgmp-flh8q
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-hchzw   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-hchzw   Deleting Pod battle-server-mlgmp-hchzw
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-jl542   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-jl542   Deleting Pod battle-server-mlgmp-jl542
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-kw5c5   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-kw5c5   Deleting Pod battle-server-mlgmp-kw5c5
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-lv6pv   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-lv6pv   Deleting Pod battle-server-mlgmp-lv6pv
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-mxk8j   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-mxk8j   Deleting Pod battle-server-mlgmp-mxk8j
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-pvxkq   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-pvxkq   Deleting Pod battle-server-mlgmp-pvxkq
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-qbmsz   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-qbmsz   Deleting Pod battle-server-mlgmp-qbmsz
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-rljxn   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-rljxn   Deleting Pod battle-server-mlgmp-rljxn
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-tcd98   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-tcd98   Deleting Pod battle-server-mlgmp-tcd98
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-tvhjw   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-tvhjw   Deleting Pod battle-server-mlgmp-tvhjw
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-xc8rc   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-xc8rc   Deleting Pod battle-server-mlgmp-xc8rc
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-zhzdx   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-zhzdx   Deleting Pod battle-server-mlgmp-zhzdx

What you expected to happen:

Curious as to why this is happening, is this a bug or a feature?

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Agones version: 1.30.0
• Kubernetes version (use kubectl version): v1.24.6
• Cloud provider or hardware configuration:
• Install method (yaml/helm): helm
• Troubleshooting guide log(s):
• Others:

[zmerlynn]

That's curious, for sure. And there's nothing that could have triggered the game server itself to call Shutdown() (simultaneously)?

[aimuz]

No call, He is concentrated around 10:00 Eastern time, around 2:00 UTC

[aimuz]

A new day reappeared, it seems that the problem appeared when I upgraded to 1.30, I forget what version I belonged to before

Details

```text ➜ ~ date Thu May 11 09:43:04 CST 2023 ➜ ~ kubectl -n pg-core get gs NAME STATE ADDRESS PORT NODE AGE battle-server-mlgmp-4ljq5 Ready 10.77.0.215 7176 worker03 3m50s battle-server-mlgmp-dq25q Ready 10.77.0.215 7153 worker03 3m50s battle-server-mlgmp-hxkzk Ready 10.77.0.215 7094 worker03 3m50s battle-server-mlgmp-rkl5s Ready 10.77.0.215 7144 worker03 3m50s battle-server-mlgmp-tdpk2 Ready 10.77.0.215 7023 worker03 3m50s ➜ ~ ➜ ~ ➜ ~ ➜ ~ ➜ ~ kubectl -n pg-core-test get gs NAME STATE ADDRESS PORT NODE AGE battle-server-8kq7r-75v95 Ready 10.77.0.215 7563 worker03 4m7s battle-server-8kq7r-cswz7 Ready 10.77.0.215 7322 worker03 8s battle-server-8kq7r-l7ddv Ready 10.77.0.215 7691 worker03 4m8s battle-server-8kq7r-mgqrg Ready 10.77.0.215 7785 worker03 4m8s battle-server-8kq7r-pc7jb Ready 10.77.0.215 7316 worker03 4m8s battle-server-8kq7r-tgrsl Allocated 10.77.0.215 7922 worker03 4m8s ```

[markmandel]

Did a node do a Kubernetes version upgrade?

[aimuz]

As far as I know no operations have been done and the problem has not occurred in the last few days

[markmandel]

Probably too late, but would be interesting to see the Pod or Node events - that would probably give us a better idea on why the Pod was shutdown.

[aimuz]

When I find the problem again, I'll add the pod and node-related events

[github-actions[bot]]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[github-actions[bot]]

This issue is marked as obsolete due to inactivity for last 60 days. To avoid issue getting closed in next 30 days, please add a comment or add 'awaiting-maintainer' label. Thank you for your contributions