search

PlayFab / thundernetes

Thundernetes makes it easy to run your game servers on Kubernetes
https://playfab.github.io/thundernetes
Apache License 2.0
307 stars 46 forks source link

Add controller performance metrics #391

Closed dsmith111 closed 1 year ago

dsmith111 commented 1 year ago

Problem: Currently we do not have any metrics in place to track the performance of controlling/reconciling GameServers: #361

Solution: This PR adds in 2 new Prometheus metrics:

These new metrics can potentially show any issues in the controller itself (time taken to begin GameServer creation) as well as issues related to the servers themselves, or performance of the cluster (time taken to complete GameServer initialization).

Testing used to verify the change: I created a temporary custom Docker image to build the thundernetes controller manager; using the netcore sample GameServerBuild, I proceeded to:

  1. Scale StandBy GS up
  2. Allocated a server
  3. Gracefully terminated an active server
  4. Scaled StandBy GS back down

All of these events were monitored within the modified Grafana dashboard:

image
dsmith111 commented 1 year ago

@dgkanatsios In the excel file the description for this issue was:

We should add some performance metrics to controllers, like how much time it takes to reconcile, how many servers were processed etc. We should also include the relevant Geneva charts.

Did you mean Grafana charts, or should this connect to an existing (or to be built) Geneva/Jarvis system?

dgkanatsios commented 1 year ago

@dsmith111 thank you for the PR! The charts were mean to be Grafana, which aligns with your PR, thanks!

dgkanatsios commented 1 year ago

A couple of comments, then I think we're good to merge! Appreciate all the work and the communication, thank you!

dgkanatsios commented 1 year ago

Also FYI @abbasahmed and @ghov since they have #414 open with changes to the Grafana dashboards. You'll need to rebase after we merge this one.

dsmith111 commented 1 year ago

@dgkanatsios I believe that's all of the current comments wrapped up

dgkanatsios commented 1 year ago

Really appreciate all the effort and the discussion, will merge as soon as tests pass! Thank you!