search

akka / akka-management

Akka Management is a suite of tools for operating Akka Clusters.
https://doc.akka.io/docs/akka-management/
Other
254 stars 160 forks source link

Only request running pods from Kubernetes API #1322

Closed leviramsey closed 2 months ago

leviramsey commented 2 months ago

It seems like we'd be most interested in the pods which are currently running (cc: @girdharshubham ). Without this filter, especially in environments where rollouts are frequent (e.g. dev/test environments), there's a chance that getting so many pods could cause HTTP responses which akka http rejects.

Can hide this behind a config setting if needed, but I'm not sure why one would want to discover pods which used to be there.

leviramsey commented 2 months ago

I assume the link checker failure is expected, post #1319

johanandren commented 2 months ago

Not sure, but, could this cause problems with ready check on a cold cluster start?

leviramsey commented 2 months ago

Not sure, but, could this cause problems with ready check on a cold cluster start?

From a conversation with @girdharshubham, it appears that running does not take readiness into account (viz. pods which aren't ready will show as status.phase==Running):

kubectl --context=arn:aws:eks:us-east-2:xyz:cluster/dev-us-east-2 get pods -n some-uuid --field-selector=status.phase==Running 
NAME                                                READY   STATUS             RESTARTS           AGE
some-name-644cc5669-pwn62   2/3     CrashLoopBackOff   1550 (4m26s ago)   5d9h
some-name-cf679b76b-2hm46   2/3     CrashLoopBackOff   1576 (2m51s ago)   5d11h
some-name-cf679b76b-p8crf   2/3     CrashLoopBackOff   1570 (112s ago)    5d11h
some-name-cf679b76b-zr772   2/3     CrashLoopBackOff   1580 (3m43s ago)   5d11h
leviramsey commented 2 months ago

Note that this does not remove the filtering of the returned results for running pods on our side. It may make sense to leave that in as a "defense in depth".