Open schwesig opened 1 month ago
Note: Maybe I am overcomplicating things, then please hold me back. But the rules for what resources can when and how reclaimed when in idle mode etc seems to be not trivial to me. And how to inform etc.
from wednesday meeting: planned to discuss in one of the next ones. when e.g. Kim and Naved are back. but, the technical part could be checked/researched already. gives maybe a quick direction: if we can not free allocated GPU, then this billing issue can be reduced in its possible ways.
The prometheus query that gathers pods using GPUs specifically looks for pods that have been assigned a node to run. This means pods that are in pending state are not counted.
My understanding is that once a pod is scheduled then the resources are now claimed. From the kubernetes documention [1]:
The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node. Note that although actual memory or CPU resource usage on nodes is very low, the scheduler still refuses to place a Pod on a node if the capacity check fails.
I would say as soon as a GPU is not usable by others it should be charged for. That being said if it is not clear to a user that they have a GPU claimed by them we should make sure there is a way to alert them to this fact.
This topic captures two aspects:
To discuss/define:
Originally posted by @schwesig in https://github.com/nerc-project/operations/issues/635#issuecomment-2216831790
/CC @msdisme @knikolla @naved001 @joachimweyl @hpdempsey