Support container creation when resource quotas are hit

joyrex2001 / kubedock

Kubedock is a minimal implementation of the docker api that will orchestrate containers on a Kubernetes cluster, rather than running containers locally.

MIT License

226 stars 33 forks source link

Support container creation when resource quotas are hit #87

Open Dutchy- opened 7 months ago

Dutchy- commented 7 months ago

Currently, when kubedock creates pods and you hit a resource quota, the request fails with this error:

{"message":"pods \"kubedock-a4a0dbae1bd2\" is forbidden: exceeded quota: tenant-quota, requested: limits.cpu=2, used: limits.cpu=31650m, limited: limits.cpu=32"}

Context: we run kubedock as a sidecontainer on Tekton to support java testcontainers.

Is it possible to support creating pods in this situation without having the request fail immediately?

Some solutions I considered:

Use a Deployment instead of a Pod to schedule the container
Use a retry mechanism

And maybe there are other options?

joyrex2001 commented 7 months ago

In earlier versions kubedock was using deployments, and had the option to use jobs. Deployments were problematic when the orchestrated containers were actually one-off jobs (jobs did solve that).

A retry option might make sense and probably leads to increased succes when orchestrating, but tests might still fail because of time-outs in the actual tests as well.

Something that might work already, is setting lower requests and limits (via labels, global settings, or a global pod template).

Dutchy- commented 7 months ago

Yes, lowering requests and limits relieves the problem temporarily, and it's definitely something we're doing now to work around the problem, but it does not solve the problem permanently - even on lower limits the pod might hit the quota and get denied.