Required resource setting for injected containers

[addozhang]

Please describe the Improvement and/or Feature Request

Scope (please mark with X where applicable)

• New Functionality [ ]
• Install [ ]
• SMI Traffic Access Policy [ ]
• SMI Traffic Specs Policy [ ]
• SMI Traffic Split Policy [ ]
• Permissive Traffic Policy [ ]
• Ingress [ ]
• Egress [ ]
• Envoy Control Plane [ ]
• CLI Tool [ ]
• Metrics [ ]
• Certificate Management [ ]
• Sidecar Injection [X]
• Logging [ ]
• Debugging [ ]
• Tests [ ]
• CI System [ ]
• Project Release [ ]

Possible use cases

We have quota configured for each namespaces and injected containers have no resourte limitation setting. Bellow error occurs when pod initializing.

Error creating: pods "xxxxx-gray-5444bddb9-cmbgd" is forbidden: [minimum cpu usage per Pod is 10m.  No request is specified., minimum memory usage per Pod is 2Mi.  No request is specified., maximum cpu usage per Pod is 4.  No limit is specified., maximum memory usage per Pod is 6Gi.  No limit is specified., minimum cpu usage per Container is 10m.  No request is specified., minimum memory usage per Container is 2Mi.  No request is specified., maximum memory usage per Container is 6Gi.  No limit is specified., maximum cpu usage per Container is 4.  No limit is specified., minimum cpu usage per Container is 10m.  No request is specified., minimum memory usage per Container is 2Mi.  No request is specified., maximum cpu usage per Container is 4.  No limit is specified., maximum memory usage per Container is 6Gi.  No limit is specified.]

[eduser25]

@addozhang just verified, we are not currently setting limits on envoy, this can be arranged.

However, see minimum cpu usage per Pod is 10m. No request is specified., that seems to come from the very pod you are trying to push. Can you share with us the full yaml of the pod trying to be pushed here? It seems to me the namespace limit also complains of the pod specification itself having no requests/limits set.

I'll see to it we add configurable resource limits for proxies too.

[addozhang]

@eduser25 yes, we have limitranges per namespace. It also has default resource setting for container. But the default config is applied before admission webhook phase. So the injected containers have no default limit applied.

I tried to set resource limit to init and enovy containers, no error occurs.

You can try with below yamls.

apiVersion: v1
kind: LimitRange
metadata:
  creationTimestamp: null
  name: limits-example
spec:
  limits:
  - max:
      cpu: "4"
      memory: 6Gi
    min:
      cpu: 10m
      memory: 2Mi
    type: Pod
  - default:
      cpu: "1"
      memory: 200Mi
    defaultRequest:
      cpu: 10m
      memory: 10Mi
    max:
      cpu: "4"
      memory: 6Gi
    min:
      cpu: 10m
      memory: 2Mi
    type: Container

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

[ckittel]

just verified, we are not currently setting limits on envoy, this can be arranged.

Thanks @eduser25, I'm running into this as well - and am indeed blocked by this from going forward. I have OPA gatekeeper (via Azure Policy) blocking containers, cluster-wide, that do not have resource limits defined. Only option is to add the workload namespace to the "ignore" list -- but that's totally defeating the purpose of the policy specifically targeting the workloads :)

replicaset-controller Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [denied by azurepolicy-container-limits-493a1c442e86216b9926] container <envoy> has no resource limits

[eduser25]

Hey @addozhang @ckittel we just merged https://github.com/openservicemesh/osm/pull/3330 on ToT which should allow OSM to add resource limits and requests for all envoys in the mesh. Should allow to define resource limits/requests for envoy containers that are to be onboarded on the mesh.

[addozhang]

@eduser25 Great job.

[eduser25]

@addozhang, did you observe similar problems/issues with init containers?