Add option to increase the resources on init containers to prevent OOM

[andrey-dubnik]

Hi,

We have added PGSQL dependency to the ISPN in a following way

  dependencies:
    artifacts:
    - maven: org.postgresql:postgresql:42.7.3

Which resulted in OOM as 64Mi is not enough for the container to work

 external-artifacts-download:
    Container ID:   containerd://84afdcde8f73d7f43df8ca14ea9fadfa74a332a7fe6810f99ffb44d34ac9854c
    Image:          quay.io/infinispan/server:15.0.0.Final
    Image ID:       quay.io/infinispan/server@sha256:7d49103730a73f04584e5a0da8a11175e7ed9775407bda93276927a5bc9bfa35
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 16 May 2024 14:18:18 +0100
      Finished:     Thu, 16 May 2024 14:19:18 +0100
    Ready:          False
    Restart Count:  21
    Limits:
      cpu:     30m
      memory:  64Mi
    Requests:
      cpu:     30m
      memory:  64Mi
    Environment:
      SERVER_LIBS:      org.postgresql:postgresql:42.7.3
      SERVER_LIBS_DIR:  server/lib/external-artifacts
      INIT_CONTAINER:   TRUE
      MANAGED_ENV:      TRUE

CPU and RAM does not seem to be configurable accordingly to https://github.com/infinispan/infinispan-operator/blob/a19b424f66a122cf30a220aa80944c46e190427b/pkg/reconcile/pipeline/infinispan/handler/provision/dependencies.go#L49

Does it makes sense adding the option to configure CPU & RAM for the specific init containers?

Thanks

[ryanemerson]

@andrey-dubnik By default we do not specify the Limits and Requests field of the initContainer. Do you have a LimitRange in place which is configuring the default values?

[andrey-dubnik]

Missed that bit and we do actually, it is set by our OPS team... whatever comes beyond that limit has to have the overrides in-place. Does it makes sense to have the configuration option to apply overrides for ISPN?

[ryanemerson]

The simplest solution, but maybe not the easiest :slightly_smiling_face:, would be to request your OPs team to increase the LimitRange values for your Infinispan namespace. Or create a dedicated namespace with a higher LimitRange purely for hosting Infinispan. I appreciate that might be a tough ask though.

On the Operator side we could add something like the following to the Infinispan CR:

spec:
  dependencies:
    initContainer:
      cpu: "2000m:1000m"
      memory: "2Gi:1Gi"

Alternatively, we could just use the same limit/requests configured for the server pods. This is potentially wasteful as we only execute the Infinispan CLI, however the initContainer is very short-lived and this has the added benefit of not requiring any user configuration.

@rigazilla Any thoughts?

[andrey-dubnik]

I was thinking in a direction of the Operator configuration option similar to what you have suggested which would address wider use cases where OPS may not be very flexible in that sense (I'm about to find out).

[rigazilla]

@rigazilla Any thoughts?

I also would go with new conf options, if not provided OPs defaults applies

[andrey-dubnik]

Would you like a PR for that change?

[ryanemerson]

That would be great @andrey-dubnik

[andrey-dubnik]

@ryanemerson have made the changes, so-far it is working fine.