ansible / awx-resource-operator

41 stars 31 forks source link

Set the max-reconciles to 15 to throttle the jobs so the operator does not crash #150

Closed rebeccahhh closed 7 months ago

rebeccahhh commented 7 months ago

The resource operator was not throttling the jobs, so it 300 are created all at once, the since operator container tries to reconcile them all at one, which make it get OOMKilled. We limit the number of concurrent reconciliations for the operator container to 5 so that it will only try to process 5 at a time. That should help the OOMKilled bit.

This can be overridden by the user if desired on the resource-operator-controller-manager deployment resource.

oc edit deployment resource-operator-controller-manager
  env:
    # This value is used
    - name: MAX_CONCURRENT_RECONCILES_ANSIBLEJOB_TOWER_ANSIBLE_COM
      value: "10" 

If a user wants to increase the number of concurrent reconciles, they should also increase the memory and cpu for that operator, which can also be done on the operator deployment

oc edit deployment resource-operator-controller-manager
    resources:
      requests:
        memory: "1Gi"
        cpu: "250m"
      limits:
        memory: "4Gi"
        cpu: "500m"