Why node_shape and node_shape_config is hard coded?

[safoorsafdar]

Hi,

Hope this finds you well.

I am trying to implement auto-scaler on the OKE, I have few questions that would like learn about

1. Node Shape and configuration is hard coded, is there a way to reconfigure it while provisioning the cluster? https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/c3d5a1aa46795ab50b7e27566d90b5984c9e3b93/modules/oke/autoscaler.tf#L65

2. What the condition of scale up and scale down? is it based on memory or CPU? can we define it using this module?

3. the minimum and maximum sizes for the pool

Thanks in advance.

[hyder]

We chose a minimal nodepool to run the autoscaler pod.

We haven't yet added options to configure the autoscaler. So, it will trigger if k8s cannot schedule pods. If you have ideas/needs, we would like to hear them.

[safoorsafdar]

I have tried this configuration as per the documentation at https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/main/docs/clusterautoscaler.adoc

node_pools = {
    np0 = {
      shape              = "VM.Standard.E4.Flex",
      ocpus              = 4,
      memory             = 16,
      node_pool_size     = 1,
      max_node_pool_size = 5,
      boot_volume_size   = 50,
      autoscale          = true
    }
  }
  autoscaler_pools = {
    # asp_v124 = {
    # }
  }

First time it throw below the error

module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec): Traceback (most recent call last):
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/home/opc/create_autoscaler_pool_taint_list.py", line 23, in <module>
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     list_pools = oce.list_node_pools(compartment_id,cluster_id=cluster_id,name=p)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/oci/container_engine/container_engine_client.py", line 1833, in list_node_pools
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     api_reference_link=api_reference_link)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/oci/retry/retry.py", line 308, in make_retrying_call
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     response = func_ref(*func_args, **func_kwargs)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 476, in call_api
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     return self.request(request, allow_control_chars, operation_name, api_reference_link)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/circuitbreaker.py", line 52, in wrapper
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     return self.call(function, *args, **kwargs)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/circuitbreaker.py", line 67, in call
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     result = func(*args, **kwargs)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 632, in request
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     self.raise_service_error(request, response, service_code, message, operation_name, api_reference_link, target_service, request_endpoint, client_version, timestamp, deserialized_data)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):   File "/usr/lib/python3.6/site-packages/oci/base_client.py", line 801, in raise_service_error
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec):     deserialized_data=deserialized_data)
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec): oci.exceptions.ServiceError: {'target_service': 'container_engine', 'status': 400, 'code': 'InvalidParameter', 'opc-request-id': '4EF255766D9649EE806895A7AAC89FD3/B7F20CE2BACD96B22F6B4C4081D303DD/04D90979CDAE3353B93A3CC948AA8A20', 'message': 'Invalid name', 'operation_name': 'list_node_pools', 'timestamp': '2023-06-16T22:30:20.962887+00:00', 'client_version': 'Oracle-PythonSDK/2.93.1', 'request_endpoint': 'GET https://containerengine.me-jeddah-1.oci.oraclecloud.com/20180222/nodePools', 'logging_tips': 'To get more info on the failing request, refer to https://docs.oracle.com/en-us/iaas/tools/python/latest/logging.html for ways to log the request/response details.', 'troubleshooting_tips': "See https://docs.oracle.com/iaas/Content/API/References/apierrors.htm#apierrors_400__400_invalidparameter for more information about resolving this error. Also see https://docs.oracle.com/iaas/api/#/en/containerengine/20180222/NodePoolSummary/ListNodePools for details on this operation's requirements. If you are unable to resolve this container_engine issue, please contact Oracle support and provide them this full error message."}
module.iwoke.module.extensions.null_resource.taint_nodes[0] (remote-exec): /home/opc/taint_autoscaler_pools.sh: line 5: taint_autoscaler_pool_list.txt: command not found

but on second apply, its created the cluster but cluster-autoscaler pod is in pending state always with below events

  Warning  FailedScheduling  18m   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  13m   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

So here is my question, what the purpose and usage of autoscale=true, and max_node_pool_size=5 attributes when auto-scaler can not be deployed without autscalerpool -> asp*** as it also set in default of variable.

autoscaler_pools = {
    asp_v124 = {
    }
  }

Tx,

[safoorsafdar]

@hyder thanks for your response, so what is your suggestion to maintain min and max number node in a cluster? May be one node pool with static size, and one auto scaler pool?

[hyder]

You can control the size of your other nodepools.

But the nodepool where we run cluster autoscaler, there's really no need to go more than 1. We also create it separately so that it's "unmanaged" i.e. it will not be included in the list of node pools to be managed by the autoscaler. It's purpose is to just run the autoscaler pod. So there's really no need to have its nodepool > 1.

Your other normal workloads is another matter. That you have to determine qnd there're many factors to take into account.

[hyder]

Maybe it's not clear in the docs, we'll check this again but you need to do this to enable the autoscaler nodepool:

autoscaler_pools = { asp_v124 = { } }

Only then will this nodepool be created and the autoscaler pod will be scheduled on the worker node of that pool. Thisnis because of the need to run the autoscaler pod on a nodepool that is "unmanaged" i.e. unmanaged by the cluster autoscaler. That's why we create it separately from the others.

The node pools you specify in the nodepools variable can then be managed by the autoscaler if you set its autoscaler parameter to true.

node_pool_size is used as the size of the nodepool if autoscaler is false and as minimum size if autoscaler is true.

max_node_pool_size is only used when autoscaler is true.

Hope this is helpful. We are working to improve the documentation in 5.x. Can I please ask you to open an issue to better document cluster autoscaler?