[rel-0.10] `go test` flags are added to the cluster-autoscaler binary

[ialidzhikov]

What happened:

$ docker run eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:v0.10.2 /cluster-autoscaler --help

      --test.bench regexp                                                  run only benchmarks matching regexp
      --test.benchmem                                                      print memory allocations for benchmarks
      --test.benchtime d                                                   run each benchmark for duration d (default 1s)
      --test.blockprofile file                                             write a goroutine blocking profile to file
      --test.blockprofilerate rate                                         set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
      --test.count n                                                       run tests and benchmarks n times (default 1)
      --test.coverprofile file                                             write a coverage profile to file
      --test.cpu list                                                      comma-separated list of cpu counts to run each test with
      --test.cpuprofile file                                               write a cpu profile to file
      --test.failfast                                                      do not start new tests after the first test failure
      --test.list regexp                                                   list tests, examples, and benchmarks matching regexp then exit
      --test.memprofile file                                               write a memory profile to file
      --test.memprofilerate rate                                           set memory profiling rate (see runtime.MemProfileRate)
      --test.mutexprofile string                                           write a mutex contention profile to the named file after execution
      --test.mutexprofilefraction int                                      if >= 0, calls runtime.SetMutexProfileFraction() (default 1)
      --test.outputdir dir                                                 write profiles to dir
      --test.parallel n                                                    run at most n tests in parallel (default 6)
      --test.run regexp                                                    run only tests and examples matching regexp
      --test.short                                                         run smaller test suite to save time
      --test.testlogfile file                                              write test action log to file (for use only by cmd/go)
      --test.timeout d                                                     panic test binary after duration d (default 0, timeout disabled) (default 0s)
      --test.trace file                                                    write an execution trace to file
      --test.v                                                             verbose: print additional output

What you expected to happen: No go test flags to be present in the --help of the cluster-autoscaler.

How to reproduce it (as minimally and precisely as possible): See above.

Environment: Version: v0.10.2

[himanshu-kun]

With the latest CA 1.25.0 we don't see this happening.

command output for CA 1.25.0

$ docker run eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:v1.25.0 /cluster-autoscaler --help Unable to find image 'eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:v1.25.0' locally v1.25.0: Pulling from gardener-project/gardener/autoscaler/cluster-autoscaler 8fdb1fc20e24: Pull complete cf404dfdd502: Pull complete Digest: sha256:9e5791b8375fffce15a80369c02645dacea7fd192d0e08aff792507cf0f012de Status: Downloaded newer image for eu.gcr.io/gardener-project/gardener/autoscaler/cluster-autoscaler:v1.25.0 pflag: help requested Usage of /cluster-autoscaler: --add-dir-header If true, adds the file directory to the header of the log messages --address string The address to expose prometheus metrics. (default ":8085") --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true) --aws-use-static-instance-list Should CA fetch instance types in runtime or use a static list. AWS only --balance-similar-node-groups Detect similar node groups and balance the number of nodes between them --balancing-ignore-label MultiStringFlag Specifies a label to ignore in addition to the basic and cloud-provider set of labels when comparing if two node groups are similar (default []) --balancing-label MultiStringFlag Specifies a label to use for comparing if two node groups are similar, rather than the built in heuristics. Setting this flag disables all other comparison logic, and cannot be combined with --balancing-ignore-label. (default []) --cloud-config string The path to the cloud provider configuration file. Empty string for no configuration file. --cloud-provider string Cloud provider type. Available values: [aws,azure,gce,alicloud,cherryservers,cloudstack,baiducloud,magnum,digitalocean,exoscale,externalgrpc,huaweicloud,hetzner,oci,ovhcloud,clusterapi,ionoscloud,kamatera,linode,bizflycloud,brightbox,packet,vultr,tencentcloud,civo,scaleway,rancher] (default "gce") --cloud-provider-gce-l7lb-src-cidrs cidrs CIDRs opened in GCE firewall for L7 LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16) --cloud-provider-gce-lb-src-cidrs cidrs CIDRs opened in GCE firewall for L4 LB traffic proxy & health checks (default 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16) --cluster-name string Autoscaled cluster name, if available --clusterapi-cloud-config-authoritative Treat the cloud-config flag authoritatively (do not fallback to using kubeconfig flag). ClusterAPI only --control-apiserver-burst int Throttling burst configuration for the client to control cluster's apiserver. (default 10) --control-apiserver-qps float Throttling QPS configuration for the client to control cluster's apiserver. (default 5) --cordon-node-before-terminating Should CA cordon nodes before terminating during downscale process --cores-total string Minimum and maximum number of cores in cluster, in the format :. Cluster autoscaler will not scale the cluster beyond these numbers. (default "0:320000") --daemonset-eviction-for-empty-nodes DaemonSet pods will be gracefully terminated from empty nodes --daemonset-eviction-for-occupied-nodes DaemonSet pods will be gracefully terminated from non-empty nodes (default true) --debugging-snapshot-enabled Whether the debugging snapshot of cluster autoscaler feature is enabled --emit-per-nodegroup-metrics If true, emit per node group metrics. --estimator string Type of resource estimator to be used in scale up. Available values: [binpacking] (default "binpacking") --expander string Type of node group expander to be used in scale up. Available values: [random,most-pods,least-waste,price,priority,grpc]. Specifying multiple values separated by commas will call the expanders in succession until there is only one option remaining. Ties still existing after this process are broken randomly. (default "random") --expendable-pods-priority-cutoff int Pods with priority below cutoff will be expendable. They can be killed without any consideration during scale down and they don't cause scale up. Pods with null priority (PodPriority disabled) are non expendable. (default -10) --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are: APIListChunking=true|false (BETA - default=true) APIPriorityAndFairness=true|false (BETA - default=true) APIResponseCompression=true|false (BETA - default=true) APIServerIdentity=true|false (ALPHA - default=false) APIServerTracing=true|false (ALPHA - default=false) AllAlpha=true|false (ALPHA - default=false) AllBeta=true|false (BETA - default=false) AnyVolumeDataSource=true|false (BETA - default=true) AppArmor=true|false (BETA - default=true) CPUManager=true|false (BETA - default=true) CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false) CPUManagerPolicyBetaOptions=true|false (BETA - default=true) CPUManagerPolicyOptions=true|false (BETA - default=true) CSIMigrationAzureFile=true|false (BETA - default=true) CSIMigrationPortworx=true|false (BETA - default=false) CSIMigrationRBD=true|false (ALPHA - default=false) CSIMigrationvSphere=true|false (BETA - default=true) CSINodeExpandSecret=true|false (ALPHA - default=false) CSIVolumeHealth=true|false (ALPHA - default=false) ContainerCheckpoint=true|false (ALPHA - default=false) CronJobTimeZone=true|false (BETA - default=true) CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false) CustomResourceValidationExpressions=true|false (BETA - default=true) DelegateFSGroupToCSIDriver=true|false (BETA - default=true) DevicePlugins=true|false (BETA - default=true) DisableCloudProviders=true|false (ALPHA - default=false) DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false) DownwardAPIHugePages=true|false (BETA - default=true) EndpointSliceTerminatingCondition=true|false (BETA - default=true) ExpandedDNSConfig=true|false (ALPHA - default=false) ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false) GRPCContainerProbe=true|false (BETA - default=true) GracefulNodeShutdown=true|false (BETA - default=true) GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true) HPAContainerMetrics=true|false (ALPHA - default=false) HPAScaleToZero=true|false (ALPHA - default=false) HonorPVReclaimPolicy=true|false (ALPHA - default=false) IPTablesOwnershipCleanup=true|false (ALPHA - default=false) InTreePluginAWSUnregister=true|false (ALPHA - default=false) InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false) InTreePluginAzureFileUnregister=true|false (ALPHA - default=false) InTreePluginGCEUnregister=true|false (ALPHA - default=false) InTreePluginOpenStackUnregister=true|false (ALPHA - default=false) InTreePluginPortworxUnregister=true|false (ALPHA - default=false) InTreePluginRBDUnregister=true|false (ALPHA - default=false) InTreePluginvSphereUnregister=true|false (ALPHA - default=false) JobMutableNodeSchedulingDirectives=true|false (BETA - default=true) JobPodFailurePolicy=true|false (ALPHA - default=false) JobReadyPods=true|false (BETA - default=true) JobTrackingWithFinalizers=true|false (BETA - default=true) KMSv2=true|false (ALPHA - default=false) KubeletCredentialProviders=true|false (BETA - default=true) KubeletInUserNamespace=true|false (ALPHA - default=false) KubeletPodResources=true|false (BETA - default=true) KubeletPodResourcesGetAllocatable=true|false (BETA - default=true) KubeletTracing=true|false (ALPHA - default=false) LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true) LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (BETA - default=true) LogarithmicScaleDown=true|false (BETA - default=true) MatchLabelKeysInPodTopologySpread=true|false (ALPHA - default=false) MaxUnavailableStatefulSet=true|false (ALPHA - default=false) MemoryManager=true|false (BETA - default=true) MemoryQoS=true|false (ALPHA - default=false) MinDomainsInPodTopologySpread=true|false (BETA - default=false) MixedProtocolLBService=true|false (BETA - default=true) MultiCIDRRangeAllocator=true|false (ALPHA - default=false) NetworkPolicyStatus=true|false (ALPHA - default=false) NodeInclusionPolicyInPodTopologySpread=true|false (ALPHA - default=false) NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false) NodeSwap=true|false (ALPHA - default=false) OpenAPIEnums=true|false (BETA - default=true) OpenAPIV3=true|false (BETA - default=true) PodAndContainerStatsFromCRI=true|false (ALPHA - default=false) PodDeletionCost=true|false (BETA - default=true) PodDisruptionConditions=true|false (ALPHA - default=false) PodHasNetworkCondition=true|false (ALPHA - default=false) ProbeTerminationGracePeriod=true|false (BETA - default=true) ProcMountType=true|false (ALPHA - default=false) ProxyTerminatingEndpoints=true|false (ALPHA - default=false) QOSReserved=true|false (ALPHA - default=false) ReadWriteOncePod=true|false (ALPHA - default=false) RecoverVolumeExpansionFailure=true|false (ALPHA - default=false) RemainingItemCount=true|false (BETA - default=true) RetroactiveDefaultStorageClass=true|false (ALPHA - default=false) RotateKubeletServerCertificate=true|false (BETA - default=true) SELinuxMountReadWriteOncePod=true|false (ALPHA - default=false) SeccompDefault=true|false (BETA - default=true) ServerSideFieldValidation=true|false (BETA - default=true) ServiceIPStaticSubrange=true|false (BETA - default=true) ServiceInternalTrafficPolicy=true|false (BETA - default=true) SizeMemoryBackedVolumes=true|false (BETA - default=true) StatefulSetAutoDeletePVC=true|false (ALPHA - default=false) StorageVersionAPI=true|false (ALPHA - default=false) StorageVersionHash=true|false (BETA - default=true) TopologyAwareHints=true|false (BETA - default=true) TopologyManager=true|false (BETA - default=true) UserNamespacesStatelessPodsSupport=true|false (ALPHA - default=false) VolumeCapacityPriority=true|false (ALPHA - default=false) WinDSR=true|false (ALPHA - default=false) WinOverlay=true|false (BETA - default=true) WindowsHostProcessContainers=true|false (BETA - default=true) --gce-concurrent-refreshes int Maximum number of concurrent refreshes per cloud object type. (default 1) --gce-expander-ephemeral-storage-support Whether scale-up takes ephemeral storage resources into account for GCE cloud provider --gpu-total MultiStringFlag Minimum and maximum number of different GPUs in cluster, in the format ::. Cluster autoscaler will not scale the cluster beyond these numbers. Can be passed multiple times. CURRENTLY THIS FLAG ONLY WORKS ON GKE. (default []) --grpc-expander-cert string Path to cert used by gRPC server over TLS --grpc-expander-url string URL to reach gRPC expander server. --ignore-daemonsets-utilization Should CA ignore DaemonSet pods when calculating resource utilization for scaling down --ignore-mirror-pods-utilization Should CA ignore Mirror pods when calculating resource utilization for scaling down --ignore-taint MultiStringFlag Specifies a taint to ignore in node templates when considering to scale a node group (default []) --initial-node-group-backoff-duration duration initialNodeGroupBackoffDuration is the duration of first backoff after a new node failed to start. (default 5m0s) --kubeconfig string Path to kubeconfig file with authorization and master location information. --kubernetes string Kubernetes master location. Leave blank for default --leader-elect Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability. (default true) --leader-elect-lease-duration duration The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled. (default 15s) --leader-elect-renew-deadline duration The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than or equal to the lease duration. This is only applicable if leader election is enabled. (default 10s) --leader-elect-resource-lock string The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'. (default "leases") --leader-elect-resource-name string The name of resource object that is used for locking during leader election. (default "cluster-autoscaler") --leader-elect-resource-namespace string The namespace of resource object that is used for locking during leader election. --leader-elect-retry-period duration The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled. (default 2s) --log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log-dir string If non-empty, write log files in this directory (no effect when -logtostderr=true) --log-file string If non-empty, use this log file (no effect when -logtostderr=true) --log-file-max-size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --max-autoprovisioned-node-group-count int The maximum number of autoprovisioned groups in the cluster. (default 15) --max-bulk-soft-taint-count int Maximum number of nodes that can be tainted/untainted PreferNoSchedule at the same time. Set to 0 to turn off such tainting. (default 10) --max-bulk-soft-taint-time duration Maximum duration of tainting/untainting nodes as PreferNoSchedule at the same time. (default 3s) --max-drain-parallelism int Maximum number of nodes needing drain, that can be drained and deleted in parallel. (default 1) --max-empty-bulk-delete int Maximum number of empty nodes that can be deleted at the same time. (default 10) --max-failing-time duration Maximum time from last recorded successful autoscaler run before automatic restart (default 15m0s) --max-graceful-termination-sec int Maximum number of seconds CA waits for pod termination when trying to scale down a node. (default 600) --max-inactivity duration Maximum time from last recorded autoscaler activity before automatic restart (default 10m0s) --max-node-group-backoff-duration duration maxNodeGroupBackoffDuration is the maximum backoff duration for a NodeGroup after new nodes failed to start. (default 30m0s) --max-node-provision-time duration Maximum time CA waits for node to be provisioned (default 15m0s) --max-nodegroup-binpacking-duration duration Maximum time that will be spent in binpacking simulation for each NodeGroup. (default 10s) --max-nodes-per-scaleup int Max nodes added in a single scale-up. This is intended strictly for optimizing CA algorithm latency and not a tool to rate-limit scale-up throughput. (default 1000) --max-nodes-total int Maximum number of nodes in all node groups. Cluster autoscaler will not grow the cluster beyond this number. --max-pod-eviction-time duration Maximum time CA tries to evict a pod before giving up (default 2m0s) --max-scale-down-parallelism int Maximum number of nodes (both empty and needing drain) that can be deleted in parallel. (default 10) --max-total-unready-percentage float Maximum percentage of unready nodes in the cluster. After this is exceeded, CA halts operations (default 45) --memory-total string Minimum and maximum number of gigabytes of memory in cluster, in the format :. Cluster autoscaler will not scale the cluster beyond these numbers. (default "0:6400000") --min-replica-count int Minimum number or replicas that a replica set or replication controller should have to allow their pods deletion in scale down --min-resync-period duration The minimum resync period configured for the shared informers used by the MCM provider cached listers (default 1h0m0s) --namespace string Namespace in which cluster-autoscaler run. (default "kube-system") --new-pod-scale-up-delay duration Pods less than this old will not be considered for scale-up. (default 0s) --node-autoprovisioning-enabled Should CA autoprovision node groups when needed --node-deletion-delay-timeout duration Maximum time CA waits for removing delay-deletion.cluster-autoscaler.kubernetes.io/ annotations before deleting the node. (default 2m0s) --node-group-auto-discovery :[[=]] One or more definition(s) of node group auto-discovery. A definition is expressed :[[=]]. The `aws` and `gce` cloud providers are currently supported. AWS matches by ASG tags, e.g. `asg:tag=tagKey,anotherTagKey`. GCE matches by IG name prefix, and requires you to specify min and max nodes per IG, e.g. `mig:namePrefix=pfx,min=0,max=10` Can be used multiple times. (default []) --node-group-backoff-reset-timeout duration nodeGroupBackoffResetTimeout is the time after last failed scale-up when the backoff duration is reset. (default 3h0m0s) --node-info-cache-expire-time duration Node Info cache expire time for each item. Default value is 10 years. (default 87600h0m0s) --nodes MultiStringFlag sets min,max size and other configuration data for a node group in a format accepted by cloud provider. Can be used multiple times. Format: :: (default []) --ok-total-unready-count int Number of allowed unready nodes, irrespective of max-total-unready-percentage (default 3) --one-output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true) --profiling Is debug/pprof endpoint enabled --record-duplicated-events enable duplication of similar events within a 5 minute window. --regional Cluster is regional. --scale-down-candidates-pool-min-count int Minimum number of nodes that are considered as additional non empty candidatesfor scale down when some candidates from previous iteration are no longer valid.When calculating the pool size for additional candidates we takemax(#nodes * scale-down-candidates-pool-ratio, scale-down-candidates-pool-min-count). (default 50) --scale-down-candidates-pool-ratio float A ratio of nodes that are considered as additional non empty candidates forscale down when some candidates from previous iteration are no longer valid.Lower value means better CA responsiveness but possible slower scale down latency.Higher value can affect CA performance with big clusters (hundreds of nodes).Set to 1.0 to turn this heuristics off - CA will take all nodes as additional candidates. (default 0.1) --scale-down-delay-after-add duration How long after scale up that scale down evaluation resumes (default 10m0s) --scale-down-delay-after-delete duration How long after node deletion that scale down evaluation resumes, defaults to scanInterval (default 0s) --scale-down-delay-after-failure duration How long after scale down failure that scale down evaluation resumes (default 3m0s) --scale-down-enabled Should CA scale down the cluster (default true) --scale-down-gpu-utilization-threshold float Sum of gpu requests of all pods running on the node divided by node's allocatable resource, below which a node can be considered for scale down.Utilization calculation only cares about gpu resource for accelerator node. cpu and memory utilization will be ignored. (default 0.5) --scale-down-non-empty-candidates-count int Maximum number of non empty nodes considered in one iteration as candidates for scale down with drain.Lower value means better CA responsiveness but possible slower scale down latency.Higher value can affect CA performance with big clusters (hundreds of nodes).Set to non positive value to turn this heuristic off - CA will not limit the number of nodes it considers. (default 30) --scale-down-unneeded-time duration How long a node should be unneeded before it is eligible for scale down (default 10m0s) --scale-down-unready-time duration How long an unready node should be unneeded before it is eligible for scale down (default 20m0s) --scale-down-utilization-threshold float Sum of cpu or memory of all pods running on the node divided by node's corresponding allocatable resource, below which a node can be considered for scale down (default 0.5) --scale-up-from-zero Should CA scale up when there 0 ready nodes. (default true) --scan-interval duration How often cluster is reevaluated for scale up or down (default 10s) --skip-headers If true, avoid header prefixes in the log messages --skip-log-headers If true, avoid headers when opening log files (no effect when -logtostderr=true) --skip-nodes-with-local-storage If true cluster autoscaler will never delete nodes with pods with local storage, e.g. EmptyDir or HostPath (default true) --skip-nodes-with-system-pods If true cluster autoscaler will never delete nodes with pods from kube-system (except for DaemonSet or mirror pods) (default true) --status-config-map-name string Status configmap name (default "cluster-autoscaler-status") --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2) --target-apiserver-burst int Throttling burst configuration for the client to target cluster's apiserver. (default 10) --target-apiserver-qps float Throttling QPS configuration for the client to target cluster's apiserver. (default 5) --unremovable-node-recheck-timeout duration The timeout before we check again a node that couldn't be removed before (default 5m0s) --user-agent string User agent used for HTTP calls. (default "cluster-autoscaler") -v, --v Level number for the log level verbosity --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging --write-status-configmap Should CA write status information to a configmap (default true)

We don't support rel v0.10 now, so won't be picking it.

[himanshu-kun]

/close