search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.05k stars 3.97k forks source link

VPA not producing reasonable recommendations on memory for Cronjobs #3975

Closed rdubya16 closed 1 year ago

rdubya16 commented 3 years ago

Which component are you using?: VPA Recommender What version of the component are you using?: 0.8.1 What k8s version are you using (kubectl version)?:

➜ vpa kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T18:43:01Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?: EKS What did you expect to happen?: Recommendations that make sense What happened instead?: Recommendations dont match recorded metrics from prometheus

We are noticing after configuring VPA recommender for 100+ cronjobs the results we are getting dont seem to match the memory used by the pods. Many of our cronjobs are very short lived and ive found a few notes around that say it doesnt work well for jobs that run under a minute. I think this explains the majority of our cases but im still left with multiple examples of jobs that run for multiple minutes but their recommendations are much higher or lower than the expected values.

One other thing I noticed is that multiple cronjobs all seem to get the exact same target memory value. Should this be possible? These are all different cronjobs that run different workloads. Not sure how they could end up with all the same target values. Is the issue here the cronjobs only run for a few minutes a day and there isnt enough data to work with?

628694953 job1
628694953 job2
628694953 job3
628694953 job4
628694953 job5
628694953 job6
628694953 job7
628694953 job8
628694953 job8
628694953 job9
628694953 job10
628694953 job11
628694953 job12

  name: job1
  namespace: default
  resourceVersion: "48748421"
  selfLink: /apis/autoscaling.k8s.io/v1/namespaces/default/verticalpodautoscalers/analytics-extracts-perform
  uid: 9e13d83f-87af-47c8-9e86-df99b0144818
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      controlledResources:
      - memory
      maxAllowed:
        memory: 2500Mi
      minAllowed:
        memory: 250Mi
  targetRef:
    apiVersion: batch/v1beta1
    kind: CronJob
    name: job1
  updatePolicy:
    updateMode: "Off"
status:
  conditions:
  - lastTransitionTime: "2021-03-25T13:26:00Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: rails
      lowerBound:
        memory: "293363252"
      target:
        memory: "628694953"
      uncappedTarget:
        memory: "628694953"
      upperBound:
        memory: 2500Mi

  name: job2
  namespace: default
  resourceVersion: "48748421"
  selfLink: /apis/autoscaling.k8s.io/v1/namespaces/default/verticalpodautoscalers/analytics-extracts-perform
  uid: 9e13d83f-87af-47c8-9e86-df99b0144818
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      controlledResources:
      - memory
      maxAllowed:
        memory: 2500Mi
      minAllowed:
        memory: 250Mi
  targetRef:
    apiVersion: batch/v1beta1
    kind: CronJob
    name: job2
  updatePolicy:
    updateMode: "Off"
status:
  conditions:
  - lastTransitionTime: "2021-03-25T13:26:00Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: rails
      lowerBound:
        memory: "293363252"
      target:
        memory: "628694953"
      uncappedTarget:
        memory: "628694953"
      upperBound:
        memory: 2500Mi
fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

rdubya16 commented 3 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mike-stewart commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

mike-stewart commented 2 years ago

/remove-lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mike-stewart commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/autoscaler/issues/3975#issuecomment-1344974383): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.