Karpenter controller can't schedule/add any nodes

bhavitsharma commented 12 months ago

Description

Observed Behavior: Karpenter controller can't schedule any nodes in my cluster. The following are the logs I see in the controller pod:

$ kubectl logs -f -c controller -l app.kubernetes.io/name=karpenter
2023-09-26T10:00:27.381Z    DEBUG   controller.pricing  pricing/pricing.go:382  updated spot pricing with instance types and offerings  {"commit": "637a642", "instance-type-count": 702, "offering-count": 3315}
2023-09-26T10:00:27.779Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:00:27.781Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219

Expected Behavior: Karpenter to have added new nodes, but it didn't.

Reproduction Steps (Please include YAML):

Versions:

Chart Version: v0.30.0

Kubernetes Version (kubectl version):

$ kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.4-eks-2d98532

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

bhavitsharma commented 12 months ago

This is my provisioner YAML. $ kubectl get provisioners.karpenter.sh --all-namespaces -o yaml

apiVersion: v1
items:
- apiVersion: karpenter.sh/v1alpha5
  kind: Provisioner
  metadata:
    annotations:
      karpenter.sh/provisioner-hash: "1599235016353422174"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"karpenter.sh/v1alpha5","kind":"Provisioner","metadata":{"labels":{"app.kubernetes.io/managed-by":"pulumi"},"name":"default"},"spec":{"providerRef":{"name":"aws"},"requirements":[{"key":"karpenter.k8s.aws/instance-categor
y","operator":"In","values":["m","r","c"]},{"key":"karpenter.k8s.aws/instance-cpu","operator":"In","values":["4","8","16","32"]}]}}
    creationTimestamp: "2023-09-26T09:10:55Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: pulumi
    name: default
    resourceVersion: "130107"
    uid: a7b41239-623d-4a50-a040-dfcdae716968
  spec:
    providerRef:
      name: aws
    requirements:
    - key: karpenter.k8s.aws/instance-category
      operator: In
      values:
      - m
      - r
      - c
    - key: karpenter.k8s.aws/instance-cpu
      operator: In
      values:
      - "4"
      - "8"
      - "16"
      - "32"
    - key: kubernetes.io/os
      operator: In
      values:
      - linux
    - key: kubernetes.io/arch
      operator: In
      values:
      - amd64
    - key: karpenter.sh/capacity-type
      operator: In
      values:
      - on-demand
kind: List
metadata:
  resourceVersion: ""

These are my logs: The rest of the logs are pretty much the same.

2023-09-26T10:00:10.460Z    DEBUG   controller  controllers/controllers.go:47   discovered version  {"commit": "637a642", "version": "v0.30.0"}
2023-09-26T10:00:10.461Z    DEBUG   controller  zapio/writer.go:146 Registering 1 clients   {"commit": "637a642"}
2023-09-26T10:00:10.461Z    DEBUG   controller  zapio/writer.go:146 Registering 2 informer factories    {"commit": "637a642"}
2023-09-26T10:00:10.461Z    DEBUG   controller  zapio/writer.go:146 Registering 3 informers {"commit": "637a642"}
2023-09-26T10:00:10.461Z    DEBUG   controller  zapio/writer.go:146 Registering 5 controllers   {"commit": "637a642"}
2023-09-26T10:00:10.463Z    INFO    controller  runtime/asm_amd64.s:1650    Starting server {"commit": "637a642", "path": "/metrics", "kind": "metrics", "addr": "[::]:8000"}
2023-09-26T10:00:10.464Z    INFO    controller  runtime/asm_amd64.s:1650    Starting server {"commit": "637a642", "kind": "health probe", "addr": "[::]:8081"}
2023-09-26T10:00:10.565Z    INFO    controller  leaderelection/leaderelection.go:248    attempting to acquire leader lease karpenter/karpenter-leader-election...
    {"commit": "637a642"}
2023-09-26T10:00:10.591Z    INFO    controller  injection/injection.go:63   Starting informers...   {"commit": "637a642"}
2023-09-26T10:00:11.448Z    DEBUG   controller  controllers/controllers.go:47   discovered version  {"commit": "637a642", "version": "v0.30.0"}
2023-09-26T10:00:11.448Z    DEBUG   controller  zapio/writer.go:146 Registering 1 clients   {"commit": "637a642"}
2023-09-26T10:00:11.448Z    DEBUG   controller  zapio/writer.go:146 Registering 2 informer factories    {"commit": "637a642"}
2023-09-26T10:00:11.448Z    DEBUG   controller  zapio/writer.go:146 Registering 3 informers {"commit": "637a642"}
2023-09-26T10:00:11.448Z    DEBUG   controller  zapio/writer.go:146 Registering 5 controllers   {"commit": "637a642"}
2023-09-26T10:00:11.449Z    INFO    controller  runtime/asm_amd64.s:1650    Starting server {"commit": "637a642", "path": "/metrics", "kind": "metrics", "addr": "[::]:8000"}
2023-09-26T10:00:11.449Z    INFO    controller  runtime/asm_amd64.s:1650    Starting server {"commit": "637a642", "kind": "health probe", "addr": "[::]:8081"}
2023-09-26T10:00:11.550Z    INFO    controller  leaderelection/leaderelection.go:248    attempting to acquire leader lease karpenter/karpenter-leader-election...
    {"commit": "637a642"}
2023-09-26T10:00:11.578Z    INFO    controller  injection/injection.go:63   Starting informers...   {"commit": "637a642"}
2023-09-26T10:00:26.767Z    INFO    controller  leaderelection/leaderelection.go:258    successfully acquired lease karpenter/karpenter-leader-election
    {"commit": "637a642"}
2023-09-26T10:00:26.768Z    INFO    controller.provisioner  controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.768Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-09-26T10:00:26.768Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod"}
2023-09-26T10:00:26.768Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-09-26T10:00:26.768Z    INFO    controller.deprovisioning   controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.768Z    DEBUG   controller.deprovisioning   deprovisioning/controller.go:124    waiting on cluster sync {"commit": "637a642"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "kind source: *v1.DaemonSet"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "provisioner.hash", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "provisioner.hash", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-09-26T10:00:26.769Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod"}
2023-09-26T10:00:26.770Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.770Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-09-26T10:00:26.771Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "termination", "controllerGroup": "", "controllerKind": "Node"}
2023-09-26T10:00:26.772Z    INFO    controller.metric_scraper   controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.772Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-09-26T10:00:26.773Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-09-26T10:00:26.773Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.773Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.773Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-09-26T10:00:26.773Z    INFO    controller.machine.garbagecollection    controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "machine.lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Provisioner"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Pod"}
2023-09-26T10:00:26.774Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-09-26T10:00:26.775Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-09-26T10:00:26.775Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "machine.termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-09-26T10:00:26.775Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "machine.termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-09-26T10:00:26.775Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate", "source": "kind source: *v1alpha1.AWSNodeTemplate"}
2023-09-26T10:00:26.775Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate"}
2023-09-26T10:00:26.776Z    INFO    controller  controller/controller.go:242    Starting EventSource    {"commit": "637a642", "controller": "lease.garbagecollection", "controllerGroup": "coordination.k8s.io", "controllerKind": "Lease", "source": "kind source: *v1.Lease"}
2023-09-26T10:00:26.776Z    INFO    controller  controller/controller.go:242    Starting Controller {"commit": "637a642", "controller": "lease.garbagecollection", "controllerGroup": "coordination.k8s.io", "controllerKind": "Lease"}
2023-09-26T10:00:26.776Z    INFO    controller.machine.garbagecollection    controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.777Z    INFO    controller.machine.link controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.777Z    INFO    controller.pricing  controller/singleton.go:77  starting controller {"commit": "637a642"}
2023-09-26T10:00:26.845Z    DEBUG   controller  launchtemplate/launchtemplate.go:332    hydrated launch template cache  {"commit": "637a642", "tag-key": "karpenter.k8s.aws/cluster", "tag-value": "eks-cluster", "count": 0}
2023-09-26T10:00:26.869Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "provisioner.hash", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-09-26T10:00:26.870Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-09-26T10:00:26.871Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "worker count": 10}
2023-09-26T10:00:26.872Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-09-26T10:00:26.872Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-09-26T10:00:26.873Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 1}
2023-09-26T10:00:26.873Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "worker count": 100}
2023-09-26T10:00:26.873Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-09-26T10:00:26.879Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "lease.garbagecollection", "controllerGroup": "coordination.k8s.io", "controllerKind": "Lease", "worker count": 10}
2023-09-26T10:00:26.884Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 10}
2023-09-26T10:00:26.884Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 10}
2023-09-26T10:00:26.884Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-09-26T10:00:26.884Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "machine.lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 1000}
2023-09-26T10:00:26.886Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate", "worker count": 10}
2023-09-26T10:00:26.886Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "machine.disruption", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 10}
2023-09-26T10:00:26.886Z    INFO    controller  controller/controller.go:242    Starting workers    {"commit": "637a642", "controller": "machine.termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 100}
2023-09-26T10:00:27.055Z    DEBUG   controller.awsnodetemplate  subnet/subnet.go:87 discovered subnets  {"commit": "637a642", "awsnodetemplate": "default", "subnets": ["subnet-0963c4a7e7c999302 (us-east-1b)", "subnet-0845be3ffe70af48e (us-east-1a)", "subnet-048450cca7a240236 (us-east-1c)"]}
2023-09-26T10:00:27.192Z    DEBUG   controller.awsnodetemplate  securitygroup/securitygroup.go:73   discovered security groups  {"commit": "637a642", "awsnodetemplate": "default", "security-groups": ["sg-026ec32d846919bfa", "sg-0e7f928eb4cc1c1f8"]}
2023-09-26T10:00:27.196Z    DEBUG   controller.awsnodetemplate  amifamily/ami.go:95 discovered kubernetes version   {"commit": "637a642", "awsnodetemplate": "default", "version": "1.27"}
2023-09-26T10:00:27.381Z    DEBUG   controller.pricing  pricing/pricing.go:382  updated spot pricing with instance types and offerings  {"commit": "637a642", "instance-type-count": 702, "offering-count": 3315}
2023-09-26T10:00:27.779Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:00:27.781Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:27.781Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:27.848Z    DEBUG   controller.awsnodetemplate  amifamily/ami.go:132    discovered amis {"commit": "637a642", "awsnodetemplate": "default", "ids": "ami-0c92ea9c7c0380b66, ami-001b78ba700c199b7, ami-001b78ba700c199b7, ami-059bdcbe5667e5170", "count": 4}
2023-09-26T10:00:28.555Z    DEBUG   controller.pricing  pricing/pricing.go:222  updated on-demand pricing   {"commit": "637a642", "instance-type-count": 730}
2023-09-26T10:00:37.775Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:00:37.775Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:37.775Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:47.776Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:00:47.777Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:47.777Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:57.776Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:00:57.777Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:00:57.777Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:07.777Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:01:07.778Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:07.778Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:17.777Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:01:17.778Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:17.778Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:27.778Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:01:27.779Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:27.779Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:37.779Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:01:37.780Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:37.780Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:47.780Z    INFO    controller.provisioner  provisioning/provisioner.go:238 skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default"}
2023-09-26T10:01:47.781Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-54d45649c-9tkxt"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219
2023-09-26T10:01:47.781Z    ERROR   controller.provisioner  scheduling/scheduler.go:190 Could not schedule pod, all available instance types exceed limits for provisioner: "default"   {"commit": "637a642", "pod": "default/resource-hungry-deployment-7885789675-9gw6c"}
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).recordSchedulingResults
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:190
github.com/aws/karpenter-core/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/scheduling/scheduler.go:172
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Schedule
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:330
github.com/aws/karpenter-core/pkg/controllers/provisioning.(*Provisioner).Reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/controllers/provisioning/provisioner.go:130
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:94
github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start
    github.com/aws/karpenter-core@v0.30.0/pkg/operator/controller/singleton.go:82
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1
    sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager/runnable_group.go:219

tzneal commented 12 months ago

What does the node template look like?

rknightion commented 12 months ago

Seeing the same thing in one of our clusters. All 20 clusters have a pretty much identical configuration yet this one refuses to launch any nodes. Karpenter ver 0.30.0

Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: id-dev-intel
spec:
  labels: 
    nodeintent: karpenter-worker-node
  weight: 50
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["r"]
    - key: karpenter.k8s.aws/instance-family
      operator: In
      values: ["r6a"]
    - key: "karpenter.k8s.aws/instance-cpu"
      operator: In
      values: ["1", "2", "4", "6", "8"]
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64"]
    - key: "karpenter.sh/capacity-type" # If not included, the webhook for the AWS cloud provider will default to on-demand
      operator: In
      values: ["on-demand"]
#   - key: karpenter.k8s.aws/instance-size
#     operator: NotIn
#      values: [nano, micro, small, large]
  limits:
    resources:
      memory: 5Ti
  consolidation:
    enabled: true
  ttlSecondsUntilExpired: 1814400 # 7 Days
 # ttlSecondsAfterEmpty: 60
  providerRef:
    name: default

2023-09-28T12:17:16.252Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "pulselive-id-dev-intel"    {"commit": "637a642", "pod": "kube-system/ebs-csi-controller-696f98bdbf-d9tt4"}
2023-09-28T12:17:26.251Z    INFO    controller.provisioner  skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "id-dev-intel"}

tzneal commented 12 months ago

What does the node template look like and can you show the provisioner's status (status section from kubectl get provisioner the-provisioner-name -o yaml)?

rknightion commented 12 months ago

@tzneal please accept my apologies! - It would appear the helm chart we use to install our custom nodetemplate had been failing silently in CI and there was no node template present! As soon as that was added everything worked as expected.

tzneal commented 12 months ago

No problem, we should have a much better indicator of the problem in this case. I'll fix that.

bhavitsharma commented 12 months ago

Hey thanks, the problem was with my providerRef. I wasn't specifying the correct one. Would be great to have a better error message here

hassammustafaa commented 11 months ago

@tzneal I do have the same error, this is odd

Here is the output of `kubectl get provisioner the-provisioner-name -o yaml)?`

apiVersion: karpenter.sh/v1alpha5 kind: Provisioner metadata: annotations: karpenter.sh/provisioner-hash: "15591017375434095235" kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"karpenter.sh/v1alpha5","kind":"Provisioner","metadata":{"annotations":{},"name":"default-lt"},"spec":{"consolidation":{"enabled":true},"kubeletConfiguration":{"containerRuntime":"containerd","maxPods":110},"labels":null,"limits":{"cpu":100,"resources":null},"providerRef":{"name":"default"},"requirements":[{"key":"topology.kubernetes.io/zone","operator":"In","values":["ap-south-1a","ap-south-1b","ap-south-1c"]},{"key":"karpenter.sh/capacity-type","operator":"In","values":["spot","on-demand"]},{"key":"karpenter.k8s.aws/instance-category","operator":"In","values":["t","m","c"]},{"key":"karpenter.k8s.aws/instance-cpu","operator":"Lt","values":["17"]},{"key":"karpenter.k8s.aws/instance-cpu","operator":"Gt","values":["7"]},{"key":"karpenter.k8s.aws/instance-memory","operator":"Gt","values":["8192"]},{"key":"kubernetes.io/arch","operator":"In","values":["amd64"]}],"ttlSecondsUntilExpired":864000,"version":"2023"}} creationTimestamp: "2023-10-20T06:33:10Z" generation: 1 name: default-lt resourceVersion: "95587" uid: c1708cb7-400d-49ed-b522-7f83fb4d9e73 spec: consolidation: enabled: true kubeletConfiguration: containerRuntime: containerd maxPods: 110 limits: {} providerRef: name: default requirements:

key: topology.kubernetes.io/zone operator: In values:
- ap-south-1a
- ap-south-1b
- ap-south-1c
key: karpenter.sh/capacity-type operator: In values:
- spot
- on-demand
key: karpenter.k8s.aws/instance-category operator: In values:
- t
- m
- c
key: karpenter.k8s.aws/instance-cpu operator: Lt values:
- "17"
key: karpenter.k8s.aws/instance-cpu operator: Gt values:
- "7"
key: karpenter.k8s.aws/instance-memory operator: Gt values:
- "8192"
key: kubernetes.io/arch operator: In values:
- amd64
key: kubernetes.io/os operator: In values:
- linux ttlSecondsUntilExpired: 864000
```
Here goes the karpenter logs
```

ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-vd6hp"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-vdp5v"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-qlsnw"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-ptfrn"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-rb7t4"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-777zg"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-2vc5r"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-827mk"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-9dtw8"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-g52ln"}
2023-10-20T12:15:23.216Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-pdvrl"}
2023-10-20T12:15:33.211Z    INFO    controller.provisioner  skipping, no resolved instance types found  {"commit": "637a642", "provisioner": "default-lt"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-mm6bn"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-nrcdc"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-tc7w5"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-qtrv7"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-6jmh9"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-h9jlh"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod": "default/inflate-7849c696cd-f4zmq"}
2023-10-20T12:15:33.215Z    ERROR   controller.provisioner  Could not schedule pod, all available instance types exceed limits for provisioner: "default-lt"    {"commit": "637a642", "pod":

dgrozaMstar commented 11 months ago

@hassammustafaa did you managed to solve the error?I have the same problem and it's driving me crazy...

hassammustafaa commented 11 months ago

Yes, I did, can you share your AWSNodeTemplate and output of kubectl get provisioner the-provisioner-name -o yaml)?

dgrozaMstar commented 11 months ago

I've just realize that i had a typo...:D I was using "node.kubernetes.io/instance-category" instead of "karpenter.k8s.aws/instance-category" Sorry to bother you.

advishnuprasad commented 10 months ago

I had the same issue. I resolved it by adding the right tag to the subnet and security group.

MateusCastello commented 9 months ago

I had the same issue. I resolved it by adding the right tag to the subnet and security group.

My SG and Subnets tag whas missing the last character in "karpenter.sh/discovery"

aws / karpenter-provider-aws

Karpenter controller can't schedule/add any nodes #4698

Description