0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient neonvm/kvm.

[william-lbn]

I followed https://github.com/neondatabase/autoscaling?tab=readme-ov-file#building-and-running

postgres16-disk-test always pending

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl get pod 
NAME                         READY   STATUS    RESTARTS   AGE
postgres16-disk-test-b88qv   0/2     Pending   0          6m1s

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl describe pod postgres16-disk-test-b88qv
Name:             postgres16-disk-test-b88qv
...
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From                 Message
  ----     ------            ----   ----                 -------
  Warning  FailedScheduling  6m10s  autoscale-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient neonvm/kvm.
  Warning  FailedScheduling  45s    autoscale-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node

autoscale-scheduler logs as follows

{"level":"error","ts":1713319685.30246,"logger":"autoscale-scheduler.plugin","caller":"plugin/plugin.go:343","msg":"Pod rejected by all Filter method calls","method":"Filter","virtualmachine":{"namespace":"default","name":"postgres16-disk-test"},"pod":{"namespace":"default","name":"postgres16-disk-test-b88qv"},"stacktrace":"github.com/neondatabase/autoscaling/pkg/plugin.(*AutoscaleEnforcer).
PostFilter\n\t/workspace/pkg/plugin/plugin.go:343\nk8s.io/kubernetes/pkg/scheduler/framework/runtime.(*frameworkImpl).runPostFilterPlugin\n\t/go/pkg/mod/k8s.io/kubernetes@v1.25.16/pkg/scheduler/framework/runtime/framework.go:776\nk8s.io/kubernetes/pkg/scheduler/framework/runtime.(*frameworkImpl).RunPostFilterPlugins\n\t/go/pkg/mod/k8s.io/kubernetes@v1.25.16/pkg/scheduler/framework/runtime/framework.go:759\nk8s.io/kubernetes/pkg/scheduler.
(*Scheduler).scheduleOne\n\t/go/pkg/mod/k8s.io/kubernetes@v1.25.16/pkg/scheduler/schedule_one.go:110\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:190\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:157\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:135\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:190\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.25.16/pkg/util/wait/wait.go:101"}

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# more vm-deploy.yaml 
---
apiVersion: vm.neon.tech/v1
kind: VirtualMachine
metadata:
  name: postgres16-disk-test
  annotations:
    # In this example, these bounds aren't necessary. So... here's what they look like :)
    autoscaling.neon.tech/bounds: '{ "min": { "cpu": 0.25, "mem": "1Gi" }, "max": { "cpu": 1.25, "mem": "1Gi" } }'
  labels:
    autoscaling.neon.tech/enabled: "true"
    # Set to "true" to continuously migrate the VM (TESTING ONLY)
    autoscaling.neon.tech/testing-only-always-migrate: "false"
spec:
  schedulerName: autoscale-scheduler
  enableSSH: true
  guest:
    cpus: { min: 0.25, use: 0.25, max: 0.25 }
    memorySlotSize: 1Gi
    memorySlots: { min: 1, use: 1, max: 1 }
    rootDisk:
      image: pg16-disk-test:dev
      size: 1Gi
    ports:
      - port: 5432 # postgres
      - port: 9100 # metrics
      - port: 10301 # monitor

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl get neonvm
NAME                   CPUS   MEMORY   POD                          EXTRAIP   STATUS    RESTARTS   AGE
postgres16-disk-test                   postgres16-disk-test-b88qv             Pending              5h10m

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl get pod postgres16-disk-test-b88qv -ojson -ojson
...
    "status": {
        "conditions": [
            {
                "lastProbeTime": null,
                "lastTransitionTime": "2024-04-17T02:08:05Z",
                "message": "0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient neonvm/kvm.",
                "reason": "Unschedulable",
                "status": "False",
                "type": "PodScheduled"
            }
        ],
        "phase": "Pending",
        "qosClass": "Burstable"
    }
}

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl get node
NAME                        STATUS   ROLES           AGE   VERSION
neonvm-root-control-plane   Ready    control-plane   23h   v1.25.11
neonvm-root-worker          Ready    <none>          23h   v1.25.11
neonvm-root-worker2         Ready    <none>          23h   v1.25.11

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# kubectl describe node neonvm-root-worker
Name:               neonvm-root-worker
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                5560m (34%)   18360m (114%)
  memory             3118Mi (10%)  6030Mi (19%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
  neonvm/kvm         0             0
  neonvm/vhost-net   0             0
Events:              <none>

How can I make postgres16-disk-test scheduling successful?
I'm a beginner, thank you very much for your help.

In addition, which service of neon can postgres16-disk-test be compared to? Is there a way to simulate scaling testing?

---

Running pgbench

root@iZbp19lce9chqq1glegm26Z:~/serverless/neon/autoscaling# scripts/run-bench.sh
If you don't see a command prompt, try pressing enter.
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/8) Installing postgresql-common (1.2-r1)
Executing postgresql-common-1.2-r1.pre-install
(2/8) Installing lz4-libs (1.9.4-r5)
(3/8) Installing libpq (16.2-r0)
(4/8) Installing ncurses-terminfo-base (6.4_p20231125-r0)
(5/8) Installing libncursesw (6.4_p20231125-r0)
(6/8) Installing readline (8.2.1-r2)
(7/8) Installing zstd-libs (1.5.5-r8)
(8/8) Installing postgresql16-client (16.2-r0)
Executing busybox-1.36.1-r15.trigger
Executing postgresql-common-1.2-r1.trigger
* Setting postgresql16 as the default version
OK: 12 MiB in 23 packages
Running pgbench. Query:
   select length(factorial(length(factorial(1223)::text)/2)::text);
pgbench: error: too many command-line arguments (first is "postgres")
pgbench: hint: Try "pgbench --help" for more information.
pod "pgbench-postgres16-disk-test" deleted
pod default/pgbench-postgres16-disk-test terminated (Error)

root@iZbp19lce9chqq1glegm26Z:~# kubectl get pod -w
NAME                         READY   STATUS    RESTARTS   AGE
postgres16-disk-test-b88qv   0/2     Pending   0          14m

pgbench-postgres16-disk-test   0/1     Pending   0          0s
pgbench-postgres16-disk-test   0/1     Pending   0          0s
pgbench-postgres16-disk-test   0/1     ContainerCreating   0          0s
pgbench-postgres16-disk-test   0/1     ContainerCreating   0          1s
pgbench-postgres16-disk-test   1/1     Running             0          16s
pgbench-postgres16-disk-test   0/1     Error               0          18s
pgbench-postgres16-disk-test   0/1     Error               0          20s
pgbench-postgres16-disk-test   0/1     Terminating         0          20s
pgbench-postgres16-disk-test   0/1     Terminating         0          20s

During Running pgbench, there were no neonvm resources, and an error occurred and exited.

root@iZbp19lce9chqq1glegm26Z:~# kubectl get neonvm
NAME                   CPUS   MEMORY   POD                          EXTRAIP   STATUS    RESTARTS   AGE
postgres16-disk-test                   postgres16-disk-test-b88qv             Pending              17m

[william-lbn]

neonvm-controller error :

{"level":"info","ts":"2024-04-17T07:57:34.045Z","msg":"Adding Finalizer for VirtualMachine","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"5bccee1b-3cf1-431e-96f7-10782877d6aa"}
{"level":"error","ts":"2024-04-17T07:57:34.059Z","msg":"Failed to update VirtualMachine status after reconcile loop","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"9bb70b2e-30b8-4e22-8b79-a9fea7f8e50c","virtualmachine":"postgres16-disk-test","error":"Operation cannot be fulfilled on virtualmachines.vm.neon.tech \"postgres16-disk-test\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2024-04-17T07:57:34.059Z","msg":"Reconciler error","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"9bb70b2e-30b8-4e22-8b79-a9fea7f8e50c","error":"Operation cannot be fulfilled on virtualmachines.vm.neon.tech \"postgres16-disk-test\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2024-04-17T07:57:34.059Z","msg":"Creating a new SSH Secret","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"ff6962c5-068c-48ef-b080-ae5c9b3ec155","Secret.Namespace":"default","Secret.Name":"ssh-neonvm-postgres16-disk-test"}
{"level":"info","ts":"2024-04-17T07:57:34.061Z","msg":"SSH Secret was created","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"ff6962c5-068c-48ef-b080-ae5c9b3ec155","Secret.Namespace":"default","Secret.Name":"ssh-neonvm-postgres16-disk-test"}
{"level":"info","ts":"2024-04-17T07:57:34.061Z","msg":"Creating a new Pod","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"ff6962c5-068c-48ef-b080-ae5c9b3ec155","Pod.Namespace":"default","Pod.Name":"postgres16-disk-test-cgf8w"}
{"level":"info","ts":"2024-04-17T07:57:34.066Z","msg":"Runner Pod was created","controller":"virtualmachine","controllerGroup":"vm.neon.tech","controllerKind":"VirtualMachine","VirtualMachine":{"name":"postgres16-disk-test","namespace":"default"},"namespace":"default","name":"postgres16-disk-test","reconcileID":"ff6962c5-068c-48ef-b080-ae5c9b3ec155","Pod.Namespace":"default","Pod.Name":"postgres16-disk-test-cgf8w"}

Operation cannot be fulfilled on virtualmachines.vm.neon.tech \"postgres16-disk-test\": the object has been modified; please apply your changes to the latest version and try again

What is the latest version？

[Omrigan]

Hi @william-lbn!

The important part is 2 Insufficient neonvm/kvm

I think you probably don't have /dev/kvm on the node. Try enabling kvm on the node where you are running this or setting enableAcceleration: false in the vm-deploy.yaml if you are fine running without hardware virtualization.

In addition, which service of neon can postgres16-disk-test be compared to?

Our autoscaling runs computes, which are modified postgreses able to operate on top of the neon storage.

postgres16-disk-test is a vanilla postgres though, so that we can test it without setting up neon storage. You can check how it is constructed in the vm-builder spec

Is there a way to simulate scaling testing?

You can look at our autoscaling test and do the same steps by hand outside of kuttl.

What is the latest version？

neonvm-controller operates on VirtualMachine CRD object. So, in this context, the version is the CRD object version. Small rate of transient reconciling failures is normal, neonvm-controller should auto-retry the reconciliation.

Hope this helps! Can you share the usecase you have in mind for neon autoscaling?

P. S. feel free to join our discord!

[william-lbn]

Hi @Omrigan

Thank you very much for your reply.

1. Is the construction method of postgres image the same as postgres16-disk-test? Can the image built by compute in neon be used directly? 
2. There are relatively few neon documents, especially for building a complete set of documents on k8s (neon+autoscaling). I haven't deployed a complete environment yet, and without a deep understanding, I'm afraid it will be difficult to make very valuable suggestions. But I'm familiar with serverless faas. Neon has announced that it can scale to zero, but the goal of autoscaling is to dynamically change the amount of CPU and memory of a running postgres instance without interrupting the TCP connection with postgres. Is this contradictory to scaling to zero? compute (postgres) is a stateless service, so it scales to zero and only connects and disconnects to the database at startup and end (first access requested). Did neon take special circumstances into consideration here? 
3. The autoscaling environment has been deployed, and neon can be quickly deployed through docker compose. I would like to ask, can neon and autoscaling be combined to see the entire scaling process? I am very interested in the field of database scaling, and think it is a difficult part of the resource optimization and scaling process, and I very much hope to participate in this project. Finally thank you again for your help.

[Omrigan]

Hi @Omrigan

Thank you very much for your reply.

1. Is the construction method of postgres image the same as postgres16-disk-test? Can the image built by compute in neon be used directly?

The main compute image is constructed in the same way as postgres16-disk-test. Here is the vm-builder spec: https://github.com/neondatabase/neon/blob/0353a72a00887173f802ba044d169a4c278ea8f8/vm-image-spec.yaml#L4

2. There are relatively few neon documents, especially for building a complete set of documents on k8s (neon+autoscaling). I haven't deployed a complete environment yet, and without a deep understanding, I'm afraid it will be difficult to make very valuable suggestions. But I'm familiar with serverless faas. Neon has announced that it can scale to zero, but the goal of autoscaling is to dynamically change the amount of CPU and memory of a running postgres instance without interrupting the TCP connection with postgres. Is this contradictory to scaling to zero?

Yes, scaling while the VM is running and scaling-to-zero are different things. In the current "autoscaling" repo we don't handle scale-to-zero, it is handled external by control plane.

compute (postgres) is a stateless service, so it scales to zero and only connects and disconnects to the database at startup and end (first access requested). Did neon take special circumstances into consideration here?

What kind of special circumstances are you asking about?

3. The autoscaling environment has been deployed, and neon can be quickly deployed through docker compose. I would like to ask, can neon and autoscaling be combined to see the entire scaling process? I am very interested in the field of database scaling, and think it is a difficult part of the resource optimization and scaling process, and I very much hope to participate in this project.

They sure can be combined, that's how we run it in production. Unfortunately, the full scale integration depends on control plane presence, which is not open source at the moment. So you'd have to write your own control plane to integrate everything.

Finally thank you again for your help.

You're welcome!

[Omrigan]

@william-lbn Will close this, feel free to reopen if more questions come up.