How to use Neon deployed by docker-compose with autoscaling Kubernetes cluster deployed by kind?

[yaoyinnan]

I have now deployed a Neon cluster using docker-compose, and the node information is as follows:

# docker ps
CONTAINER ID   IMAGE                                              COMMAND                  CREATED          STATUS             PORTS                                              NAMES
e6ad7d839d55   docker-compose_compute                             "/shell/compute.sh"      45 minutes ago   Up 45 minutes      0.0.0.0:3080->3080/tcp, 0.0.0.0:55433->55433/tcp   docker-compose_compute_1
ce312f9c47bc   neondatabase/neon:latest                           "/bin/sh -c '/usr/lo…"   46 minutes ago   Up 45 minutes      6400/tcp, 0.0.0.0:9898->9898/tcp                   docker-compose_pageserver_1
d6c3aa8c81bc   neondatabase/neon:latest                           "/bin/sh -c 'safekee…"   46 minutes ago   Up 46 minutes      6400/tcp, 9898/tcp, 0.0.0.0:7677->7676/tcp         docker-compose_safekeeper2_1
5d949edd0cfa   neondatabase/neon:latest                           "/bin/sh -c 'safekee…"   46 minutes ago   Up 45 minutes      6400/tcp, 9898/tcp, 0.0.0.0:7676->7676/tcp         docker-compose_safekeeper1_1
f896b7535c2c   neondatabase/neon:latest                           "/bin/sh -c 'safekee…"   46 minutes ago   Up 45 minutes      6400/tcp, 9898/tcp, 0.0.0.0:7678->7676/tcp         docker-compose_safekeeper3_1
7321a496e8a0   quay.io/minio/minio:RELEASE.2022-10-20T00-55-09Z   "/usr/bin/docker-ent…"   47 minutes ago   Up 46 minutes      0.0.0.0:9000-9001->9000-9001/tcp                   docker-compose_minio_1
4a16732eb4fb   neondatabase/neon:latest                           "storage_broker --li…"   47 minutes ago   Up 46 minutes      6400/tcp, 9898/tcp, 0.0.0.0:50051->50051/tcp       docker-compose_storage_broker_1

It contains the following nodes:

• compute * 1
• pageserver * 1
• safekeeper * 3
• storage_broker * 1
• MinIO * 1 (S3)

I can connect to PostgreSQL running in compute:

psql -h localhost -p 55433 -U cloud_admin postgres

At the same time, I also run an autoscaling Kubernetes cluster started by kind, which contains a series of nodes. The function implemented in it is to run an elastically scalable postgres14-disk-test-cxkmw, which runs in a NeonVM, through the virtual The machine hot-swappable function realizes vertical elastic scaling, and realizes horizontal elastic scaling through virtual machine migration. But only postgresql nodes are running in this cluster, running in NeonVM.

# kubectl get pods --all-namespaces -o wide
NAMESPACE            NAME                                         READY   STATUS    RESTARTS      AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
cert-manager         cert-manager-5b8f9b9d96-5924t                1/1     Running   1 (92m ago)   7h9m    10.244.1.7   kind-worker2         <none>           <none>
cert-manager         cert-manager-cainjector-54f68bfb64-ndg9z     1/1     Running   6 (92m ago)   7h9m    10.244.1.3   kind-worker2         <none>           <none>
cert-manager         cert-manager-webhook-f6c8487d6-c6zvb         1/1     Running   6 (92m ago)   7h9m    10.244.1.6   kind-worker2         <none>           <none>
default              postgres14-disk-test-cxkmw                   1/1     Running   0             79m     10.244.2.6   kind-worker          <none>           <none>
kube-system          autoscale-scheduler-6c7495dcb9-84gcb         1/1     Running   1 (92m ago)   6h12m   10.244.2.4   kind-worker          <none>           <none>
kube-system          autoscaler-agent-5kg2r                       1/1     Running   1 (92m ago)   6h12m   10.244.1.5   kind-worker2         <none>           <none>
kube-system          autoscaler-agent-86w9c                       1/1     Running   1 (92m ago)   6h12m   10.244.2.3   kind-worker          <none>           <none>
kube-system          coredns-565d847f94-rk8j2                     1/1     Running   1 (92m ago)   7h9m    10.244.0.3   kind-control-plane   <none>           <none>
kube-system          coredns-565d847f94-tvpr2                     1/1     Running   1 (92m ago)   7h9m    10.244.0.2   kind-control-plane   <none>           <none>
kube-system          etcd-kind-control-plane                      1/1     Running   0             82m     172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kindnet-48v92                                1/1     Running   1 (92m ago)   7h9m    172.19.0.3   kind-worker2         <none>           <none>
kube-system          kindnet-6m6pr                                1/1     Running   1 (92m ago)   7h9m    172.19.0.2   kind-worker          <none>           <none>
kube-system          kindnet-xxrfw                                1/1     Running   1 (92m ago)   7h9m    172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0             82m     172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   1 (92m ago)   7h9m    172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kube-multus-ds-4k9kh                         1/1     Running   1 (92m ago)   6h13m   172.19.0.3   kind-worker2         <none>           <none>
kube-system          kube-multus-ds-7jbdd                         1/1     Running   1 (92m ago)   6h13m   172.19.0.2   kind-worker          <none>           <none>
kube-system          kube-multus-ds-bfsvv                         1/1     Running   1 (92m ago)   6h13m   172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kube-proxy-9wxnc                             1/1     Running   1 (92m ago)   7h9m    172.19.0.2   kind-worker          <none>           <none>
kube-system          kube-proxy-dmkpk                             1/1     Running   1 (92m ago)   7h9m    172.19.0.3   kind-worker2         <none>           <none>
kube-system          kube-proxy-t7m95                             1/1     Running   1 (92m ago)   7h9m    172.19.0.4   kind-control-plane   <none>           <none>
kube-system          kube-scheduler-kind-control-plane            1/1     Running   1 (92m ago)   7h9m    172.19.0.4   kind-control-plane   <none>           <none>
kube-system          metrics-server-6644dbd5f9-gvsx6              1/1     Running   4 (92m ago)   7h8m    10.244.2.5   kind-worker          <none>           <none>
kube-system          whereabouts-gtwsn                            1/1     Running   1 (92m ago)   6h13m   172.19.0.3   kind-worker2         <none>           <none>
kube-system          whereabouts-psk4z                            1/1     Running   1 (92m ago)   6h13m   172.19.0.2   kind-worker          <none>           <none>
kube-system          whereabouts-qqx8g                            1/1     Running   1 (92m ago)   6h13m   172.19.0.4   kind-control-plane   <none>           <none>
local-path-storage   local-path-provisioner-7cf55d6b5d-wddfl      1/1     Running   4 (92m ago)   7h9m    10.244.0.5   kind-control-plane   <none>           <none>
neonvm-system        neonvm-controller-669c6f4c7-wst6d            2/2     Running   2 (92m ago)   6h12m   10.244.1.2   kind-worker2         <none>           <none>
neonvm-system        neonvm-device-plugin-5swjx                   1/1     Running   1 (92m ago)   6h12m   10.244.2.2   kind-worker          <none>           <none>
neonvm-system        neonvm-device-plugin-nvwhj                   1/1     Running   1 (92m ago)   6h12m   10.244.1.4   kind-worker2         <none>           <none>
neonvm-system        neonvm-device-plugin-vgsgv                   1/1     Running   1 (92m ago)   6h12m   10.244.0.4   kind-control-plane   <none>           <none>
neonvm-system        neonvm-vxlan-controller-cpq7t                1/1     Running   5 (81m ago)   6h12m   172.19.0.4   kind-control-plane   <none>           <none>
neonvm-system        neonvm-vxlan-controller-k9cmx                1/1     Running   5 (80m ago)   6h12m   172.19.0.3   kind-worker2         <none>           <none>
neonvm-system        neonvm-vxlan-controller-zfnbj                1/1     Running   5 (80m ago)   6h12m   172.19.0.2   kind-worker          <none>           <none>

How can I use the PostgreSQL node in Kubernetes as the computing node of the cluster, and use other storage layer nodes deployed by docker-compose as the storage node of the cluster?

Is there any relevant documentation that can help me?

Thanks~!

[skyzh]

You should be able to create tenants using the HTTP control plane APIs and then you may access the database (see control_plane). For local developments, we recommend using the cargo neon utility instead of deploying everything by yourself.

[yaoyinnan]

@skyzh thanks for your help. I tried deploying with cargo neon. But the main problem I want to solve now is: how to configure the network topology so that the PostgreSQL nodes deployed in K8s can interact with the storage layer nodes (safekeeper, pageserver, minio) of cargo neon / docker-compose? Do I have to do service discovery manually? Is there any relevant case for my reference?

[kesavkolla]

Is there any solution for neondata in kubernetes? Any helm chart any operator?

[skyzh]

https://github.com/neondatabase/helm-charts?

[yaoyinnan]

@skyzh Thank you so much. This work is amazing. At present, helm-charts does not seem to support the deployment of Safekeeper and Pageserver? Are there plans to support it in the future?

[skyzh]

I don't think it's on the roadmap because we currently scale them manually.

[kesavkolla]

Even scaling is manual but it will be helpful to install them via charts. Otherwise the instructions of manual + helm is going to be tough to understand.

[rennokki]

Something in my mind is telling me I should start and create a chart myself. :^)