spidernet-io / spiderpool

Underlay and RDMA network solution of the Kubernetes, for bare metal, VM and any public cloud
https://spidernet-io.github.io/spiderpool/
Apache License 2.0
507 stars 73 forks source link

the Spiderpool IPAM allocates IPs slowly with large subnet #3541

Closed iziang closed 3 weeks ago

iziang commented 1 month ago

What help do you need?

I'm using Calico with Spiderpool to provide fixed IP capabilities for StatefulSet applications. Since there's no need for underlay network connectivity, I've configured the pod CIDR as an overlay network segment, 100.64.0.0/10. The SpiderIPPool subnet is also set to 100.64.0.0/10, with the ips spanning the entire range from 100.64.0.1 to 100.127.255.254. The issue I'm encountering is that after the Pods are created, they remain in the ContainerCreating state for an extended period, waiting for the CNI to allocate IPs-sometimes over a minute. Upon checking the Spiderpool agent logs, it appears that the requests are queuing up. I captured some CPU profiles using pprof and noticed that most of the time is spent on ipdiffset. Suspecting that the large network segment might be the issue, I changed the IPs from a /10 to a /20 subnet, which significantly sped up the allocation process.

Additionally, only about 10 Pods were created, so the concurrency was quite low. Please help identify the reasons for the slow allocation. Thank you.

Here is the IPAM duration with the /10 subnet: image

Here is the IPAM duration with the /20 subnet: image

Here are the CPU performance metrics: image

Here are the memory performance metrics: image

ty-dc commented 1 month ago

Thank you, this question is very meaningful and we have paid attention to it.

weizhoublue commented 4 weeks ago

@iziang the fix is on the way https://github.com/spidernet-io/spiderpool/pull/3583

weizhoublue commented 3 weeks ago

done