This PR introduces an option to set MaxConcurrency for the Scanner, allowing users to control the level of concurrent downloads in Scanner. This configuration can be beneficial for workloads running multiple simultaneous queries.
Additionally, this PR changes the default value of max concurrency from runtime.NumCPU() to runtime.GOMAXPROCS. While both are initially set to the number of available CPUs, GOMAXPROCS is adjustable by users, providing greater flexibility. This change addresses a limitation of using runtime.NumCPU(), which does not account for cgroup limits. For instance, in Kubernetes environments, using runtime.NumCPU() may incorrectly assume access to all CPU cores on a node, rather than only the cores allocated to the specific pod. This mismatch can lead to performance degradation in Kubernetes-deployed applications. See uber-go/automaxprocs.
This PR introduces an option to set
MaxConcurrencyfor theScanner, allowing users to control the level of concurrent downloads in Scanner. This configuration can be beneficial for workloads running multiple simultaneous queries.Additionally, this PR changes the default value of max concurrency from
runtime.NumCPU()toruntime.GOMAXPROCS. While both are initially set to the number of available CPUs,GOMAXPROCSis adjustable by users, providing greater flexibility. This change addresses a limitation of usingruntime.NumCPU(), which does not account for cgroup limits. For instance, in Kubernetes environments, usingruntime.NumCPU()may incorrectly assume access to all CPU cores on a node, rather than only the cores allocated to the specific pod. This mismatch can lead to performance degradation in Kubernetes-deployed applications. See uber-go/automaxprocs.