kube-controller-manager/workload-controller-manager on admin cluster will crash when running 5000 hollow-nodes pods

[sonyafenge]

What happened: using kubemark to simulate 5000 nodes, kube-controller-manager/workload-controller-manager on admin cluster will crash when running 5000 hollow-nodes pods

What you expected to happen: no crash on admin cluster How to reproduce it (as minimally and precisely as possible):

1. run the commands below to start admin cluster and kubemark cluster

   export MASTER_DISK_SIZE=750GB KUBE_GCE_ZONE=us-west2-b MASTER_SIZE=n1-highmem-96 KUBE_GCE_NETWORK=default
   ./cluster/kube-up.sh
   export MASTER_DISK_SIZE=750GB KUBE_GCE_ZONE=us-west2-b MASTER_SIZE=n1-highmem-96 KUBE_GCE_NETWORK=kubemark
   ./test/kubemark/start-kubemark.sh

2. start 5000 hollow nodes
3. check admin cluster status and found kube-controller-manager/workload-controller-manager will crash around 500 nodes. Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):

  commit 126c90ad2230452054a1a2b8e5f3c4f2c5cf4540 (HEAD)
  Merge: b474e793 f0b3df9d
  Author: Yin Ding <yin.ding@futurewei.com>
  Date:   Fri Feb 7 14:55:51 2020 -0800
  
  Merge pull request #9 from Sindica/controller-s3-merge
  
  Support multiple api server configurations

• Cloud provider or hardware configuration: Google Cloud
• OS (e.g: cat /etc/os-release):

  NAME="Ubuntu"
  VERSION="18.04.3 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.3 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):

  Linux alkaiddev 5.0.0-1029-gcp #30~18.04.1-Ubuntu SMP Mon Jan 13 05:40:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[sonyafenge]