[Scale out POC] Tenant zeta request was not redirected to TP2 master correctly [current proxy limitation]

[Sindica]

What happened:

• Using kubeup/kubemark to start a 2TP/2RP cluster, create tenant arktos and zeta via proxy

  • Able to get both tenant from proxy kubeconfig
  • Can also get both tenant from tp1 kubeconfig
  • Cannot get tenant from tp2 kubeconfig
  • Delete tenant zeta via proxy: reported as tenant not found.
  • Can delete tenant zeta via tp1 kubeconfig

• Grepping log in tp 1 master.

  • Nothing in api server
  • zeta found in KCM log

• Grepping log in tp 2 master.

  • One 404 delete in api server audit log.
  • Nothing else

• Grepping proxy log, can only find DELETE redirect $ grep zeta haproxy.log Mar 27 02:07:28 ying-poc-merge-0326-2x500-kubemark-proxy haproxy[25876]: 34.83.188.222:40648 [27/Mar/2021:02:07:28.616] scale-out-proxy~ tenant_api_2/tp_2 0/0/4/3/+7 404 +350 - - ---- 1/1/1/1/0 0/0 "DELETE /api/v1/tenants/zeta HTTP/1.1"

• Analysis

  • Create tenant zeta request via proxy was not redirected to tp2, instead, it went to tp1
  • Delete tenant zeta request via proxy was redirected to tp2

What you expected to happen:

• arktos tenant create in tp1, zeta tenant create in tp2
• Query via tp1 kubeconfig find tenant arktos only, query via tp2 kubeconfig find tenant tenant only

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Arktos version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: IP: . Perf test host: 34.83.188.222, 10.138.15.193 . Proxy: 35.239.75.110, 10.40.0.15 . TP1 master: 34.121.148.105, 10.40.0.16 . TP2 master: 35.188.201.244, 10.40.0.17 . RP1 master: 146.148.47.50, 10.40.0.18 . RP2 master: 104.197.175.35, 10.40.0.19

• Log saved at ying-dev1:~/logs/perf-test/gce-500/arktos/merge-poc-0326-etcd343-2x500

[Sindica]

One second thought, this may not be an bug for now since proxy probably only redirects based on url, not by post data. Create tenant is a post request that tenant name is contained in body but not url.

[XiaoningDing]

@Sindica Yes, currently our proxy cannot redirect the tenant creation requests since the tenant name is not in the POST url. In previous tests we also create tenants against different TPs directly.

In the complete design there will be a module to decide which TP this new tenant should be placed on. This module should support various plugin-style placement algorithms, so the placement policy can be very flexible (like random, round-robin, load-balanced, name-based, whitelist-based, etc.)