[Bug]: Dashboard cannot launch 300 user notebooks with 1s or 2s delay

[lucferbux]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Follow up of #627

Details

Perform a scale test of notebook creation at user-level

• 300 users
• 1s or 2s delay between each user
• 2 consecutive runs

300 users, 1s delay 0/300 successes 300 users, 2s delay 31/300 successes And similar results with m5a.2xlarge masters instead of m6a.xlarge masters:

300 users, 1s delay, m5a.2xlarge masters 3/300 successes (first run) second run 215/300 successes

We got that some of these performance issues are related to LIST k8s calls. We have already addressed the nb-events endpoint in #627 and now we need to find all the bottlenecks left and fix them.

Expected Behavior

No issues with multiple users spawning notebooks at the same time.

Steps To Reproduce

Perform a scale test of notebook creation at user-level

• 300 users
• 1s or 2s delay between each user
• 2 consecutive runs

Workaround (if any)

No response

OpenShift Infrastructure Version

No response

Openshift Version

No response

What browsers are you seeing the problem on?

No response

Open Data Hub Version

No response

Relevant log output

No response

[andrewballantyne]

This was marked closed downstream and thus we should not spend anymore time on it until we get another report of the perf issue. See RHODS-5411