Productization of GH runners for census

[beroy]

Description

While currently GH runners are functional, they need to be more stable and product level.

Context

We still have issues here and there with stability of the runners either due to runner or the scheduler.

Ideal behavior

Runners must execute with not much delay or long queuing or authentication issues. Also the resources get freed right after the runs. Also we must manage migrating the runner from playground EKS cluster to a more stable one (current census EKS cluster vs a new one). In general the next steps can be broken down as following

A few steps toward productization (all will move to a new repo):

• Authentication: User token vs system user token vs Okta based (beroy2000@gmail.com Emanuele Bezzi)
• Automation: Adding new node types or modifying existing ones (beroy2000@gmail.com)
• Stabilization: Queuing delay time and freeing resources in a timely manner mostly affected by the scheduler (Jake Heath)
• Migrating from playground EKS cluster to a more stable one (beroy2000@gmail.com Jake Heath Emanuele Bezzi)