Spike: Investigate viability of ephemeral autoscaling shell and/or docker autoscaling runners using the fleet plugin

Is your feature request related to a problem? Please describe.

We need a way to have secure gitlab runners in a cloud environment where there is no trust of the individual executing a job. Why ephemeral jobs can be implemented using the k8s executor there are numerous security and usability issues in this approach. Github public runners are ephemeral vms and this is a fairly reasonable security model that limits risk.

Describe the solution you'd like

Given a job is kicked off as part of a pipeline by an untrusted individual, a job will be scheduled on an ephemeral locked down vm that can only process a single job. As a result the job is able to run with the full functionality provides from a shell or docker based runner in a secure fashion where the build system clean has limited risk.

Describe alternatives you've considered

k8s based extending something like this to only allow one job per instance and the kill the instance https://github.com/cmdlabs/terraform-aws-gitlab-runner-scale

Additional context

Ideally, the runner manager could exist in kubernetes, although a separate, persisent ec2 is also fine for a proof of concept. That manager will have access to manage the autoscaling vms via an iam policy and IRSA and once configured single job will run on ephemeral vms. See https://docs.gitlab.com/runner/executors/instance.html#one-job-per-instance for reference.

This spike is in support of the decision point that needs to be made here

defenseunicorns / uds-package-gitlab-runner