Implement new GitLab Runner Autoscaling architecture

[kayman-mk]

Docker+machine is outdated and the Gitlab group is developing on a new infrastructure (taskscaler + AWS fleeting).

This module should move away from the old Docker+machine and support the new Gitlab plugin technology.

Refs: https://docs.gitlab.com/ee/architecture/blueprints/runner_scaling/index.html https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29312 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29219#note_1169146211

[npalm]

Fully agree docker machine is outdated. And as soon GitLab has something ready we should ty to adopt the new direction.

[vparmeland]

Any news about this ? The next release 16.0 is moving forward, we'll be impacted soon 😞

[kayman-mk]

Do we have a link at hand on how to use this new feature, which GitLab version I need, ....?

See

• https://gitlab.com/gitlab-org/fleeting/fleeting-plugin-aws
• https://docs.gitlab.com/runner/development/
• https://gitlab.com/gitlab-org/fleeting/taskscaler/

[Tiduster]

Indeed @kayman-mk @NK74 , this is far from ready: https://gitlab.com/groups/gitlab-org/-/epics/8856

Current planned "GitLab Runner Fleeting plugin for AWS EC2 instances (GA)": 16.3 (2023-08-22)

[Kadeux]

I'm currently working on this issue but the lack of documentation (source code mostly) makes it really difficult. I'm planning to continue my work when the beta will be released (16.2).

For those who might be interested, my error is ERROR: Failed to remove network for build when the runner is trying to dial the newly created instance.

[kayman-mk]

Did a quick analysis. Some people suggest that a) Docker is missing on the machine and b) the Runner is not running in privileged mode. But I guess if you take your standard configuration it should work out of the box.

[veshant]

They've added it to the official docs.

@Kadeux are you still working on this?

https://docs.gitlab.com/runner/executors/docker_autoscaler.html

[lenasappvia]

This will be great to add!

[dsalaza4]

Would love to see this!

Tired of Docker+machine errors, they're so hard to debug 😿

[nesty92]

Hey everyone,

GitLab Runner Autoscaler for AWS

Since I didn't have enough time to integrate it here, I decided to start from scratch. I've also included the Packer file for building the AMI.

Give it a try, share your feedback, and even contribute to make it even better!

Happy scaling! 🚀🔧