Open gbonnefille opened 1 year ago
Is it even supported by the official kubernetes executor? On top of my head (I am on the phone ATM) its limited to one reg,thus requiring multiple runners if the split in reg/tags/config is needed.
On Tue, Oct 4, 2022, 17:21 Guilhem Bonnefille @.***> wrote:
I'm actively looking for a way to have a single gilab-runner instance running multiple «runners» in order to create a single shared runner for dedicated groups/projects (cf. https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/230) while controlling the number of total jobs via a single concurrent property.
Do you think it make sense to improve your operator with a new CRD (for example SharedRunner) allowing to define multiple registrations while keeping a single gitlab-runner?
— Reply to this email directly, view it on GitHub https://github.com/alekc/gitlab-runner-operator/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACJ5V4EZTBL7IGFBQOFPWTWBRKQPANCNFSM6AAAAAAQ4XSQP4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
@gbonnefille I've checked the docs and it looks like you are after this scenario: https://docs.gitlab.com/runner/fleet_scaling/#intermediate-configuration-one-runner-multiple-workers
concurrent = 3
[[runners]]
name = "instance_level_shell_001"
url = ""
token = ""
executor = "shell"
[[runners]]
name = "instance_level_shell_002"
url = ""
token = ""
executor = "shell"
[[runners]]
name = "instance_level_shell_003"
url = ""
token = ""
executor = "shell"
It might be achievable with something like
apiVersion: gitlab.k8s.alekc.dev/v1beta1
kind: MultiRunner
metadata:
name: multi-runner
spec:
runners:
- registration_config:
token: "xxx"
tag_list:
- group1
- log_level: debug
executor_config:
image: "debian:slim"
memory_limit: "150Mi"
memory_request: "150Mi"
volumes:
config_map:
- mount_path: /cm/
name: test-config
secret:
- mount_path: /secrets/1/
name: test-secret
- mount_path: /secrets/2/
name: test-secret-2
registration_config:
token: "xxx"
tag_list:
- advanced-runner
I can definitely see advantages to this, however I need to think about it (due to a lack of personal free time atm sadly). Biggest catches to look for would be:
I can definitely see advantages to this, however I need to think about it (due to a lack of personal free time atm sadly).
No problem, the most important is that the idea is acceptable, so I can start looking how I can contribute to that.
Here are some though after reading code.
The most challenging part is, of course, to take into account any changes on the config.
The gitlab-runner is able to reload automatically its config if a change occurs. So, no need to look for change in the executor_config
, simply rewrite again and again the configuration file.
The gitlab-runner-controler has to detect changes on registration_config
. As a registration token is certainly uniq, it could serves as a key for an internal map. Base on it, it would be possible to detect addition, change or deletion of runner. Such events trigger new registration/unregistration on the gitlab.
Note: if the user simply change the registration token, keeping everything else, this would be processed as a deletion+addition.
@alekc I also started playing with the code (nothing functional yet).
Finally, I started looking at a different solution: extending the current CRD with a runners:
array.
https://github.com/alekc/gitlab-runner-operator/pull/19
One idea is to stay as close as the Gitlab Runner configuration.
Another idea is to reuse the « root » configuration as a default configuration. When I set multiple configurations in runners:
tree, I can share some details by declaring a global registration_config
.
What do you think about such implementation direction?
I am doing some work on it as well (you can see it in PR). I had to split Runner and Multirunner in 2 different CRD/controllers.
Main reason is: as right now, Runner saves the information about latest registration tag, registration and auth tokens as a single field in the status (they are needed because otherwise we do not know if the reg spec has been changed or not). That obviously won't work with the multiple runners approach because we need to keep track of this data for every runner specified in the multirunner section.
WHile I am at it, I am doing some minor/major refactoring around components and making them as generic as possible. You can expect a working testing demo in couple of days top.
I'd rather not to touch too much existing runner because its simple and already used in some environments, so I would like to keep the change as small as possible for legacy (well, I am going to drop the secret containing token, but thats another story).
I understand you don't want to risk breaking the legacy. But I have the feeling that we can create a solution keeping compatibility with previous logic. I will try to work on this.
Hey @gbonnefille , sorry for the delay. Past days were a bit more challenging that expected.
You can try to build from the latest branch mentioned in the pr above. so far in my tests given this object
apiVersion: gitlab.k8s.alekc.dev/v1beta1
kind: MultiRunner
metadata:
name: multirunner-simple
spec:
concurrent: 10
gitlab_instance_url: https://gitlab.com/
entries:
- name: runner 1
registration_config:
tag_list:
- dev1
- test-gitlab-runner
token: xxx
description: "Runner 1desc"
executor_config:
memory_request: "50Mi"
- name: runner 2
registration_config:
tag_list:
- dev2
- runner2
token: xxx
description: "Runner 2 desc"
A single runner (on kubernetes) is built and launched having following config:
apiVersion: v1
data:
config.toml: |
listen_address = ":9090"
concurrent = 10
check_interval = 3
log_level = "info"
[[runners]]
name = "runner 1"
limit = 10
url = "https://gitlab.com/"
token = "mSMjXSd-x"
executor = "kubernetes"
[runners.kubernetes]
host = ""
namespace = "default"
[[runners]]
name = "runner 2"
limit = 10
url = "https://gitlab.com/"
token = "xx"
executor = "kubernetes"
[runners.kubernetes]
host = ""
namespace = "default"
kind: ConfigMap
metadata:
creationTimestamp: "2022-10-19T19:49:50Z"
name: gitlab-mrunner-multirunner-simple
namespace: default
ownerReferences:
- apiVersion: gitlab.k8s.alekc.dev/v1beta1
controller: true
kind: MultirRunner
name: multirunner-simple
uid: 97867551-b586-4b4f-b2a3-249a00b17698
resourceVersion: "84100"
uid: 1ba25c67-0b60-4210-9dd0-402b658ff363
There are still a quiet bit of work to do before this sees the end release but, so far looks good for most aspects.
Todo list:
I'm actively looking for a way to have a single gilab-runner instance running multiple «runners» in order to create a single shared runner for dedicated groups/projects (cf. https://gitlab.com/gitlab-org/charts/gitlab-runner/-/issues/230) while controlling the number of total jobs via a single
concurrent
property.Do you think it make sense to improve your operator with a new CRD (for example SharedRunner) allowing to define multiple registrations while keeping a single gitlab-runner?