Deploy a "simple" Github runner

[benoit74]

For https://github.com/openzim/zimit/issues/402, we would like to deploy a simple Github runner.

This runner might be used for few other cases where this runner will be superior to Github runners. This is probably going to be rare cases since we have to maintain our own image, we do not have access to Github runner images.

TbC how this should be done (Docker container on one of our k8s node? which node? dedicated small cloud instance? ...)

[benoit74]

For various reasons exposed in https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#self-hosted-runner-security, I recommend to:

• create a new Cloud instance (Hetzner or Scaleway?), probably a very small one is sufficient, we do not plan to start many jobs
• create a new private repo (kiwix/private-workflows?) for configuring/hosting workflows running on this machine
• enable the runner only on this private repo

On this machine I would setup Github runner application manually, and install missing packages if needed (e.g. install Docker since we use this in our zimit job).

WDYT?

[kelson42]

Code coming from non-org members should go through validation before running the workflows. This is our approach to secure things. Doing this allows us to run workflows in a secure manner at the exact place where they make sense (on public repositories too).

[benoit74]

From my PoV, this seems a bit fragile to resort only to proper workflow configuration and code review.

First because code review of workflow is hard, especially regarding security.

Second because all our maintainers (e.g. GSoC students, ...) have write access to their repo(s), and hence are not subject to the workflow validation.

I understand that not having the workflow in a public repository is a pain.

Maybe a middle-ground is to continue working on existing public repos but still create a dedicated virtual machine with "nothing" on it, considering this machine might be compromised at any point in time?

[kelson42]

Maybe a middle-ground is to continue working on existing public repos but still create a dedicated virtual machine with "nothing" on it, considering this machine might be compromised at any point in time?

I'm in favour of this and had the feeling this is already what you propose (you have written "create a new Cloud instance").

[benoit74]

Do you have any idea of which specs we should target for the new VM?

I would recommend 1 CPU and 8G RAM, with at least 40-50G SSD disk). Do you have more insight?

Should we continue with Scaleway (despite recent support issues) or continue to test Hetzner (since this is not a critical service at all)?

[benoit74]

I'm going to interpret the absence of feedback as an implicit "do the best you can, I don't know and we will fix what's wrong afterwards".

[benoit74]

I've order a CAX21 server at Hetzner Helsinki. Important: it's an arm64 machine. Wait and see if it is a good decision ^^

Bare VM setup is ready, host is at github-runner.kiwix.org

I'm going forward with runner setup.

[benoit74]

Machine finally re-provisionned on amd64 (CX32), zimit daily tests needs chrome-for-tests ... which is available only on amd64 on Linux, not arm64 ... Price difference is negligible, IPs kept identical.

Procedure is documented at https://github.com/kiwix/operations/wiki/Restore-Github-Runner-VM

[benoit74]

This is still not working ... because Youtube is still blocking us ... they have probably blacklisted the whole AS IP ranges from Hetzner ...

Not sure how to move this forward: use ProtonVPN (or something else) to mask our IP (not sure it is very reliable for accessing Youtube) or move to another cloud provider?

[rgaudin]

Not specifically tested with ProtonVPN but in my experience, all VPN IPs are painful to use on such platforms: captcha, etc which is to be expected since those are shared across a large number of users. I wouldn't go that route.

[benoit74]

I've deleted the github-runner from Hetzner for now so that we do not pay for something useless.