search

llvm / llvm-iwg

The LLVM Infrastructure Working Group
https://foundation.llvm.org/docs/infrastructure-wg/
Other
17 stars 14 forks source link

Create a proof-of-concept self-hosted runner for Github Actions #66

Open tstellar opened 3 years ago

tstellar commented 3 years ago

We should create a proof-of-concept self-hosted runner for Github Actions and enable it for some of the existing CI jobs in the release branch. Once this is working, we should document how to do this, so that community members can easily create their own self-hosted runners.

lazyparser commented 3 years ago

I'm just curious that how many workers/bots are sufficient for LLVM project. I had built a few internal CI for LLVM. It taks ~15mins for clean build and ~10mins to run the regression tests on 52 core Xeon 2.5GHz / 200GB RAM VM.

ChristianKuehnel commented 3 years ago

I guess that mostly depends on which and how many configurations we want to build...

On Fri, Aug 20, 2021 at 9:31 AM Wei Wu @.***> wrote:

I'm just curious that how many workers/bots are sufficient for LLVM project. I had built a few internal CI for LLVM. It taks ~15mins for clean build and ~10mins to run the regression tests on 52 core Xeon 2.5GHz / 200GB RAM VM.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/llvm/llvm-iwg/issues/66#issuecomment-902495210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYJMCAWKNQPVHIUKZMKNBTT5YAERANCNFSM5CPTBHMA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

lazyparser commented 3 years ago

So I've create two type of runners for a downstream llvm repo.

Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

CI for pull requests are a bit complicated. I forked the runner that the riscv v8 team used, and set up a CI runner for pull requests successfully. the repo is here. It written in JavaScript, and the source code is kinda straight forward. feel free to check it out.

tstellar commented 3 years ago

So I've create two type of runners for a downstream llvm repo.

Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

Are you interested in using your runner for some release branch testing?

tstellar commented 3 years ago

There is some interested in setting up a self-hosted runner on Mac hardware, so we are going to request Mac resources from the LLVM Foundation.

lazyparser commented 3 years ago

So I've create two type of runners for a downstream llvm repo. Turns out the runner for github actions (or post commit runners) are kinda easy to set up: I just followed the github docs and the commands showed in the setting tab of the repo.

Are you interested in using your runner for some release branch testing?

Yes. Will do. I can set up 2~3 runners in next week.

lazyparser commented 3 years ago

There is some interested in setting up a self-hosted runner on Mac hardware, so we are going to request Mac resources from the LLVM Foundation.

I have two mac mini for (RISC-V) CI purpose. turned out that neither the x86 mini or M1 mini with the 8GB memory could not build LLVM quickly enough. for pre or post commit CI. IMHO the iMac or Mac Pro might suit.

If we just want to a nightly build, then it would be ok to use mac mini as a start.

tstellar commented 3 years ago

@lazyparser The Github runners take 2hr 20min to build and run make check for llvm (and no other sub-projects). Anything that is faster than that would be an improvement.

lazyparser commented 3 years ago

@lazyparser The Github runners take 2hr 20min to build and run make check for llvm (and no other sub-projects). Anything that is faster than that would be an improvement.

It takes 7-10 minutes for building (clang+llvm on x86 and RISCV) and 8-12 minutes for regression testing on the 128c200g VM. For a 32c128g VM it takes 30-35 minutes.

These workers(VMs) are idle most of the time, because my team had turned to upstream directly. I'd like to use the VMs for LLVM building. Either adding IWG members to root access or config actions on the VMs myself are good to me.

btw thete is yet another daily CI for LLVM created by me: https://ci.rvperf.org/job/upstream-llvm-main-branch/

lazyparser commented 3 years ago

Not sure If one project can has multiple build workers. If so, then perhaps I can connect 2 big workers and a few medium workers to github actions.

lazyparser commented 2 years ago

So what should to be done next?

tstellar commented 2 years ago

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

lazyparser commented 2 years ago

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

If we want to use self-built runner for PRs, then

  1. a webhook need to be added.
  2. a github token that can access and comment on the repo should be created.

We can start from the actions.

tstellar commented 2 years ago

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

I don't understand what this is, can you clarify?

lazyparser commented 2 years ago

@lazyparser Do you have a runner set up that we can add to the llvm-project repo?

Yes. For the action, just copy the command list on the action tab (Please send it in private email or other non-public channel).

I don't understand what this is, can you clarify?

If we want to add a new runner for github actions, we can follow

https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository

For example:

image

Press the add new self-runner button, then

image

You can see several hash string in the command script. That is the secrets/token for the runner which should be keep private.

The other way is that I can provide an account on the runner VM, so that you (or other maintainer) can log into the runner VM and run the commands directly.

ChristianKuehnel commented 2 years ago

meeting on 2021-10-26: