search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.18k stars 14.19k forks source link

Replacing ofBorg with Github actions #355847

Open Mic92 opened 2 days ago

Mic92 commented 2 days ago

This is one of the two plans to ensure we can also perform github evaluation checks in the future.

See https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025 for more information.

To replace OfBorg’s functions with GitHub Actions the following tasks need to be implemented:

I already created a proof of concept pull request here: https://github.com/NixOS/nixpkgs/pull/352808

Update

We have our first jitsi meeting to coordinate the migration on the 14.11 (today) at 17:00 UTC (18:00 Berlin time) at https://jitsi.lassul.us/nixos-infra

nixos-discourse commented 2 days ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025/2

Bot-wxt1221 commented 2 days ago

evaluation checks takes too many resource. I'm worried about if github action's machine can run it in reasonable time.

Mic92 commented 2 days ago

@Bot-wxt1221 I managed to run it in 5 minutes for naive nix-env evaluation based on the default.nix entry point and 15 minutes using the same logic that ofborg uses: https://github.com/Mic92/nixpkgs/actions/workflows/eval.yml

Both seem already faster compared to the hours of waiting for the ofborg queue that we experience today.

Also this is not yet the end of the line of optimizations. We still have https://github.com/Mic92/nixpkgs/blob/main/pkgs/top-level/release-attrpaths-superset.nix to split evaluation in smaller parts that can run even in parallel.

JohnRTitor commented 2 days ago

Will PR commands like @ofborg build hello be supported with GitHub action?

JohnRTitor commented 2 days ago

https://github.com/NixOS/nixpkgs/pull/352808#issuecomment-2475893708 and

I worry that bot accounts like ryantm-r can easily hit the limit of CI. CC @ryantm

Mic92 commented 2 days ago

@JohnRTitor

Yes it's possible:

name: Trigger on PR Comment

on:
  issue_comment:
    types: [created]

jobs:
  run-on-comment:
    if: github.event.issue.pull_request != null && contains(github.event.comment.body, '/build')
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v3
Mic92 commented 2 days ago

#352808 (comment) and

I worry that bot accounts like ryantm-r can easily hit the limit of CI. CC @ryantm

Well. We have to try and see. Just now it's speculation if it works or not.

JohnRTitor commented 2 days ago

Good to know, though huge builds like kernel and its modules, chromium and firefox will obviously not work. And we'll possibly have to setup a blacklist else even individual contributors will hit their limits.

Bot-wxt1221 commented 2 days ago

According to github doc:

https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions

GitHub Actions usage is free for standard GitHub-hosted runners in public repositories, and for self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage for use with GitHub-hosted runners, depending on the account's plan. Any usage beyond the included amounts is controlled by spending limits.

So maybe we don't need to worry about time?

Mic92 commented 2 days ago

Good to know, though huge builds like kernel and its modules, chromium and firefox will obviously not work. And we'll possibly have to setup a blacklist else even individual contributors will hit their limits.

You can run builds for 12h. Obviously we should establish some reasonable timeouts to be a good citizen in the ecosystem.

Mic92 commented 2 days ago

Added a ^ meeting date for this.

ibizaman commented 2 days ago

Maybe of interest for this issue, at least just for inspiration, but I've also (ab)used GitHub actions to build tests in my project using a dynamically generated matrix. My project uses flakes but this should be adaptable to non-flakes https://github.com/ibizaman/selfhostblocks/blob/main/.github/workflows/build.yaml This matrix then produces a big list of jobs, one job per test https://github.com/ibizaman/selfhostblocks/actions/runs/11502502422 like so: image

Mic92 commented 2 days ago

See the meeting notes for today's infra meeting where we mainly discussed the CI situation: https://github.com/NixOS/infra/blob/7688f20babbeb27a10e4d8669fffe4b0ed00e17f/docs/meeting-notes/2024-11-14.md

Here is the high-level plan:

Independently from meeting we also have other discussions about how we can develop ofborg in the future. However this might not happen before February, so we need some alternative solution in the meantime if not longer.

infinisil commented 2 days ago

I've opened a draft PR here for evaluating Nixpkgs using GitHub Actions: https://github.com/NixOS/nixpkgs/pull/356023. For just evaluation (and those only taking 5 minutes on each arch) instead of also building, I don't think we need to do the running-on-forks dance. Building is harder to get, but it's arguably also less important (and very orthogonal to evaluation).

nixos-discourse commented 2 days ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025/27

adisbladis commented 2 days ago

One important aspect that ofborg currently provides, and that this issue doesn't mention, is the performance report. This currently works by evaluating nixpkgs twice, once before the PR and once after.

For the majority of PRs the performance report is not important, but for work on lib & stdenv, it can be very important.

The report currently does not report the impact of checkMeta, something that has lead to a less than stellar review experience since contributors & reviewers don't actually understand the real performance impact.

Mic92 commented 1 day ago

One important aspect that ofborg currently provides, and that this issue doesn't mention, is the performance report. This currently works by evaluating nixpkgs twice, once before the PR and once after.

For the majority of PRs the performance report is not important, but for work on lib & stdenv, it can be very important.

The report currently does not report the impact of checkMeta, something that has lead to a less than stellar review experience since contributors & reviewers don't actually understand the real performance impact.

Could that be another on-demand GitHub actions job? We could even run automatically if certain paths has been changed.

azuwis commented 1 day ago

Good to know, though huge builds like kernel and its modules, chromium and firefox will obviously not work. And we'll possibly have to setup a blacklist else even individual contributors will hit their limits.

Building linux kernel is fine on Github Actions, the CPU time is sufficient, it takes less than 2 hours to build Jovian-NixOS linux kernel, and Github Actions offer max 6 hours per run.

The only concern is disk space, workarounds:

1) Bind mount /mnt/nix to /nix, /mnt is 66G free by default. 2) Set build-dir = /nix/var in nix.conf, by default nix use /tmp to hold /build in the sandbox, and takes up disk space in /, 20G free, not enough for building linux kernel. 3) Remove files we don't need, docker images, /usr/local, /usr/share/swift, etc. It's possible to get more than 63G free disk space in / without affecting nix. 4) Use BTRFS RAID0 to combine / and /mnt, and enable zstd compression, it's possible to get total 126 free disk space, and should be sufficient for most build tasks.

All of the above workarounds are implemented in https://github.com/azuwis/actions/blob/main/nix/prepare.sh.

Well, expect for 2), which can be set by:

    - uses: cachix/install-nix-action@v30
      with:
        extra_nix_config: |
          build-dir = /nix/var
adisbladis commented 17 hours ago

One important aspect that ofborg currently provides, and that this issue doesn't mention, is the performance report. This currently works by evaluating nixpkgs twice, once before the PR and once after. ...

Could that be another on-demand GitHub actions job? We could even run automatically if certain paths has been changed.

Sounds good to me.

JohnRTitor commented 16 hours ago

Building linux kernel is fine on Github Actions, the CPU time is sufficient, it takes less than 2 hours to build Jovian-NixOS linux kernel, and Github Actions offer max 6 hours per run.

I am concerned about building the kernel modules (both in tree and out of tree).

Mic92 commented 11 hours ago

Building linux kernel is fine on Github Actions, the CPU time is sufficient, it takes less than 2 hours to build Jovian-NixOS linux kernel, and Github Actions offer max 6 hours per run.

I am concerned about building the kernel modules (both in tree and out of tree).

Well. We should be quickly able to filter out and blacklist packages we don't want to build once the source of truth lives in the repository? Also we can actually stop github actions, which was not possible with ofborg builds.