CI/CD Pipeline Overhaul 2022

[feisuzhu]

This is an on-going work primarily to reduce CI/CD build time and improve maintainability. Discussions and feature requests are welcome.

The status quo

Buildbots

• 5 Linux, 3 + 2(VM) Windows, 3 macOS, organization shared

Pipeline

• release.yml: Build nightly wheels, run complete tests, upload to pypi. GA releases also goes here.

• testing.yml: Build PR and master, run partial or complete tests.

• perf.yml: Performance monitoring, only run on master.

Problems

Code & logic duplication

• release.yml is largely the same with testing.yml except publishing and different build matrix.
• Different code path for CPU and GPU building tasks.

Slow build

Latency !

• 1h30min for each complete run (master branch) (!!)
• Slightly over 1h for each PR run (!!!)

Wasted cycles & bandwidth on same things

• For now C++ build cache are stored locally on each buildbot machine.
• Auxiliary git repos are not cached at all.
• Perf Monitoring and Android Demo needs to compile taichi binary separately.
• Pulling huge docker images for every run on GitHub Hosted Runners (cost 30min).
• ti.init() dominates test case time consumption

Buildbot cluster of a low capacity

• Only a single M1 Mac Mini operating (SPOF)
• Unlimited(?) but weak GitHub Hosted Runners, better not to have them.
• Beefy runners are prohibitively expensive.

Under-utilized buildbot cluster

• Cores sitting idle due to extra low usage when running tests
• Different test suites run in serial

Strong coupling with GitHub Actions

• Difficult for developers to run workflows locally

Non-reproducible building environment (partially addressed)

• Windows and macOS builds still run on bare metal
• Buildbots are provisioned manually

Minor mess

• Most of buildbot tags are useless

Proposed changes

• [ ] Exercise single responsibility principle in workflow definition
  • [x] Merge CPU & GPU tasks
  • [ ] Separate build & test & release & perf into 4 different stages
    • Also prevents from compiling multiple times
    • Guarantees Perf Mon monitors the same wheel published
    • Resolves: Code & logic duplication

• [ ] Cluster shared caching mechanism

  • sccache running at cluster scale
  • Resolves: Slow build, cluster utilization
  • Possible blocker: Windows build is currently using ccache because sccache can't recognize a particular compiler option.

• [x] apt & pip caching proxy

  • [x] squid-deb-proxy
  • [x] PyPI cacing (https://gist.github.com/dctrwatson/5785638)
  • Resolves: Slow build, cluster utilization

• [x] Home grown git clone caching proxy

  • Open source solutions are not polished (goblet, jonasmalacofilho/git-cache-http-server, etc)
  • Long term, may not implement this time
  • Could grow up to a non-trivial open source project.
  • Could benefit satellite projects like soft2d as well.
  • Resolves: Slow build, cluster utilization

• [x] Adding new buildbots

  • 15 x HPE ProLiant DL360 Gen 9 with 2x Xeon E5-2696v4 + 128GB RAM + Tesla P4 (total 44 cores 88 threads, CNY 5k per node)
  • 40GbE interconnect via Mellanox CX3 Pro, may add InfiniBand if necessary
  • macOS 10.14 phased out, the Mac Pro will be upgraded to 10.15
  • New M1 Mac Mini will join buildbot cluster after Monterey issue resolved
  • Resolves: Slow build

Go distributed

• [ ] Distributed compiling

  • [x] sccache running at cluster scale (sccache can do both caching and distributed compiling)
  • [ ] Unify toolchain
  • [ ] Cross compiling (build wheels for all platforms/archs on Linux/x86_64 machines)
    • Will continue to support native compiling workflow on Windows & macOS
  • Could enable https://github.com/taichi-dev/taichi/issues/5924
  • Resolves: Slow build, cluster utilization

• [ ] Distributed testing

  • Long term, will only start if necessary
  • Long running daemons stealing testing work from distributor
  • Compile taichi kernels, and then run, on different machines
    • Can speed up and test offline cache at the same time
  • Resolves: Slow build, cluster utilization

• [x] Runner roles

  • Build runners coexist with test runners on same machine
  • Build runners run tasks with higher nice (lower priority)
  • Resolves: Cluster utilization

• [ ] Migrate to dagger (or other similar solutions)

  • Low priority, may not implement this time
  • Resolves:
    • Strong coupling with GitHub Actions
    • Ability to run pipelines locally
    • Non-reproducible building environment

• [x] Drop github hosted runners

  • Resolves: Strong coupling with GitHub Actions, slow build

• [x] Infrastructure states condense into ansible code

  • [x] Install common tools, CUDA, docker
  • [x] Configure GitHub Action runners
  • [x] sccache and redis infrastructures
  • Resolves: Non-reproducible building environment

• [ ] Containerize all the things !

  • [x] Linux & Windows+CPU containers are already done
  • [x] Windows+GPU process isolation container
  • macOS on Apple M1 containers may not be feasible
  • [ ] macOS x86 containers by sickcodes/Docker-OSX
  • Resolves: Non-reproducible building environment

Miscellaneous

• [ ] Buildbot tags cleanup
• [ ] Issues/Actions dashboard
• [x] buildbot resource monitoring

Non-goals

Virtualization &&/|| Kubernetes integration

• Both are fun tasks not on critical path, leaving them to (for now non-existent) interns

[imcom]

Dapper now provides Go SDK, dev may use Go to describe its CICD

[feisuzhu]

According to the document, containerization w/ GPU on Windows is non-feasible now(requires CUDA & OpenGL). Moving all workloads to bare metal on Windows.