Migrating pre-merge testing to foundation owned infrastructure

[ChristianKuehnel]

We started the proof of concept for pre-merge testing in Oct 2019 with a first usable version in April 2020 (or so). By now we have around 1.5 years of experience with that and the feature set has stabilized. We demonstrated that it is possible and we learned a lot around what works well and what the limitations are (performance, stability, resource requirements, Phabricator integration, ...).

I suppose this is a good time to decide how we move forward.

I basically see 2 options here:

1. If the community sees value in pre-merge testing, we should move from the proof-of-concept phase to a regular LLVM service with the some ideas outlined below.
2. If the community does not see value in this service, the logical consequence would be to turn it down.

current limitations

• Pre-merge testing was build on Google-internal Google cloud infrastructure to get things started and to demonstrate it's value and feasibility.
• The adoption of pre-merge testing is growing, with some projects (e.g. MLIR, libc++) as power users.
• Right now only @metaflow and @ChristianKuehnel have admin access to some of the infrastructure hosted on Google Cloud (per Google policy).
• The scripts and config files are hosted on a Google-owned repo.
• Improvements to the service have stalled, with limited community contributions.
• Pre-merge testing is not anchored in the developer policy, there is no agreement on how is should be used.

desired state

• There is community buy-in to pre-merge testing (at least for a certain scope and set of features).
• There is a technical design/documentation for the pre-merge infrastructure when run by the community/foundation.
• The developer policy was updated to reflect the current expectations and agreements around pre-merge testing.
• Pre-merge testing is running on infrastructure owned by the LLVM foundation. The old infrastructure was turned down.
• A trusted group of people have admin access for the migration, maintenance and improvements.
  • The migration is done by this admin group.
  • The admin is driving future improvements of the service.
• The scripts and config files are in an official LLVM repo (see issue #186), as part of the migration. The old repo was archived.

[metaflow]

Some updates from me: I am in the middle of writing an internal doc of how things are set up, that should definitely help whoever is going to migrate infra to other platform - I will release portions of it later. Then I am planning to move the scripting part to LLVM github.

[ChristianKuehnel]

from meeting on 2022-01-19:

• next steps:
  • @metaflow is writing an overview of the current setup
  • @ChristianKuehnel figure out funding this through the LLVM foundation
  • Figure out what community members would be interested in helping maintain pre-merge testing
  • Handing this over to the foundation is blocked on #27

[ChristianKuehnel]

Some updates:

• Google might be able to donate Google Cloud credits to the LLVM foundation, so that the foundation can run the infrastructure needed. We should check if other LLVM sponsors are willing to donate some cloud resources as well.
• @tlattner will discuss proposal with the board

[ChristianKuehnel]

from meeting on 2022-02-15:

• @metaflow discussing with Galina about migration of code into llvm-zorg (or creating a new repo or the monorepo).
• We would need community support for the migration from Google-internal to a public cloud instance, also to get familiar with the infrastructure.

[ChristianKuehnel]

from meeting on 2022-03-29:

• LLVM foundation willing to take over the infrastructure for pre-merge testing and Phabricator.
• Google-internal discussion on donation of cloud credits is still ongoing.

[ChristianKuehnel]

from meeting on 2022-07-06, update from Christian:

• Most approvals completed, no blockers
• scope of the Google cloud donation and migration of services:
  • pre-merge testing
  • phabricator
  • LNT server/website behind http://lnt.llvm.org/
  • bazel build machines
• proposal for migration: move VMs/databases/... over as they are now. Do updates/improvements/changes where needed in the LLVM ecosystem so the whole community can participate.
• Christian still needs to update LLVM board
• Christian to announce this on Discourse, get input from the community

[ChristianKuehnel]

The donation of GCP credits was done, we can not start migrating the services. I'm creating separate issues to discuss the migration of the individual services.