Investigate Prow for Codewind PR test coverage on Kube

[johnmcollier]

Background

As part of our long-term plan to get test coverage on Kubernetes for our Codewind PRs (and knowing we can’t run Kube on the Eclipse VPS) we should consider looking at Prow to supplement Jenkins and to get Kube test coverage.

Prow is a CI/CD solution designed for Kubernetes and is used by a number of Kube projects like: Tekton, OpenShift, Istio, Jenkins-X, and Kubernetes itself. From the Prow repo, some notable features that it has are:

• Job execution for testing, batch processing, artifact publishing.
  • GitHub events are used to trigger post-PR-merge (postsubmit) jobs and on-PR-update (presubmit) jobs.
• Pluggable GitHub bot automation that implements /foo style commands and enforces configured policies/processes.
• GitHub merge automation with batch testing logic.
• Front end for viewing jobs, merge queue status, dynamically generated help information, and more.
• High availability as benefit of running on Kubernetes. (replication, load balancing, rolling updates...)
• JSON structured logs.

For Codewind, we could configure a job to set up a Codewind on Che install and run all of our existing functional tests. We might also have another job that runs our end-to-end tests. We could configure these to be triggered automatically, or manually by a committer (with a comment like /test all to the PR). If we wanted, we could extend it even further and look at using prow for merge automation (have Prow automatically merge PRs that meet certain criteria). Prow jobs also just run as pods/containers on the cluster (or the remote cluster), meaning there's much less to worry about with regards to cluster setup for the tests.

Prow can even be run its jobs against remote clusters, allowing us to have test coverage against multiple distributions of Kubernetes, without having to define multiple jobs,.

Todo:

We should investigate Prow's feasibility for Codewind to see how well it could work for us and what all would need to be done to get it running:

• [ ] Investigate what's needed to get Prow running on a cluster
  • As Prow uses Webhooks, it would need to run on an external cluster in order for Github to communicate with it
  • Prow's footprint isn't too big, so would just need a cluster that can handle a few concurrent instances of Codewind
• [ ] Investigate what's needed to setup a Github bot account with the roles Prow needs.
  • The bot would need write access to the Codewind repos (and likely maintainer or admin status too). We would need to reach out to Eclipse Webmaster if this is okay
    • Other Eclipse projects (like Che) have their own bots with write access, so this should be fine
• [ ] Determine what level of integration we want with Prow
  • Do we just want it to run Kube tests on PRs? Do we want it to run scheduled / nightly tests too?
  • How do we want it to supplement our JIPP as best as possible?
• [ ] Test Prow on an internal cluster
  • We're already running our nightly tests on an internal OKD cluster, let's see if we can get them running as part of a Prow job
  • While the cluster we'd use for the Codewind repos would need to be external, we could still do a proof of concept on an internal cluster

[johnmcollier]

Investigated this further this week and got Prow up on IKS.

Adding back the tech-topics label to discuss the following: 1) IKS cluster to run the tests

• Need to install Prow on the cluster and expose it over the clusters ingress 2) Github bot account (such as codewind-test or codewind-bot, or something more creative)
• Needs to have the public_repo and repo:status scopes for the Codewind repositories that we want prow to run on
  • Eclipse webmaster needs to do this 3) Add the prow hook URL as a webhook to the Github repos that will need access to our prow tests
    • Eclipse webmaster needs to do this (I think) 4) Containerize our Kube test setup scripts (PFE and sidecar functional).
    • Standalone issue will be opened for this

Optional 1) Set up a Google Cloud Storage (GCS) account for bucket storage to store the logs from the prow jobs

• Prow can be configured to send the logs from its jobs to a GCS bucket so that they're publicly available (especially for those opening PRs)
• Prow recommends this for Open Source projects

[deboer-tim]

Discussed on tech topics, no objections. Need to pull Carl in to confirm things can be locked down securely. Will estimate GCS requirements and confirm we can fund it.

[johnmcollier]

So it looks like Prow supports running tests on a separate target cluster (separate from where Prow itself is running). However, it's only supported on Google Kubernetes (GKE) right now, and not on IKS or ROKS.

[johnmcollier]

Adding back the techtopics label to discuss options from a security perspective this Thurs

@cccanderson FYI