Create Concourse implementation for CI runner

gsaslis commented 1 year ago

The concourse implementation:

relies on a pre-downloaded fly CLI from the concourse server that the CI broker has been configured with

with that in place, it has 3 (or maybe 2) roles to fill:

Setup: Create and unpause a pipeline in concourse

acquires token from concourse with fly login
- need some kind of mechanism that handles re-authenticating / authentication errors (and re-logins).
creates pipeline (more on this later)

Trigger jobs

There are 2 alternative approaches we are considering:

let the broker trigger the jobs (using the fly CLI or concourse HTTP API)
set up a pipeline in concourse that uses a git resource to watch for events triggering the pipeline. This has the added benefit that we don't need to do the triggering, but we would still need to find out about these events somehow, so we can watch for them later on and update the job status.

Triggering should only trigger the job in concourse without watching for the output. We need to capture the build ID from the output and then add this build ID to a data structure of "patches with pending builds".

Watch for results and update Patch with outcome

In terms of watching for the results, we should create some worker thread pool that:

looks at pending builds
for each build, uses some polling policy on the job status endpoint to verify whether the build has finished (whether failed or succeeded)
once the job has finished, it is time to update the patch with a comment.

In terms of "updating the Patch", we will be adding a simple comment to the Patch for now.

NOTES:

it might make sense to use the API that the fly CLI and web app uses as opposed to wrapping the fly CLI itself.
try to maintain flat hierarchy of projects (under which pipelines get created) in concourse to match flat hierarchy in radicle
we don't want to persist job logs in radicle (can get very messy - very fast)
we discussed at length about what would be the best way to expose a ci_server-agnostic HTTP API endpoint that can be used by clients (web, etc.) to show job status, etc. Most preferred approach for the time being is to have the radicle-ci-broker write to an sqlite database the information about pending jobs and for radicle-httpd to read from that (with an added benefit that the rad cli could then read that information and incorporate that into the Terminal User Interface (TUI).

gsaslis commented 1 year ago

creating the pipeline in concourse can work with:

https://concourse-ci.org/set-pipeline-step.html

More discussion about this feature here:

https://github.com/concourse/concourse/discussions/5732

Archimidis commented 1 year ago

Commands excuted:

fly -t tutorial login -c http://localhost:8080 -u test -p test
fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
fly -t tutorial trigger-job --job hello-world/hello-world-job

Archimidis commented 1 year ago


fly -t tutorial trigger-job --job hello-world/hello-world-job --verbose
2023/06/23 11:14:32 GET /api/v1/info HTTP/1.1
Host: localhost:8080
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

2023/06/23 11:14:32 HTTP/1.1 200 OK
Content-Length: 299
Cache-Control: no-store, private
Content-Security-Policy: *
Content-Type: application/json
Date: Fri, 23 Jun 2023 08:14:32 GMT
Vary: Accept-Encoding
X-Concourse-Version: 7.9.1
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: allow
X-Xss-Protection: 1; mode=block

{"version":"7.9.1","worker_version":"2.4","feature_flags":{"across_step":false,"build_rerun":false,"cache_streamed_volumes":false,"global_resources":false,"pipeline_instances":false,"redact_secrets":false,"resource_causality":false},"external_url":"http://localhost:8080","cluster_name":"tutorial"}

2023/06/23 11:14:32 POST /api/v1/teams/main/pipelines/hello-world/jobs/hello-world-job/builds HTTP/1.1
Host: localhost:8080
User-Agent: Go-http-client/1.1
Content-Length: 0
Accept-Encoding: gzip

2023/06/23 11:14:32 HTTP/1.1 200 OK
Content-Length: 185
Cache-Control: no-store, private
Content-Security-Policy: *
Content-Type: application/json
Date: Fri, 23 Jun 2023 08:14:32 GMT
Vary: Accept-Encoding
X-Concourse-Version: 7.9.1
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: allow
X-Xss-Protection: 1; mode=block

{"id":11,"team_name":"main","name":"11","status":"pending","api_url":"/api/v1/builds/11","job_name":"hello-world-job","pipeline_id":1,"pipeline_name":"hello-world","created_by":"test"}

started hello-world/hello-world-job #11

gsaslis commented 1 year ago

@Archimidis Here is how a pipeline can be created in concourse and triggered to run:

1. Obtain credentials

$ curl -X POST --location "http://127.0.0.1:8080/sky/issuer/token" \
    -H "Authorization: Basic Zmx5OlpteDU=" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "grant_type=password&username=${CONCOURSE_CI_USER}&password=${CONCOURSE_CI_PASS}&scope=openid%20profile%20email%20federated:id%20groups"

Get access token from response and use in the Authorization: Bearer ${TOKEN} HTTP header in subsequent requests.

2. Create a new pipeline in concourse that will read pipelines from the git repo

$ curl -X PUT --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/config" \
    -H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA" \
    -H "Content-Type: application/x-yaml" \
    -H "x-concourse-config-version: 1" \
    -d "jobs:
- name: configure-pipeline
  plan:
  - get: ${RADICLE_PROJECT_NAME}
    version: ${PATCH_HEAD}
    trigger: false # we don't want concourse to trigger jobs that the radicle-ci-broker won't know about
  - set_pipeline: ${RADICLE_PROJECT_ID}  # this creates a new pipeline and sets its config to that of the file discovered in the git resource
    file: ${RADICLE_PROJECT_NAME}/.concourse/config.yaml  # by convention, we'll require the initial concourse pipeline to be in `.concourse/config.yaml` 

resources:
- name: ${RADICLE_PROJECT_NAME}
  type: git
  icon: git
  source:
    uri: ${RADICLE_DOMAIN}/${RADICLE_PROJECT_ID}.git
    branch: ${PATCH_BRANCH}"

3. Unpause the pipeline

curl -X PUT --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/unpause" \
    -H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"

4. Trigger the pipeline configuration

It is important to understand that this build we are triggering is on the ${RADICLE_PROJECT_NAME}-configure pipeline, so this build does not actually run whatever pipeline is configured in the project's git repo. It only triggers the update of the ${RADICLE_PROJECT_NAME} pipeline in concourse from the one in the repo.

$ curl -X POST --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/jobs/configure-pipeline/builds" \
    -H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"

5. Get the name of the first job in the project's pipeline

This is necessary for step 6 below.

$ curl -X GET --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}/jobs" \
    -H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"

the name of the first job can be found by e.g. piping the output to | jq '.[0].name'

6. (finallly!) trigger the job in the project's pipeline

$ curl -X POST --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}/jobs/${JOB_NAME}/builds" \
    -H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"

from the response, we can keep the build id to later watch for the job status, etc..

gsaslis commented 1 year ago

And here is an example of .concourse/config.yaml

---
jobs:
- name: my-job
  plan:
  - task: my-task
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: busybox}
      run:
        path: echo
        args: ["Hello, Nikos!"]

please note that this offers a lot of flexibility in either defining a specific pipeline, or using the concourse set_pipeline steps to create additional pipelines, etc. etc.

gsaslis / radicle-ci-broker