Open gsaslis opened 1 year ago
creating the pipeline in concourse can work with:
More discussion about this feature here:
Commands excuted:
fly -t tutorial login -c http://localhost:8080 -u test -p test
fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
fly -t tutorial trigger-job --job hello-world/hello-world-job
fly -t tutorial trigger-job --job hello-world/hello-world-job --verbose
2023/06/23 11:14:32 GET /api/v1/info HTTP/1.1
Host: localhost:8080
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip
2023/06/23 11:14:32 HTTP/1.1 200 OK
Content-Length: 299
Cache-Control: no-store, private
Content-Security-Policy: *
Content-Type: application/json
Date: Fri, 23 Jun 2023 08:14:32 GMT
Vary: Accept-Encoding
X-Concourse-Version: 7.9.1
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: allow
X-Xss-Protection: 1; mode=block
{"version":"7.9.1","worker_version":"2.4","feature_flags":{"across_step":false,"build_rerun":false,"cache_streamed_volumes":false,"global_resources":false,"pipeline_instances":false,"redact_secrets":false,"resource_causality":false},"external_url":"http://localhost:8080","cluster_name":"tutorial"}
2023/06/23 11:14:32 POST /api/v1/teams/main/pipelines/hello-world/jobs/hello-world-job/builds HTTP/1.1
Host: localhost:8080
User-Agent: Go-http-client/1.1
Content-Length: 0
Accept-Encoding: gzip
2023/06/23 11:14:32 HTTP/1.1 200 OK
Content-Length: 185
Cache-Control: no-store, private
Content-Security-Policy: *
Content-Type: application/json
Date: Fri, 23 Jun 2023 08:14:32 GMT
Vary: Accept-Encoding
X-Concourse-Version: 7.9.1
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: allow
X-Xss-Protection: 1; mode=block
{"id":11,"team_name":"main","name":"11","status":"pending","api_url":"/api/v1/builds/11","job_name":"hello-world-job","pipeline_id":1,"pipeline_name":"hello-world","created_by":"test"}
started hello-world/hello-world-job #11
@Archimidis Here is how a pipeline can be created in concourse and triggered to run:
$ curl -X POST --location "http://127.0.0.1:8080/sky/issuer/token" \
-H "Authorization: Basic Zmx5OlpteDU=" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=password&username=${CONCOURSE_CI_USER}&password=${CONCOURSE_CI_PASS}&scope=openid%20profile%20email%20federated:id%20groups"
Get access token from response and use in the Authorization: Bearer ${TOKEN}
HTTP header in subsequent requests.
$ curl -X PUT --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/config" \
-H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA" \
-H "Content-Type: application/x-yaml" \
-H "x-concourse-config-version: 1" \
-d "jobs:
- name: configure-pipeline
plan:
- get: ${RADICLE_PROJECT_NAME}
version: ${PATCH_HEAD}
trigger: false # we don't want concourse to trigger jobs that the radicle-ci-broker won't know about
- set_pipeline: ${RADICLE_PROJECT_ID} # this creates a new pipeline and sets its config to that of the file discovered in the git resource
file: ${RADICLE_PROJECT_NAME}/.concourse/config.yaml # by convention, we'll require the initial concourse pipeline to be in `.concourse/config.yaml`
resources:
- name: ${RADICLE_PROJECT_NAME}
type: git
icon: git
source:
uri: ${RADICLE_DOMAIN}/${RADICLE_PROJECT_ID}.git
branch: ${PATCH_BRANCH}"
curl -X PUT --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/unpause" \
-H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"
It is important to understand that this build we are triggering is on the ${RADICLE_PROJECT_NAME}-configure
pipeline, so this build does not actually run whatever pipeline is configured in the project's git repo. It only triggers the update of the ${RADICLE_PROJECT_NAME}
pipeline in concourse from the one in the repo.
$ curl -X POST --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}-configure/jobs/configure-pipeline/builds" \
-H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"
This is necessary for step 6
below.
$ curl -X GET --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}/jobs" \
-H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"
the name of the first job can be found by e.g. piping the output to | jq '.[0].name'
$ curl -X POST --location "${CONCOURSE_CI_FQDN}/api/v1/teams/main/pipelines/${RADICLE_PROJECT_ID}/jobs/${JOB_NAME}/builds" \
-H "Authorization: Bearer ow71g/e1EBPRYuWm2yjBY5aqqsMnhZ1kAAAAAA"
from the response, we can keep the build id to later watch for the job status, etc..
And here is an example of .concourse/config.yaml
---
jobs:
- name: my-job
plan:
- task: my-task
config:
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
run:
path: echo
args: ["Hello, Nikos!"]
please note that this offers a lot of flexibility in either defining a specific pipeline, or using the concourse set_pipeline
steps to create additional pipelines, etc. etc.
The concourse implementation:
fly
CLI from the concourse server that the CI broker has been configured withwith that in place, it has 3 (or maybe 2) roles to fill:
Setup: Create and unpause a pipeline in concourse
fly login
Trigger jobs
There are 2 alternative approaches we are considering:
fly
CLI or concourse HTTP API)Triggering should only trigger the job in concourse without watching for the output. We need to capture the build ID from the output and then add this build ID to a data structure of "patches with pending builds".
Watch for results and update Patch with outcome
In terms of watching for the results, we should create some worker thread pool that:
In terms of "updating the Patch", we will be adding a simple comment to the Patch for now.
NOTES:
fly
CLI and web app uses as opposed to wrapping thefly
CLI itself.radicle-ci-broker
write to an sqlite database the information about pending jobs and forradicle-httpd
to read from that (with an added benefit that therad
cli could then read that information and incorporate that into the Terminal User Interface (TUI).