jadecarino commented 6 months ago

Story

As a contributor to the galasa open source project, I want builds to be in the open, and use github actions, so that I can see build logs, see built artifacts, benefit from integration with github actions, and all without using any resources within the secretive IBM firewall.

As a developer of galasa open source code, I want to try out building a single galasa component to see if github actions are OK to use, and the expected benefits can be observed. We need to add a parallel build process for the build utils component so we have a practical idea of how it compares to tekton, so we can get some experience with github actions, and use of the related artifact stores.

Background

There are many benefits of moving our infrastructure to use github actions.

Builds will be immediately kicked off, no 2-minute delay for them to start.
build logs are available for ages
built artifacts can be made available within the build results

To test these theories, we have to start small, with a small investment in a spike story. This story.

Objectives:

Learn GH Actions skills
Side by side comparison with Tekton pipeline
Try out the Docker registry

For reference, look at the existing Tekton pipelines for buildutils, but remove/adapt Tasks as appropriate for GitHub actions.

Tasks

[x] Copy all Dockerfiles used in the buildutils pipelines into the 'buildutils' repository into a dockerfiles subdirectory (don't move them as we don't want to disturb the Tekton builds)
[x] Use GitHub actions to build the 'buildutils' repository similar to the existing Tekton pipelines but adapt where necessary
[x] Build the galasabld and openapi2beans binaries into Docker images that contain the executable (see tasks build-galasabld-executable, build-galasabld-ibm-executable, and build-openapi2beans-executable)
[x] Push the Docker images to the GitHub Container Registry if doing a main workflow, just build if PR
[ ] Investigate how to make downloadable links to each executable available from GitHub packages, like at this URL (see below)
[ ] Store the executables as downloadable links that can be downloaded from GitHub packages (see below)
[ ] Do a Maven build to wrap the buildutils and openapi2beans executables inside a Maven artefact (similar to how the galasactl executables are wrapped into a Maven artefact in the Isolated build - see pomGalasactl.xml and the Task)
[ ] Publish the Maven artefact to the GitHub Maven Registry
[x] The GitHub actions workflow should be integrated with PRs as well as merges into the main branch, so we see the tekton and github actions build processes side by side
[x] We gather metrics where we can, so we can discuss it further to see what we need to do next

jaydee029 commented 1 month ago

Hey @jadecarino some of the docker images fetch the base image from galasa.harbor.dev, i am assuming this needs to be changed right? for ex FROM harbor.galasa.dev/docker_proxy_cache/library/httpd:2.4.59

do we use substitutes from docker hub or similar or any specific base image needs to be used ?

jadecarino commented 1 month ago

Hi @jaydee029 - for now, this can continue to pull from harbor.galasa.dev. I have opened another work item here to create a Docker proxy cache in GHCR which we will eventually use instead of Harbor's Docker proxy cache, but this work item isn't scheduled into the iteration yet. For now, continue to pull from harbor.galasa.dev, and when the issue I just mentioned is complete, we will update all of the FROM tags throughout the Dockerfiles.

techcobweb commented 1 month ago

Note from Druv

I have made some progress in my forked repository here , since I didn't raise a draft PR I have written about the changes in detail here

Comments from Mike C

It's looking good so far. Very encouraging start Druv. Thanks for sharing. I think using the ubuntu runner is ok for things like make and docker because the versions of those don't tend to matter. When it comes to gradle and maven we may have a different story... given the Galasa builds need specific versions of those right now, unless you can control the version used from the basic runner ? A PR even in draft might be a good thing to do as it's a great way of collecting review comments...etc. even before you are ready for it to be reviewed. (for example, we'd be having and recoding these comments in the PR, where others can read them later on, and you could avoid the readme file debris in the repository). We could also explore ideas for a naming strategy for the images also in the story perhaps ? In terms of names, I suspect using :latest might be something we want to reserve for the very latest docker image of the main build. eg: galasabld:main and galasabld:pr14. I think galasabld: or something similar might be the way to go, so if we build from a tag we'd get the tag version in there. eg: galasabld:v0.34.0 perhaps.