Use caching CI builds

[mbrandenburger]

The current CI build could be improved to leverage caching. In particular, installing software dependencies (mostly in go) is time consuming and may benefit from caching.

• [x] go caching #762
• [ ] fabric binaries
• [ ] docker builds

[osamamagdy]

Hello @mbrandenburger, can I start working on this?

[mbrandenburger]

Hi @osamamagdy, yes please give it a try.

[osamamagdy]

Thanks @mbrandenburger. I started by investigating how to solve the issue. Here is what I found:

1. To cache any folder with github actions we need to configure a cache key and a path for the artifacts to be cached upon every execution of the pipeline. Normally for projects using golang, it would be $GOPATH/pkg/mod, and if $GOPATH=~/go the caching step would be - name: Cache Go dependencies uses: actions/cache@v2 with: path: ~/go/pkg/mod key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
2. The most time-consuming step is the run make inside dev container step.
3. Since the go dependencies installation is running inside a docker container, we can mount a volume to the location where the packages are installed ($GOPATH value inside the container is /project) and cache it on the runner itself -v /project/pkg/mod:~/go/mod/pkg
4. I locally tried to leave the container running and run go mod download outside and then copy the directory to the container and it didn't take time to install the dependencies

[osamamagdy]

Would this approach affect the running container by any means? like it would fail tests for some reason?

[mbrandenburger]

This looks like the right direction to me. 💪

In addition to the go deps, we may also think about caching other artifacts we consume (i.e., fabric binaries, docker images)

[osamamagdy]

@mbrandenburger Thanks. I am trying to activate the volume mount in the pipeline by assigning a value to DOCKER_DEV_RUN_OPTS before calling the make file like here I assumed this should work as the makefile is using the ?= operator in here so if the variable is set it should only append to it. but this doesn't seem to work as the make command here is not as expected

[osamamagdy]

@mbrandenburger It worked by adding a new variable to the make file and appending it to the DOCKER_DEV_RUN_OPTS. Is this acceptable? can you review PR #762 I will revert the changes to the workflow triggers before the changes are merged

[mbrandenburger]

@osamamagdy thanks! I will look into this asap!

[mbrandenburger]

I re-opened the issue to capture other options for caching.

[mbrandenburger]

Thanks @osamamagdy for improving CI caching with the go mod package caching. 🚀💪🎉