Unfortunately circom compiler does not currently upload binaries to crates.io and instead recommends cloning the repo (with target tag) and compiling circom.
Currently by default all ci jobs that rely on circom re-compile it from scratch. 'unit-test-circuits' does this inline and also uses the default branch rather than v2.1.2 used by e2e test, other jobs build the ghcr.io/eyblockchain/local-circom image. That also applies to developers building nf3 locally.
By building a multi-architecture (arm64, amd64) image based on the target circom version and pushing it to GitHub container registry (ghcr.io) both ci jobs and local runs on developer machines can avoid compiling circom.
I tested this on my nf3 fork with a script I ran manually from my machine to build and publish (I use podman but here's the docker buildx variant you could use):
#!/usr/bin/env bash
set -e
# Set your manifest name
MANIFEST_NAME="local-circom-manifest"
# Set the required variables
REGISTRY="ghcr.io"
USER="eyblockchain"
IMAGE_NAME="local-circom"
CIRCOM_VERSION="v2.1.2"
# Make sure to login via docker CLI using your GitHub PAT with 'packages:write' permissions
# echo $CR_PAT | docker login ghcr.io -u $USER --password-stdin
# multi-platform build requires using 'docker-container' buildx driver
docker buildx create --use
# Build both amd64 and arm64 architecure containers and push all tags
docker buildx build \
--no-cache \
--platform linux/amd64/v8,linux/arm64 \
--tag "${REGISTRY}/${USER}/${IMAGE_NAME}:${CIRCOM_VERSION}" \
--tag "${REGISTRY}/${USER}/${IMAGE_NAME}:latest" \
--push - < docker/circom.Dockerfile
I used it to avoid the full circom compilation succesfully on both github ubuntu ci runners (amd64) and my macbook pro with apple silicon (arm64).
To use in ci
Define the following env var at the workflow scope to skip 'local-circom' image build:
SKIP_CIRCOM_BUILD: 1 (example)
Define the following permissions at the workflow scope to allow pulling container images from 'ghcr.io':
You need a GitHub PAT with 'packages:read' permission for the repo (you may declare the package visibility as internal to the org or public) and run echo $CR_PAT | docker login ghcr.io -u $USER --password-stdin (USER is github handle)
unit-test-circuits
Compiling circom can be avoided by defining the job as follows:
unit-test-circuits:
runs-on: ubuntu-20.04
container:
image: ghcr.io/eyblockchain/local-circom
credentials:
username: ${{ github.actor }}
password: ${{ secrets.github_token }}
options: --cpus 2
timeout-minutes: 30
steps:
- name: Add 'CIRCOM_HOME' to PATH
shell: bash
run: |
echo "$CIRCOM_HOME" >> $GITHUB_PATH
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '16.17.0'
- name: Unit Circuit Tests
run: |
npm ci
cd common-files
npm ci
cd ../nightfall-deployer
npm ci
cd ../
npm run unit-test-circuits
So it reuses the 'local-circom' container image as a job runner image
Motivation
This greatly reduces container image build time both on dev machines and in ci.
The process of building a multi-architecture image and publising to 'ghcr.io' can instead run in ci by using docker/build-push-action as described here but that might be an overkill since circom updates (I assume) are not that frequent. Anyway such ci job can also be triggered manually with a parameter that defiens the target circom version.
It is not mandatory to also build for 'arm64' if you do not wish to engage in a multi-architecture build. That suffices for ci.
I can submit a PR but it will only be applicable if you build and publish the 'local-circom' image...
Unfortunately circom compiler does not currently upload binaries to crates.io and instead recommends cloning the repo (with target tag) and compiling circom.
Currently by default all ci jobs that rely on circom re-compile it from scratch. 'unit-test-circuits' does this inline and also uses the default branch rather than v2.1.2 used by e2e test, other jobs build the
ghcr.io/eyblockchain/local-circom
image. That also applies to developers building nf3 locally.By building a multi-architecture (arm64, amd64) image based on the target circom version and pushing it to GitHub container registry (ghcr.io) both ci jobs and local runs on developer machines can avoid compiling circom.
I tested this on my nf3 fork with a script I ran manually from my machine to build and publish (I use podman but here's the docker buildx variant you could use):
I used it to avoid the full circom compilation succesfully on both github ubuntu ci runners (amd64) and my macbook pro with apple silicon (arm64).
To use in ci
SKIP_CIRCOM_BUILD: 1
(example)example
example
*To use locally
echo $CR_PAT | docker login ghcr.io -u $USER --password-stdin
(USER is github handle)unit-test-circuits Compiling circom can be avoided by defining the job as follows:
So it reuses the 'local-circom' container image as a job runner image
Motivation This greatly reduces container image build time both on dev machines and in ci.
circuits-test - build images step (circom build) - 5m 46s circuits-test - build images step (skips circom build) - 3m 57s
adversary-test - build images step (circom build) - 7m 36s adversary-test - build images step (skips circom build) - 3m 49s
ping-pong-test - build images step (circom build) - 6m 11s ping-pong-test - build images step (skips circom build) - 3m 12s
Notes
docker/build-push-action
as described here but that might be an overkill since circom updates (I assume) are not that frequent. Anyway such ci job can also be triggered manually with a parameter that defiens the target circom version.