Question for docker image on mac M1 and golang version upgrade?

[SamYuan1990]

Hi team,

May I know do we have any plan to use buildx to provide docker build for Mac M1? as when I try to run docker build, some apt package is out of date.

#5 145.0 However the following packages replace it:
#5 145.0    git-svn
#5 145.0 
#5 145.0 E: Version '1:2.34.1-1ubuntu1.5' for 'git' was not found
------
executor failed running [/bin/sh -c apt-get update && apt-get install --no-install-recommends -y   ca-certificates=20211016 -V   gcc=4:11.2.0-1ubuntu1 -V   gcc-multilib=4:11.2.0-1ubuntu1 -V   git=1:2.34.1-1ubuntu1.5 -V   golang=2:1.18~0ubuntu2 -V   pkg-config=0.29.2-1ubuntu3 -V && apt-get clean && rm -rf /var/lib/apt/lists/*]: exit code: 100

and according to go version def, maybe we can update go version to 1.18 or 1.20? https://github.com/open-quantum-safe/liboqs-go/blob/main/go.mod#L3

[baentsch]

Thanks for your interest in our package(s). We'd be glad to receive your PR updating things as per your suggestion. For the docker x-build the easiest place is probably the CircleCI config file, but we'd also be glad to see GitHub workflow used for this.

[SamYuan1990]

Thanks for your interest in our package(s). We'd be glad to receive your PR updating things as per your suggestion. For the docker x-build the easiest place is probably the CircleCI config file, but we'd also be glad to see GitHub workflow used for this.

as I had never try with CricleCI before, following here https://circleci.com/developer/orbs/orb/circleci/docker, could you please help me figure out buildx guide for CricleCI?

[SamYuan1990]

https://github.com/CircleCI-Public/docker-orb/issues/134 from github, it seems the docker buildx feature for circleCI is still open and seems no arm support? https://github.com/CircleCI-Public/docker-orb/issues/160

[baentsch]

CircleCI-Public/docker-orb#134 from github, it seems the docker buildx feature for circleCI is still open and seems no arm support? CircleCI-Public/docker-orb#160

Doesn't look good, agreed. I have no experience running docker buildx on CCI, let alone targeting M1. This is how I used buildxonce locally: https://github.com/open-quantum-safe/profiling/blob/main/perf/README.md#building.

So step by step: 1) How to use buildx to build for M1 (on Linux, AMD64 or ARM64); 2) If that works, find out whether we can make this run on CCI or GitHub CI. If you agree with these steps(?), do you have experience with/could work towards resolving either?

[SamYuan1990]

CircleCI-Public/docker-orb#134 from github, it seems the docker buildx feature for circleCI is still open and seems no arm support? CircleCI-Public/docker-orb#160

Doesn't look good, agreed. I have no experience running docker buildx on CCI, let alone targeting M1. This is how I used buildxonce locally: https://github.com/open-quantum-safe/profiling/blob/main/perf/README.md#building.

So step by step: 1) How to use buildx to build for M1 (on Linux, AMD64 or ARM64); 2) If that works, find out whether we can make this run on CCI or GitHub CI. If you agree with these steps(?), do you have experience with/could work towards resolving either?

en... I am going to do following with a PR.

1. try to fix package depends issue in docker.
2. try to use github action with buildx at my own fork. all above are poc to see if buildx works. if CCI doesn't support buildx, I suppose we are open to use github action right?

[baentsch]

if CCI doesn't support buildx, I suppose we are open to use github action right?

Absolutely.

Thanks for giving this a try.

[SamYuan1990]

if CCI doesn't support buildx, I suppose we are open to use github action right?

Absolutely.

Thanks for giving this a try.

PR with POC is #20 , let's working on it together!

[brivu]

Hey @SamYuan1990 and @baentsch,

I'm Brian from CircleCI's Community Engineering team. I'm following up on an issue here with some comments below:

1. The user who posted the issue is stating arm64 builds don't work. In the issue, I noticed that they're using version circleci/docker@2.1.4 of the docker orb. Version 2.2.0 will install docker on arm64 machines properly.
2. I'm in the process of updating the docker orb to support docker buildx with multi-architecture builds. It should be available in the next couple of weeks.
3. In the meantime, my colleague has written a sample config that will support docker buildx on CircleCI. You can find it here.

I hope this information helps! Feel free to reach out if there's anything else you need.

Best, Brian

[SamYuan1990]

Hey @SamYuan1990 and @baentsch,

I'm Brian from CircleCI's Community Engineering team. I'm following up on an issue here with some comments below:

1. The user who posted the issue is stating arm64 builds don't work. In the issue, I noticed that they're using version circleci/docker@2.1.4 of the docker orb. Version 2.2.0 will install docker on arm64 machines properly.
2. I'm in the process of updating the docker orb to support docker buildx with multi-architecture builds. It should be available in the next couple of weeks.
3. In the meantime, my colleague has written a sample config that will support docker buildx on CircleCI. You can find it here.

I hope this information helps! Feel free to reach out if there's anything else you need.

Best, Brian

Hi Brain, does circleci plan to support s390? and how to enable it for my personal fork?

[baentsch]

does circleci plan to support s390? and how to enable it for my personal fork?

^^^ @brivu

[SamYuan1990]

does circleci plan to support s390? and how to enable it for my personal fork?

^^^ @brivu

As the docker file includes go test, if the buildx success with s390, which means it functional works on s390 isn't it?

https://github.com/open-quantum-safe/liboqs-go/blob/main/docker/Dockerfile#L44

[baentsch]

As the docker file includes go test, if the buildx success with s390, which means it functional works on s390 isn't it?

Yes, if the test is run OK on CircleCI the pointer to @brivu is not necessary and I apologize for notifying him.

However, so far, all I found is a run log on GitHub CI, nothing on CircleCI: https://github.com/SamYuan1990/liboqs-go/actions/runs/4333874827/jobs/7567365625.

In there, I'm also wondering why there is no output of the test execution itself: I'd expect lines showing "Correctness etc." for all algorithms, but can't find anything. Lastly, when building and running the Dockerfile locally (only for x86_64) no tests are logged as run during build but test output is shown only when executing the container.

I'm no "Gopher", so may be misunderstanding something here -- but some logfile-documented success would help me see that everything is good here: Can you please point to/post a link to such successful CI test run logs (on all platforms)?

Don't get me wrong: If this runs successful on GitHub CI, I'd be happy. If all currently supported platforms are running in GitHub CI (have you confirmed the x-built "linux/arm64" image run OK on OSX?) we could move CI away from CircleCI and its limitations.

[SamYuan1990]

1. Some test log during build, as there not enable go test -v hence... nearly zero test report there.

   • amd64
   • arm64
   • s390

2. Hence to save time for build, at open-quantum-safe level, I would suggest we have a ubuntu image for as a general base image for this org. https://github.com/SamYuan1990/liboqs-go/blob/main/docker/Dockerfile#L1-L25 Hence for example, we can update docker file for golang as below to save time and ...

   From liboqs:base
   Run apt-get install -y golang
   ....

   and the base image can further benefits with other release if we considered java/python/...

3. From CI parts... there one thing sounds need current maintainer's help, as token to access docker hub....

[baentsch]

1. Some test log during build, as there not enable go test -v hence... nearly zero test report there.

   • amd64

   • arm64

   • s390

This is exactly what I mean: It is not OK to have no logfiles/confirmation that the tests actually ran.

2. Hence to save time for build, at open-quantum-safe level, I would suggest we have a ubuntu image for as
   a general base image for this org. https://github.com/SamYuan1990/liboqs-go/blob/main/docker/Dockerfile#L1-L25
   Hence for example, we can update docker file for golang as below to save time and ...

Completely agree. That's why we have docker images for fast CI builds, e.g., https://hub.docker.com/r/openquantumsafe/ci-ubuntu-focal-x86_64. Please feel free to use those instead and/or do PR's to https://github.com/open-quantum-safe/ci-containers to amend them if you see a need.

From liboqs:base
Run apt-get install -y golang
....

and the base image can further benefits with other release if we considered java/python/...

If you are suggesting a cross-language docker image (?) wouldn't this become way too large incorporating all language runtimes? But if you'd like to give this a try, please share your experiences (as to build time and size). Maybe using "alpine" would result in a not too large image...

3. From CI parts... there one thing sounds need current maintainer's help, as token to access docker hub....

If I understand you right, (to push the images to docker hub?), please first automate everything to work with your own docker hub account (e.g., setting "samyuan1990/go" in https://github.com/SamYuan1990/liboqs-go/blob/4b37b1eae6cfbf6f7018a3fcd505f7e742973d29/.github/workflows/build.yml#L39). If this works, we can then automate to push to "openquantumsafe/go". I therefore suggested using a variable for the target docker repository: In your github context it would point to your docker hub account and in the "open-quantum-safe" github context it would point to the "openquantumsafe" docker hub account. Same things with docker hub credentials (to successfully push): You should use yours in your environment (to see everything works) and we can add openquantumsafe ones when everything then is built in the main branch of open-quantum-safe/liboqs-go...

[SamYuan1990]

@baentsch , For point 1, I will adjust in PR. For point 2, I would like suggest we publish some deb or rpm package and then... in CI it can just download the package and installation. here you are

• deb package for bcc on a fork. https://github.com/sustainable-computing-io/kepler-ci-artifacts (due to bcc reason we have to do a fork....)
• https://github.com/sustainable-computing-io/kepler-action CI which consume the package. to save time.

[baentsch]

I'm not sure I understand what bcc does or why a fork would be needed. It'll probably be best if you implement your suggestion in #20 so I may understand it.

Finally, what about this suggested addition for the github workflow:

[current workflow job here, then adding this job:]
  deploy:
     runs-on: ubuntu-latest
     needs: build-image
     if: github.ref == 'refs/heads/main'
     steps:
      - name: Login to docker hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_PASSWORD }}
[remaining (same) work flow as in current PR -- just with push: true]        

--> That allows you to build and test the image during PR and an actual push is then only done on the main branch (when merging). All we'd do is add the github secrets used in docker login. OK for you?

[SamYuan1990]

I will try to have a final change in #20 as I just display test case in build logs and fixed some go routine leaks. after that, I will use #20 for my personal forks, and we can adjust PR to adopt with this repo.

[SamYuan1990]

updated in PR #20

[vsoftco]

Decided to drop support for this, as it introduce additional dependencies that are difficult to maintain in the long term.