Port to ARMv8 (aarch64)

[vielmetti]

I'm looking to port this to ARMv8 (aarch64) running Docker 1.12.1 on Ubuntu 16.04 on an ARMv8 machine.

At the moment, the sticking point is the lack of an aarch64/alpine image. I'm thinking about getting it to run on aarch64/ubuntu, which should be straightforward to do, since openvpn runs native on that system just fine; just a matter of doing "apt-get" instead of "apk" for package management in the Dockerfile. The image will be way heavier, but it should run.

If anyone has done this already let me know. I'm told that aarch64/alpine is coming with Alpine 3.5 which is due some time near the end of the year.

[vielmetti]

Looks like if I go back to Debian I can go back to this pull https://github.com/kylemanna/docker-openvpn/pull/70 and base off of aarch64/debian with some confidence.

[kylemanna]

My biggest concern with this is the size of the debian image, roughly 10x and it's all cruft. What are the problems with alpine? I've never run Docker on an arm machine.

[vielmetti]

The problem with alpine on armv8 (aarch64) is that there is no current build of it. It's due to be supported in alpine 3.5, as I understand, but alpine:edge is not currently working.

This issue https://github.com/tianon/jenkins-groovy/issues/15 appears to be close to where the problem is, and http://git.alpinelinux.org/cgit/aports/log/?qt=grep&q=apk-tools is the change log. From that I suspect that the port to libressl is not complete.

[vielmetti]

I'm chasing this at https://bugs.alpinelinux.org/issues/6372 and am uncertain as to the root cause (it might be libressl, it might be an unapplied patch for aarch64 support).

[kylemanna]

@vielmetti Glad to hear you're fighting upstream, please keep us posted! Would love to have an armv8 compatible container on Alpine Linux when it gets there.

[vielmetti]

Thanks @kylemanna . I haven't been able to fully understand the Alpine build process, but it looks like they are sorting out a host of issues associated with dropping openssl and there's no shortage of commits to the project. Still haven't heard back from anyone directly just yet.

[vielmetti]

OK, progress.

https://bugs.alpinelinux.org/issues/6389 is the current issue, with Alpine's apk package manager returning bad signatures on some archive repositories.

A previous issue https://bugs.alpinelinux.org/issues/6372 which had core dumped the container on "docker build" was traced down to a linker issue in building static binaries on ARMv8.

https://github.com/gliderlabs/docker-alpine/issues/190 has the blow-by-blow as @owlab-exp and I sort out build issues.

[vielmetti]

apk's signature check can be bypassed with --allow-untrusted, so I have something built with that, not for production use.

[vielmetti]

The upstream problem with bad signatures at https://bugs.alpinelinux.org/issues/6389 has been resolved, so it's time to tackle this again.

[vielmetti]

More progress - I think I have this running successfully, after the various Alpine issues were resolved. I think the best thing to contribute is the new Dockerfile, which is identical to the standard one except for a different FROM line that reads

FROM aarch64/alpine:edge

[kylemanna]

Cool! Thanks for all the hard work battling upstream issues!

This leads me to the next question: What's the best way to support this with minimal overhead? A separate git branch with a Docker tag? Or a separate Dockerfile and corresponding Docker tag?

On other projects I've accomplished similar things: https://hub.docker.com/r/kylemanna/aosp/tags/

My gut reaction is that a separate branch will make merging of the supporting shell script and utilities hard. Meanwhile, a separate Dockerfile would keep everything on the same branch and perhaps simpler. The task of maintaining multiple Dockerfiles that are 90% the same seems like a more manageable task then maintaining multiple branches that are 99.99% the same.

Pending other feedback, a pull request (reference this issue in it) for a aarch64/Dockerfile addition makes sense to me.

Comments, thoughts concerns?

Also, what aarch64 board are you using? I may have to get my hands on one! :)

[kylemanna]

Also, whatever proposed solution makes sense, we'll need some updated documentation. If it's short and sweet, perhaps in the README.md, otherwise a more verbose addition to the docs directory.

[rca]

Perhaps I can assist a bit here. The bad news is I don't think Docker's automated builds support this just yet, but building locally for ARM works perfectly. All the heavy lifting here has been done by others in the community; I'm just pitching in this issue update. ;)

The team at Resin has this great base image that allows you to cross-build an ARM image right on your Mac: resin/raspberrypi-alpine:latest. I made this the FROM line and my Mac built a usable Raspberry Pi image.

The second part of the magic would come in with Docker manifests, best explained at this blog post.

Using the manifest-tool utility referenced in the post above, you can feed it a yaml file like the one below to essentially create an openvpn meta-image that points to the right image based on the requesting machine's architecture.

The plumbing for all this upstream is there, but the UI is lacking, hence the 3rd party tool. These images can be hosted on Docker Hub, but not automatically built yet.

I've used this in production to deploy amd64 and arm images and it works quite smooth.

image: kylemanna/openvpn:{{ version }}
manifests:
  -
    image: kylemanna/amd64-openvpn:{{ version }}
    platform:
      architecture: amd64
      os: linux
  -
    image: kylemanna:arm-openvpn:{{ version }}
    platform:
      architecture: arm
      os: linux

[rca]

Updated above with the link to manifest-tool

[rca]

I pushed up a working manifest.yml on my fork with a proof of concept. The image hosted at roberto/openvpn works for both amd64 and arm!

berto@mac:~ $ docker run -t -i --rm --name test roberto/openvpn uname -a
Linux 7a19cc8a9157 4.9.4-moby #1 SMP Wed Jan 18 17:04:43 UTC 2017 x86_64 Linux

berto@mac$ ssh pi@raspberrypi
Last login: Thu Jan 26 13:22:58 2017 from b8-f6-b1-1b-a3-47

pi@raspberrypi:~ $ sudo docker run -t -i --rm --name test roberto/openvpn uname -a
Unable to find image 'roberto/openvpn:latest' locally
latest: Pulling from roberto/openvpn
Digest: sha256:7d789af00f8077cde8852b586d54755614b695f37da52dd9aa058f9e1635ba21
Status: Downloaded newer image for roberto/openvpn:latest
starting version 3.1.5
specified group 'input' unknown
Linux 466a165fa300 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l Linux

[vielmetti]

On my aarch64 machine (ARMv8) I get this when I run the image from @rca

root@docker-build-test:~# sudo docker run -t -i --rm --name test roberto/openvpn uname -a
Unable to find image 'roberto/openvpn:latest' locally
latest: Pulling from roberto/openvpn
docker: no supported platform found in manifest list.
See 'docker run --help'.

[rca]

I have confirmed this image works on a Raspberry Pi 3, which is an ARMv7 processor. After the fact, I focused on the fact that this issue was opened for ARMv8 compatibility.

I'm not familiar enough with ARM's architecture to know if compiling for ARMv8 would make the image backwards-compatible with ARMv7, however, a quick glance at the manifest docs shows there is a variant field:

The optional variant field specifies a variant of the CPU, for example armv6l to specify a particular CPU variant of the ARM CPU.

@vielmetti can you point me to an ARMv8 docker image; I'll look into incorporating it into my manifest file and see if we can get a unified Docker image. :)

[vielmetti]

Thanks @rca - I'm in the midst of getting jenkins running on aarch64 and when I have that nailed down I'll look to make a CI step that tracks this repo and does automated builds.

I'm still looking for an OpenSSL or equivalent for aarch64 that uses the AES instructions on chip to make things go faster, my first set of benchmark tests were really not performant enough.

[vielmetti]

Building locally for aarch64 works just fine; I am typing through it now. All I had to do in the Dockerfile was make it

FROM aarch64/alpine:3.5

and everything went according to plan. The issue now is performance; I am seeing less than stunning throughput on my fast.com tests, perhaps because the ARMv8 machine is not using the AES instructions that are in the hardware and is instead doing all of its crypto unaccelerated.

[vielmetti]

Pull request added, it's a very simple one liner. The eventual goal is to remove the need for in entirely when Alpine goes to multiarch support in Docker.

[kylemanna]

@vielmetti What aarch64 machine are you running this on?

[vielmetti]

It's up and running on the Packet 2A 96-core host at http://packet.net under Ubuntu 16.04 and Docker 1.12.3. They just released CentOS for that box so I'm going to reconfirm the build there too (not that it should make a difference).

[woiza]

@vielmetti Isn't openssl necessary for accelerating AES?

[NinoSkopac]

Doesn't work for me.

When I run docker pull kylemanna/openvpn-aarch64 I get Error response from daemon: manifest for kylemanna/openvpn-aarch64:latest not found: manifest unknown: manifest unknown