Support ARM64 for gvisor

[kevinzs2048]

Currently just support amd64, could we support for ARM64? What do we need to do for this support? I can help to work on it.

[prattmic]

We don't have any immediate plans to port to additional architectures. It is certainly feasible (as shown below), but certainly requires a lot of work and is something we want to do very carefully to avoid adding unnecessary complexity and technical debt.

Porting to a new architecture requires several steps:

1. Porting/creating a platform compatible with the arch. The ptrace platform would be fairly simple to port. The kvm platform would be much more complex.
2. Porting AMD64-specific assembly/host syscalls. There are several AMD64 assembly files (mostly with names _amd64.s) that obviously need ports. There may also be places where we make direct syscalls with syscall.Syscall where the syscall semantics are slightly different on a new arch.
3. Adding a new syscall table.
4. Adding support for the new arch throughout sentry internal system call, signal, etc handling. A lot of this is in the arch package, though that package is in need of refactoring. There are also several kernel structures that differ between arches (struct pt_regs is an obvious example). Many of these are in abi/linux, but others we still use directly from the syscall package (they should be moved to abi/linux). This is the hardest part, as there are still many unanswered questions around how to do all of this cleanly.

[kevinzs2048]

@prattmic Really Thanks for clarifying. In that case I think it is not a very easy work, I will talk with my workmate inside arm to discuss the development work.

[lubinsz]

@prattmic We have enabled ptrace platform on Arm64 platform. Later, we will deliver the patches. Please see following as reference:

root@entos1:/go/src/github.com/google/gvisor# uname -p aarch64 root@entos1:/go/src/github.com/google/gvisor# docker run --runtime=runsc hello-world W1010 17:46:16.510561 30359 x:0] Could not parse /proc/cpuinfo, it is empty or does not contain cpu MHz W1010 17:46:16.537007 30370 x:0] Could not parse /proc/cpuinfo, it is empty or does not contain cpu MHz W1010 17:46:16.556919 30370 x:0] Could not parse /proc/cpuinfo, it is empty or does not contain cpu MHz

Hello from Docker! This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (arm64v8)
3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/

For more examples and ideas, visit: https://docs.docker.com/engine/userguide/

[prattmic]

Wow, that's great! I didn't realize anyone was working on this. I look forward to seeing the changes.

[amscanne]

Amazing! Is there a public branch somewhere on github? We'd love to help the patches land, and it might be useful to have some early high-level guidance.

[everflux]

Any chance to try this out already?

[ianlewis]

Other related PRs: #571 #653 #696 #753 #757

[wtfismyip]

I'm willing to test this out if/when its ready.

[majek]

Ok, so I don't really care much about KVM support on arm64 at this point, but I would love to be able to test ptrace on arm64. I would appreciate if we could bring gvisor runsc cross-compile to the stage that it produces a binary, which I can run, and then complain about :)

[ianlewis]

I kind of think we can close this issue in lieu of using the ARM64 milestone. Issues regarding support for ARM64 should be tracked there. https://github.com/google/gvisor/milestone/2

[lubinszARM]

Arm64 kvm: run ffmpeg on Ampere Altra Server (arm64 neoverse-n1): https://github.com/google/gvisor/issues/4056

[magnate3]

@lubinszARM
hello, could you tell me how to make build in arm64 root@cloud:~/gvisor# make -j $(nproc) the error is : docker-ce-cli-20.10.1-3.fc31.aarch64.rpm 5.6 MB/s | 30 MB 00:05

Total 5.6 MB/s | 30 MB 00:05
warning: /var/cache/dnf/docker-ce-stable-5216070ebe39d4d5/packages/docker-ce-cli-20.10.1-3.fc31.aarch64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY Docker CE Stable - aarch64 0.0 B/s | 0 B 00:00
Curl error (35): SSL connect error for https://download.docker.com/linux/fedora/gpg [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to download.docker.com:443 ] The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. The command '/bin/sh -c dnf install -y docker-ce-cli' returned a non-zero code: 1 --- BUILD -c opt //runsc Error: No such container: gvisor-bazel-3328c4e9-aarch64

the problem mybe fail to install google-cloud-sdk

[magnate3]

@lubinszARM could you tell me how to make build gvisor in arm64 root@cloud:~/gvisor# make -j $(nproc) warning: /var/cache/dnf/docker-ce-stable-5216070ebe39d4d5/packages/docker-ce-cli-20.10.1-3.fc31.aarch64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY Docker CE Stable - aarch64 0.0 B/s | 0 B 00:00 Curl error (35): SSL connect error for https://download.docker.com/linux/fedora/gpg [OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to download.docker.com:443 ] The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. The command '/bin/sh -c dnf install -y docker-ce-cli' returned a non-zero code: 1 --- BUILD -c opt //runsc Error: No such container: gvisor-bazel-3328c4e9-aarch64

Environment

root@cloud:/gvisor# uname -a Linux cloud 5.5.19-050519-generic #202004210831 SMP Tue Apr 21 08:49:56 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux root@cloud:/gvisor#

the problem mybe fail to install google-cloud-sdk

root@cloud:~/gvisor# uname -a Linux cloud 5.5.19-050519-generic #202004210831 SMP Tue Apr 21 08:49:56 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux root@cloud:~/gvisor#

[lubinszARM]

Hi @magnate3 If you use the root user, you should use the bazel command to build runsc. Such as: bazel build runsc If you want to use make command to build or test, there are 3 things you should notice. 1, We should login as normal user. Notice: please do not log in as normal users in the root account 2, Enabling Non-root Users to Run Docker Commands https://www.google.com.hk/search? newwindow=1&safe=strict&rlz=1C5GCEA_enCN927CN927&sxsrf=ALeKk02XcV_twmwUv_RoQkZfxzIeNwkD4w%3A1609147764289&ei=dKXpX-2REfGGr7wPtYaiiAo&q=Enabling+Non-root+Users+to+Run+Docker+Commandsdocs&oq=Enabling+Non-root+Users+to+Run+Docker+Commandsdocs&gs_lcp=CgZwc3ktYWIQAzIHCCEQChCgAToECAAQR1Dz7hBY8-4QYJ7xEGgAcAV4AIABmAGIAZgBkgEDMC4xmAEAoAECoAEBqgEHZ3dzLXdpesgBCMABAQ&sclient=psy-ab&ved=0ahUKEwjtm4bHrvDtAhVxw4sBHTWDCKEQ4dUDCA0&uact=5 3, host Linux kernel version >= 5.5+

[lubinszARM]

[magnate3]

@lubinszARM , thank you . because of Great Firewall ,other problem happens and do you have other way to build gvisor ？ such as using go , not bazel. nalyzing: target //runsc:runsc (47 packages loaded, 6941 targets configured) ERROR: An error occurred during the fetch of repository 'com_github_google_subcommands': Traceback (most recent call last):

[lubinszARM]

@magnate3 Sorry. I have no idea about it.