search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
16.98k stars 1.71k forks source link

Consider compiling for ARM architecture #921

Closed ekristen closed 7 years ago

ekristen commented 7 years ago

I've got a lot of ARM devices that would benefit from being able to use teleport to include mobile devices such as iPhones and Androids.

Would you consider adding ARM architectures to your build process?

I've attempted to cross compile the teleport binary for ARM but have run into a few problems with the fact that C is imported in a single location that causes compilation problems.

russjones commented 7 years ago

Can you paste the errors you are running into?

ekristen commented 7 years ago

First I had to upgrade the boltdb vendor dependency, after doing that this is the error I get. 

gox -osarch="darwin/arm64" -ldflags '-w -s' ./tool/teleport                                                                                                                                           ⏎
Number of parallel builds: 7

-->    darwin/arm64: github.com/gravitational/teleport/tool/teleport

1 errors occurred:
--> darwin/arm64 error: exit status 2
Stderr: # github.com/gravitational/teleport/lib/client
lib/client/api.go:1282: undefined: GetLoginShell
# github.com/gravitational/teleport/lib/srv
lib/srv/exec.go:125: undefined: "github.com/gravitational/teleport/lib/utils".GetLoginShell
lib/srv/exec.go:176: undefined: "github.com/gravitational/teleport/lib/utils".GetLoginShell

It seems to be because there is an import C in the go file, so I modified the compile command to enable CGO but that spits out an entirely different set of errors ...

CGO_ENABLED=1 gox -osarch="darwin/arm64" -ldflags '-w -s' ./tool/teleport                                                                                                                             ⏎
Number of parallel builds: 7

-->    darwin/arm64: github.com/gravitational/teleport/tool/teleport

1 errors occurred:
--> darwin/arm64 error: exit status 2
Stderr: # runtime/cgo
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:27:16: error: brackets expression not supported on this target
 stp x19, x20, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:28:16: error: brackets expression not supported on this target
 stp x21, x22, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:29:16: error: brackets expression not supported on this target
 stp x23, x24, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:30:16: error: brackets expression not supported on this target
 stp x25, x26, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:31:16: error: brackets expression not supported on this target
 stp x27, x28, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:32:16: error: brackets expression not supported on this target
 stp x29, x30, [sp, #-16]!
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:33:2: error: unknown use of instruction mnemonic without a size suffix
 mov x29, sp
 ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:35:2: error: unknown use of instruction mnemonic without a size suffix
 mov x19, x0
 ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:36:2: error: unknown use of instruction mnemonic without a size suffix
 mov x20, x1
 ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:37:2: error: unknown use of instruction mnemonic without a size suffix
 mov x0, x2
 ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:39:2: error: invalid instruction mnemonic 'blr'
 blr x20
 ^~~
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:40:2: error: invalid instruction mnemonic 'blr'
 blr x19
 ^~~
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:42:16: error: brackets expression not supported on this target
 ldp x29, x30, [sp], #16
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:43:16: error: brackets expression not supported on this target
 ldp x27, x28, [sp], #16
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:44:16: error: brackets expression not supported on this target
 ldp x25, x26, [sp], #16
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:45:16: error: brackets expression not supported on this target
 ldp x23, x24, [sp], #16
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:46:16: error: brackets expression not supported on this target
 ldp x21, x22, [sp], #16
               ^
/usr/local/Cellar/go/1.8/libexec/src/runtime/cgo/gcc_arm64.S:47:16: error: brackets expression not supported on this target
 ldp x19, x20, [sp], #16
kontsevoy commented 7 years ago

@ekristen currently some teleport dependencies cannot be compiled for ARM64. have you tried 32-bit arm?

ekristen commented 7 years ago

That gets me a little closer, I can try and dig in some more, but this is the error now 

Number of parallel builds: 7

-->      darwin/arm: github.com/gravitational/teleport/tool/teleport

1 errors occurred:
--> darwin/arm error: exit status 2
Stderr: # runtime/cgo
clang: error: argument unused during compilation: '-mno-thumb' [-Werror,-Wunused-command-line-argument]
kontsevoy commented 7 years ago

darmin/arm? are you cross-compiling? if you are, we've never tried that. for ARM builds we've used Raspberry PIs and then moved to renting Linux ARM servers on https://www.scaleway.com

Here's the latest master build for you to try: http://s3.gravitational.io/ev/teleport-v2.1.0-alpha.2-linux-arm-bin.tar.gz

md5sum: 4cc50b8a459326f123c72daa9b678e71 sha256: 9d6772f397471b24484a3713aef5a6cf5003c35c2767e1a9580cd3713535a2f1

ekristen commented 7 years ago

I am attempting to cross compile using gox

Thanks for the link, this will let me PoC!

ekristen commented 7 years ago

FWIW, my target platforms are iOS ARM and Android ARM.

stephens2424 commented 7 years ago

On linux/amd64, I was able to do it with:

stephen@sdeb:~/go/src/github.com/gravitational/teleport$ GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc make

https://askubuntu.com/questions/250696/cross-compile-for-arm was my reference for how to get the cgo part working.

kontsevoy commented 7 years ago

@ekristen did you figure this out? sorry, we've been super busy with other stuff and haven't gotten to iOS ARM (and very unlikely to look into this any time soon).

ekristen commented 7 years ago

Currently I'm stuck on the following error, if anyone has experience with cross compiling, clang, etc and can offer hints or suggestions as to how to fix this I would greatly appreciate it.

Stderr: # runtime/cgo
clang: error: argument unused during compilation: '-mno-thumb' [-Werror,-Wunused-command-line-argument]

I cannot seem to figure out how to get it to ignore unused arguments unfortunately just to see if I can get it passed this point.

kontsevoy commented 7 years ago

@ekristen we do not cross-compile Teleport. Our ARM builds are tested on Raspberry Pi 3 and on scaleway ARM servers. Get the latest Golang 1.8.3, get the repo and simply run make.

kontsevoy commented 7 years ago

clarification: cross-compiling CGO targets is hard, because you get into managing gcc toolchain.

ekristen commented 7 years ago

I've made more progress.

export CGO_CFLAGS="-Wno-unused-command-line-argument"

gox -osarch="darwin/arm" -cgo -verbose ./tool/teleport
Number of parallel builds: 7

-->      darwin/arm: github.com/gravitational/teleport/tool/teleport

1 errors occurred:
--> darwin/arm error: exit status 2
Stderr: # runtime/cgo
In file included from $WORK/runtime/cgo/_obj/_cgo_export.c:3:
cgo-gcc-export-header-prolog:25:55: error: '_check_for_32_bit_pointer_matching_GoInt' declared as an array with a negative size
russjones commented 7 years ago

@ekristen Just a heads up, with Teleport 2.2 we've announced we will be releasing ARM binaries with our releases: http://gravitational.com/blog/teleport-release-2-2/

You can download the ARM binary for Teleport 2.2 from here: https://github.com/gravitational/teleport/releases/tag/v2.2.0

ekristen commented 7 years ago

+1 glad to see that, that'll be nice for a lot of different platforms. Thanks for all the work you've been doing.

Unfortunately AFAIK not all ARMs are the same, and I really need it for android and ios devices, hence my attempt to cross compile for darwin/arm(32).

kontsevoy commented 7 years ago

@ekristen AFAIK Golang only supports ARMv6 [1] so regardless of the compilation method, you're getting the same binary unless you tell me what I do not know. ;) Please do.

[1] https://golang.org/dl/

andrewbanchich commented 5 years ago

For anyone else stuck on this, I was running into the same issue when cross compiling Teleport using GitLab CI, which uses Docker. I got it working by using the golang:latest Docker image, and running 

apt-get update
apt-get install -y zip gcc-arm-linux-gnueabihf
mkdir -p $GOPATH/src/github.com/gravitational
cd $GOPATH/src/github.com/gravitational
git clone https://github.com/gravitational/teleport.git
cd teleport
make full GOARCH=arm CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc VERSION=2.7.4
cc14514 commented 5 years ago

I have also encountered this problem (about "-mno-thumb"). The solution is disable cgo , for example follows: export CGO_ENABLE=0; GOARCH=arm GOOS=linux go build -o foo.

I hope this will help you.