search

Shippable / support

Shippable SaaS customers can report issues and feature requests in this repository
100 stars 28 forks source link

arm64 build capability #5151

Open rakeshlacework opened 3 years ago

rakeshlacework commented 3 years ago

Description of your issue:

We need to build (or cross compile) aarch64 target build. Tried a dumb step using drydockaarch64/u16all image with a x86_64 node pool.

a-murphy commented 3 years ago

Since aarch64 is a different architecture, you'll need a node with that architecture for the Arm Docker images. There is a shared node pool for Arm (documentation), but it is only for public projects and the nodes are shared by all the subscriptions that use it. For a private repository, you'll have to add your own Arm node to be able to run on an Arm node.

mzeier commented 3 years ago

@a-murphy having a problem with a BYON ARM. The installer fails with golang errors. Have tried with the default 1.6 golang and a PPA-provided 1.13 version.

Have launched an Ubuntu 16.04 AWS instance using this AMI:

ubuntu/images-testing/hvm-ssd/ubuntu-xenial-daily-arm64-server-20191010 (ami-00707945601d79710)

With go 1.6, the downloaded cluster_node_init shell script runs and bombs out with:

/home/shippable/docker-ce/components/cli ~
Building statically linked build/docker-linux-arm64
/home/ubuntu/go/src/github.com/docker/cli/vendor/golang.org/x/sys/unix/affinity_linux.go:10:2: cannot find package "math/bits" in any of:
    /home/ubuntu/go/src/github.com/docker/cli/vendor/math/bits (vendor tree)
    /usr/src/go/src/math/bits (from $GOROOT)
    /home/ubuntu/go/src/math/bits (from $GOPATH)
Makefile:24: recipe for target 'binary' failed
make: *** [binary] Error 1

Google suggests this is a golang versioning issue. The default golang on 16.04 is 1.6.

On a fresh ARM64 16.04 AMI, I started off with https://github.com/golang/go/wiki/Ubuntu and installed a 1.13 version of golang and tested to make sure a sample program worked:

root@shippable-arm-03:~# cat bits.go
package main

import (
    "fmt"
    "math/bits"
)

func main() {
    fmt.Println(bits.UintSize)
}
root@shippable-arm-03:~# go run bits.go
64

The installer fails differently:

/home/shippable/docker-ce/components/cli ~
package github.com/docker/cli/cli
    imports math/bits: unrecognized import path "math/bits" (import path does not begin with hostname)
Node init script completed
__SH__SCRIPT_END_FAILURE__

Unclear how to recover. Confirmed that I only have one version of go installed.

root@shippable-arm-03:~# dpkg -l | grep golang
ii  golang                           2:1.13~1longsleep1+xenial                       all          Go programming language compiler - metapackage
ii  golang-1.13                      1.13.4-1longsleep1+xenial                       all          Go programming language compiler - metapackage
ii  golang-1.13-doc                  1.13.4-1longsleep1+xenial                       all          Go programming language - documentation
ii  golang-1.13-go                   1.13.4-1longsleep1+xenial                       arm64        Go programming language compiler, linker, compiled stdlib
ii  golang-1.13-src                  1.13.4-1longsleep1+xenial                       arm64        Go programming language - source files
ii  golang-doc                       2:1.13~1longsleep1+xenial                       all          Go programming language - documentation
ii  golang-go                        2:1.13~1longsleep1+xenial                       arm64        Go programming language compiler, linker, compiled stdlib
ii  golang-src                       2:1.13~1longsleep1+xenial                       arm64        Go programming language - source files
a-murphy commented 3 years ago

Are any other Docker versions listed when adding a node in your node pool? The Docker version dropdown should be right above then name when adding a new node to a node pool. Only the Docker 17.06 script on Arm attempts to build Docker. If you are able to select one of the other versions, that may work better.

Otherwise, I would suggest confirming that which go finds the version you intended and that another version hadn't been installed using another method.

rakeshlacework commented 3 years ago

This is the docker version -

root@shippable-arm-03:~/go/src# docker version Client: Version: 18.09.7 API version: 1.39 Go version: go1.10.4 Git commit: 2d0083d Built: Wed Oct 14 19:44:19 2020 OS/Arch: linux/arm64 Experimental: false

Server: Engine: Version: 18.09.7 API version: 1.39 (minimum version 1.12) Go version: go1.10.4 Git commit: 2d0083d Built: Wed Oct 14 17:25:58 2020 OS/Arch: linux/arm64 Experimental: false

and this is the go version -

root@shippable-arm-03:~# go version go version go1.13.4 linux/arm64

this is the env variables -

root@shippable-arm-03:~# env GOHOSTARCH=arm64 SHELL=/bin/bash TERM=xterm-256color USER=root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36: MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin PWD=/root GOARCH=arm64 LANG=en_US.UTF-8 SHLVL=1 HOME=/root GOOS=linux LOGNAME=root XDG_DATADIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop GOHOSTOS=linux GOPATH=/go LESSOPEN=| /usr/bin/lesspipe %s LESSCLOSE=/usr/bin/lesspipe %s %s =/usr/bin/env OLDPWD=/root/go/src

On Tue, Oct 20, 2020 at 10:01 PM a-murphy notifications@github.com wrote:

Are any other Docker versions listed when adding a node in your node pool? The Docker version dropdown should be right above then name when adding a new node to a node pool. Only the Docker 17.06 script on Arm attempts to build Docker. If you are able to select one of the other versions, that may work better.

Otherwise, I would suggest confirming that which go finds the version you intended and that another version hadn't been installed using another method.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Shippable/support/issues/5151#issuecomment-713304395, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJAWXSY4YBVSL2VO56FLULSLZTMNANCNFSM4RZFXLUA .

mzeier commented 3 years ago

There is only one go installed:

root@shippable-arm-03:~# which go
/usr/bin/go
root@shippable-arm-03:~# go version
go version go1.13.4 linux/arm64
rakeshlacework commented 3 years ago

@a-murphy https://github.com/a-murphy we were able to get the node init script run and can see node is functional but shippable dashboard is still reporting node status as Waiting

On Wed, Oct 21, 2020 at 2:24 AM matthew zeier notifications@github.com wrote:

There is only one go installed:

root@shippable-arm-03:~# which go /usr/bin/go root@shippable-arm-03:~# go version go version go1.13.4 linux/arm64

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Shippable/support/issues/5151#issuecomment-713435758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJAWXTLCZVEHHX6KOI2SA3SL2SFPANCNFSM4RZFXLUA .

a-murphy commented 3 years ago

If the status isn't updating, the node probably isn't able to connect to something. My first guess would be that something is preventing connections to RabbitMQ, which is how the nodes get messages with the jobs they are to process. Could you check the shippable-reqKick service logs with sudo journalctl -n 100 -u shippable-reqKick@$BASE_UUID.service? Unfortunately, $BASE_UUID varies with each node, so I'll have to suggest pressing tab after "shippable-reqKick" to find the correct name. And the reqProc-$BASE_UUID Docker container logs?

mzeier commented 3 years ago

From the container's logs:

2020-10-21T16:34:13.843Z - verbose: Since an error occurred, re-connecting reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:13.843Z - warn: Failed to close connection from reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:13.843Z - verbose: Waiting for 8 seconds before re-connecting reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:21.848Z - verbose: Initializing reqProc
2020-10-21T16:34:21.849Z - verbose: Checking health of reqProc|_common|checkHealth
2020-10-21T16:34:21.849Z - verbose: Checking health of reqProc|micro|_healthCheck|amqpMonitor
2020-10-21T16:34:22.296Z - verbose: Connected from reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:22.296Z - verbose: Closed connection from reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:22.296Z - verbose: reqProc|micro|_healthCheck|amqpMonitor Successful health checks
2020-10-21T16:34:22.296Z - verbose: Checking health of reqProc|micro|_healthCheck|checkShippableApi
2020-10-21T16:34:22.296Z - verbose: Initializing ShippableAdapter
2020-10-21T16:34:22.493Z - verbose: reqProc|micro|_healthCheck|checkShippableApi Successful health checks
2020-10-21T16:34:22.493Z - verbose: reqProc|_common|updateNodeStatus Starting
2020-10-21T16:34:22.493Z - verbose: Initializing ShippableAdapter
2020-10-21T16:34:22.758Z - warn: PUT call to https://con.shippable.com/clusterNodes/<STRING> returned status 404 with error 404
2020-10-21T16:34:22.759Z - error: reqProc|_common|updateNodeStatus|_updateClusterNodeStatus has failed to update status of cluster node <STRING> with err 404
2020-10-21T16:34:22.759Z - error: reqProc|_common|updateNodeStatus Failed to update node status
2020-10-21T16:34:22.759Z - error: reqProc|_common|checkHealth Failed health checks true
2020-10-21T16:34:22.759Z - error: true
2020-10-21T16:34:22.759Z - verbose: Since an error occurred, re-connecting reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:22.759Z - warn: Failed to close connection from reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
2020-10-21T16:34:22.760Z - verbose: Waiting for 16 seconds before re-connecting reqProc to Q amqps://kexigonoharihafu:<STRING>@msg.shippable.com:5671/shippable
a-murphy commented 3 years ago

When you initialized this node, was the script you ran the one downloaded for the most recently added node? The scripts contain the node ID and it looks like it may have the ID of a node that no longer exists from the 404 in the logs. The <STRING> in https://con.shippable.com/clusterNodes/<STRING> should match the one in the URL when on the node page.

mzeier commented 3 years ago

Have this working.