elastic / elastic-package

elastic-package - Command line tool for developing Elastic Integrations
Other
49 stars 113 forks source link

Will "elastic-package" run on ARM? #497

Closed mdelapenya closed 1 month ago

mdelapenya commented 3 years ago

We are trying to consume the tool directly in the e2e-testing framework, to simplify/unify the provisioning part of the runtime dependencies, but when executed "stack up" on ARM workers on CI, we see this error:

[2021-08-25T15:00:23.729Z] time="2021-08-25T15:00:22Z" level=error msg="Error executing command" args="[run github.com/elastic/elastic-package stack up --daemon --verbose --version 8.0.0-126d5980-SNAPSHOT --services elasticsearch,fleet-server,kibana,package-registry]" baseDir=. command=go error="exit status 1" stderr="go: downloading github.com/elastic/elastic-package v0.16.0\ngo: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b\ngo: downloading github.com/AlecAivazis/survey/v2 v2.2.16\ngo: downloading github.com/elastic/package-spec/code/go v0.0.0-20210811110254-a7173524bc76\ngo: downloading github.com/fatih/color v1.12.0\ngo: downloading github.com/olekukonko/tablewriter v0.0.5\ngo: downloading github.com/Masterminds/semver v1.5.0\ngo: downloading github.com/elastic/go-elasticsearch/v7 v7.14.0\ngo: downloading github.com/google/go-github/v32 v32.1.0\ngo: downloading golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602\ngo: downloading github.com/elastic/go-ucfg v0.8.3\ngo: downloading github.com/go-git/go-git/v5 v5.4.2\ngo: downloading github.com/google/go-querystring v1.1.0\ngo: downloading github.com/cespare/xxhash v1.1.0\ngo: downloading github.com/go-git/go-billy/v5 v5.3.1\ngo: downloading github.com/cespare/xxhash/v2 v2.1.1\ngo: downloading github.com/jedib0t/go-pretty v4.3.0+incompatible\ngo: downloading github.com/mattn/go-colorable v0.1.8\ngo: downloading github.com/mattn/go-isatty v0.0.12\ngo: downloading github.com/magefile/mage v1.11.0\ngo: downloading github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51\ngo: downloading github.com/mattn/go-runewidth v0.0.9\ngo: downloading github.com/aymerick/raymond v2.0.2+incompatible\ngo: downloading github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7\ngo: downloading github.com/imdario/mergo v0.3.12\ngo: downloading github.com/kylelemons/godebug v1.1.0\ngo: downloading github.com/Masterminds/semver/v3 v3.1.1\ngo: downloading github.com/creasty/defaults v1.5.1\ngo: downloading github.com/rakyll/statik v0.1.7\ngo: downloading github.com/xeipuuv/gojsonschema v1.2.0\ngo: downloading github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d\ngo: downloading golang.org/x/text v0.3.6\ngo: downloading helm.sh/helm/v3 v3.6.3\ngo: downloading k8s.io/apimachinery v0.22.0\ngo: downloading k8s.io/cli-runtime v0.22.0\ngo: downloading k8s.io/client-go v0.22.0\ngo: downloading github.com/go-git/gcfg v1.5.0\ngo: downloading github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415\ngo: downloading github.com/go-openapi/strfmt v0.19.6\ngo: downloading k8s.io/klog/v2 v2.9.0\ngo: downloading github.com/evanphx/json-patch v4.11.0+incompatible\ngo: downloading github.com/googleapis/gnostic v0.5.5\ngo: downloading k8s.io/api v0.22.0\ngo: downloading sigs.k8s.io/kustomize/api v0.8.11\ngo: downloading sigs.k8s.io/kustomize/kyaml v0.11.0\ngo: downloading github.com/PaesslerAG/jsonpath v0.1.1\ngo: downloading k8s.io/apiextensions-apiserver v0.21.0\ngo: downloading k8s.io/kubectl v0.21.0\ngo: downloading github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb\ngo: downloading github.com/google/gofuzz v1.1.0\ngo: downloading sigs.k8s.io/structured-merge-diff/v4 v4.1.2\ngo: downloading github.com/asaskevich/govalidator v0.0.0-20200428143746-21a406dcc535\ngo: downloading github.com/go-openapi/errors v0.19.2\ngo: downloading github.com/mitchellh/mapstructure v1.4.1\ngo: downloading go.mongodb.org/mongo-driver v1.1.2\ngo: downloading github.com/go-logr/logr v0.4.0\ngo: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de\ngo: downloading sigs.k8s.io/yaml v1.2.0\ngo: downloading github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7\ngo: downloading github.com/peterbourgon/diskv v2.0.1+incompatible\ngo: downloading github.com/google/go-cmp v0.5.5\ngo: downloading github.com/json-iterator/go v1.1.11\ngo: downloading github.com/modern-go/reflect2 v1.0.1\ngo: downloading github.com/PaesslerAG/gval v1.0.0\ngo: downloading k8s.io/kube-openapi v0.0.0-20210421082810-95288971da7e\ngo: downloading github.com/davecgh/go-spew v1.1.1\ngo: downloading k8s.io/utils v0.0.0-20210707171843-4b05e18ac7d9\ngo: downloading gopkg.in/inf.v0 v0.9.1\ngo: downloading github.com/exponent-io/jsonpath v0.0.0-20151013193312-d6023ce2651d\ngo: downloading k8s.io/component-base v0.21.0\ngo: downloading github.com/google/btree v1.0.1\ngo: downloading golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac\ngo: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd\ngo: downloading github.com/go-openapi/spec v0.19.5\ngo: downloading github.com/MakeNowJust/heredoc v0.0.0-20170808103936-bb23615498cd\ngo: downloading github.com/russross/blackfriday v1.5.2\ngo: downloading github.com/go-stack/stack v1.8.0\ngo: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510\ngo: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00\ngo: downloading github.com/stretchr/testify v1.7.0\ngo: downloading github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca\ngo: downloading github.com/go-errors/errors v1.0.1\ngo: downloading github.com/go-openapi/jsonreference v0.19.5\ngo: downloading github.com/go-openapi/swag v0.19.14\ngo: downloading github.com/mitchellh/go-wordwrap v1.0.0\ngo: downloading github.com/go-openapi/jsonpointer v0.19.5\ngo: downloading github.com/PuerkitoBio/purell v1.1.1\ngo: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5\ngo: downloading github.com/pmezard/go-difflib v1.0.0\ngo: downloading github.com/mailru/easyjson v0.7.6\ngo: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578\ngo: downloading github.com/moby/spdystream v0.2.0\ngo: downloading github.com/josharian/intern v1.0.0\nelastic-package has been installed.\n2021/08/25 14:57:31 DEBUG Enable verbose logging\n2021/08/25 14:57:31 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases\nBoot up the Elastic stack\nUsing profile /var/lib/jenkins/workspace/PR-1502-5-fd26d21a-617a-472a-8064-677b12fee491/.elastic-package/profiles/default.\nRemember to load stack environment variables using 'eval \"$(elastic-package stack shellinit)\"'.\n2021/08/25 14:57:31 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/PR-1502-5-fd26d21a-617a-472a-8064-677b12fee491/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build elasticsearch elasticsearch_is_ready fleet-server fleet-server_is_ready kibana kibana_is_ready package-registry package-registry_is_ready\nelasticsearch uses an image, skipping\nelasticsearch_is_ready uses an image, skipping\nkibana uses an image, skipping\nkibana_is_ready uses an image, skipping\npackage-registry_is_ready uses an image, skipping\nfleet-server uses an image, skipping\nfleet-server_is_ready uses an image, skipping\nBuilding package-registry\n2021/08/25 14:57:58 DEBUG running command: /usr/local/bin/docker-compose -f /var/lib/jenkins/workspace/PR-1502-5-fd26d21a-617a-472a-8064-677b12fee491/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack up -d elasticsearch elasticsearch_is_ready fleet-server fleet-server_is_ready kibana kibana_is_ready package-registry package-registry_is_ready\nCreating network \"elastic-package-stack_default\" with the default driver\nPulling elasticsearch_is_ready (tianon/true:)...\nCreating elastic-package-stack_elasticsearch_1 ... \r\nCreating elastic-package-stack_package-registry_1 ... \r\n\x1b[1A\x1b[2K\rCreating elastic-package-stack_package-registry_1 ... \x1b[32mdone\x1b[0m\r\x1b[1B\x1b[2A\x1b[2K\rCreating elastic-package-stack_elasticsearch_1 ... \x1b[32mdone\x1b[0m\r\x1b[2BCreating elastic-package-stack_package-registry_is_ready_1 ... \r\n\x1b[1A\x1b[2K\rCreating elastic-package-stack_package-registry_is_ready_1 ... \x1b[32mdone\x1b[0m\r\x1b[1BCreating elastic-package-stack_kibana_1 ... \r\nCreating elastic-package-stack_elasticsearch_is_ready_1 ... \r\n\x1b[1A\x1b[2K\rCreating elastic-package-stack_elasticsearch_is_ready_1 ... \x1b[32mdone\x1b[0m\r\x1b[1B\x1b[2A\x1b[2K\rCreating elastic-package-stack_kibana_1 ... \x1b[32mdone\x1b[0m\r\x1b[2BCreating elastic-package-stack_fleet-server_1 ... \r\nCreating elastic-package-stack_kibana_is_ready_1 ... \r\n\x1b[1A\x1b[2K\rCreating elastic-package-stack_kibana_is_ready_1 ... \x1b[32mdone\x1b[0m\r\x1b[1B\x1b[2A\x1b[2K\rCreating elastic-package-stack_fleet-server_1 ... \x1b[32mdone\x1b[0m\r\x1b[2B\nERROR: for package-registry_is_ready Container \"b4322208e286\" is unhealthy.\nEncountered errors while bringing up the project.\nError: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 1\nexit status 1\n"

I wonder if the compose implementation is requesting the right platform for all components in the stack.

mtojek commented 3 years ago

It looks like the application was built properly, but there is problem with a single container:

 Container "b4322208e286" is unhealthy

It would be nice to look what's inside, maybe logs contain something. This might be also a problem if the stack is booted on machine with low resource limits (memory), so it can't boot up in time (hence unhealthy).

mtojek commented 3 years ago

@mdelapenya If you look at the Releases page, it includes ARM support (64bit): https://github.com/elastic/elastic-package/releases .

Could you please retry your scenario to revalidate this issue?

marclop commented 3 years ago

I ran into this while trying to test the apm-server with docker-compose up -d in the `apm-server repo https://github.com/elastic/apm-server/pull/6177#issuecomment-920222496.

mtojek commented 3 years ago

I assume you're referring to signal 11 (Segmentation fault) - core dumped. I'm not sure if this is related to the problem mentioned here.

Are you sure that it isn't related to qemu?

marclop commented 3 years ago

@mtojek that's coming from inside the container which failed to start which uses the docker.elastic.co/package-registry/distribution:snapshot docker image. We use docker-compose in the APM server team, you can see the docker-compose.yml file here.

$ CONTAINER ID   IMAGE          COMMAND            CREATED          STATUS                      PORTS                     NAMES
4d9948f77f82   f32cf92344d6   "/entrypoint.sh"   33 minutes ago   Up 33 minutes (unhealthy)   0.0.0.0:51432->8080/tcp   apm-server_package-registry_1

$ docker logs -n10 apm-server_package-registry_1
2021/09/16 11:16:43 zookeeper                    1.0.0  /packages/snapshot/zookeeper/1.0.0
2021/09/16 11:16:43 zoom                         0.5.0  /packages/snapshot/zoom/0.5.0
2021/09/16 11:16:43 zoom                         0.5.1  /packages/snapshot/zoom/0.5.1
2021/09/16 11:16:43 zscaler                      0.3.0  /packages/snapshot/zscaler/0.3.0
2021/09/16 11:16:43 zscaler                      0.3.1  /packages/snapshot/zscaler/0.3.1
2021/09/16 11:16:43 zscaler                      0.3.2  /packages/snapshot/zscaler/0.3.2
2021/09/16 11:16:43 zscaler                      0.3.3  /packages/snapshot/zscaler/0.3.3
2021/09/16 11:16:43 Packages in /packages/local:
2021/09/16 11:16:43 apm                          0.5.0  /packages/local/apm/0.5.0
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

$ uname -a
Darwin REDACTED 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T8101 arm64

The problem here comes not that the elastic-package isn't being cross compiled for linux/darwin arm64, but that the docker images aren't being built and published for arm64:

$ docker inspect --format='OS: {{ .Os }}, Arch: {{ .Architecture }}' docker.elastic.co/package-registry/distribution:snapshot
OS: linux, Arch: amd64

Ideally, we'd also build the docker images for multiple architectures instead of only for amd64.

mtojek commented 3 years ago

@jlind23 I think we need a meta-issue to support ARM arch in our developer pipeline (https://github.com/elastic/package-registry/issues/723 and https://github.com/elastic/elastic-package/issues/497 can be linked there.)

jlind23 commented 3 years ago

Just created it here: https://github.com/elastic/elastic-package/issues/539

mrodm commented 1 month ago

The problem here comes not that the elastic-package isn't being cross compiled for linux/darwin arm64, but that the docker images aren't being built and published for arm64: $ docker inspect --format='OS: {{ .Os }}, Arch: {{ .Architecture }}' docker.elastic.co/package-registry/distribution:snapshot OS: linux, Arch: amd64 Ideally, we'd also build the docker images for multiple architectures instead of only for amd64.

Closing this issue since the support for Package-Registry container images as a multi-platform one (linux/amd64 and linux/arm64) was added in https://github.com/elastic/package-registry/pull/1162 and those docker images are already used in the Elastic stack started by elastic-package.