search

fleetdm / fleet

Open-source platform for IT, security, and infrastructure teams. (Linux, macOS, Chrome, Windows, cloud, data center)
https://fleetdm.com
Other
2.7k stars 384 forks source link

Enroll Linux on arm64 hosts to Fleet #1845

Open mikermcneil opened 2 years ago

mikermcneil commented 2 years ago

Goal

User story
As an endpoint operator,
I want to generate a fleetd agent w/ and w/o Fleet Desktop that I can install on Linux arm64
so that I can enroll my Linux hosts that run on arm64.

Product

Engineering

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Context

QA

Risk assessment

Manual testing steps

  1. Step 1
  2. Step 2
  3. Step 3

Testing notes

Confirmation

  1. [ ] Engineer (@____): Added comment to user story confirming successful completion of QA.
  2. [ ] QA (@____): Added comment to user story confirming successful completion of QA.
zmackie commented 2 years ago

Little bit of clarification: we've started to investigate running on ARM architecture and its becoming a requirement for considering any solution. Osquery upstream does support ARM at this time.

noahtalerman commented 2 years ago

@zmackie is this a request for releasing Orbit for Linux ARM specifically?

Thank you for the above clarification.

mikermcneil commented 2 years ago

See also https://github.com/fleetdm/fleet/issues/1031

zmackie commented 2 years ago

One of our requirements for adopting fleet is the ability to deploy osquery to aarch64 hosts (amazon graviton). So, for us, the packaging commands will need to be able to produce something that targets that architecture. ie go run ./cmd/package/package.go --type deb --arch ARM64

As I understand it, upstream osquery binaries for that architecture are available, so my guess is that this would be a matter of

  1. using those upstream binaries
  2. compiling orbit to that architecture (easy in go - the compiler can do this)
lizthegrey commented 2 years ago

You may want to be able to manage M1 Mac ARM64 too, but +1 to do Linux ARM64 first.

n8felton commented 2 years ago

+1 for arm64 (M1/Apple Silicon) support on macOS. Preference would be a Universal 2 binary.

Goals:

lucasmrod commented 2 years ago

We've created #4420 for the M1/Apple support. And this issue will be dedicated to Linux arm64 support.

sharvilshah commented 2 years ago

High level notes on how to go about building fleetctl, orbit and fleet-desktop on Linux aarch64:

There might be some unknown unknowns here, this still needs a bit of digging deeper and prioritization

zhumo commented 1 year ago

Hello, given our current capacity, we will de-prioritize this in favor of other work. We will reconsider later in the quarter once we've met our other commitments or if this becomes a customer commitment.

rhuddleston commented 1 year ago

Here is a Makefile and Dockerfile that will build both amd64 and arm64 and verified it works:

Makefile:

VERSION=4.22.1
IMAGE=fleetdm/fleet:${VERSION}

.DEFAULT: buildx

buildx:
    @export DOCKER_CLI_EXPERIMENTAL=enabled
    @docker buildx build --build-arg VERSION=${VERSION} --platform linux/amd64,linux/arm64 -t ${IMAGE} --push .
    @echo
    @echo "The image has been built: ${IMAGE}"
    @echo

Dockerfile:

FROM --platform=$BUILDPLATFORM node:16 as website

ENV BUILD_ORG fleetdm
ENV PROJECT_NAME fleet
ARG VERSION

WORKDIR /opt/fleet
RUN git clone --depth 1 --branch fleet-v$VERSION https://github.com/$BUILD_ORG/$PROJECT_NAME.git /opt/$PROJECT_NAME && \
    make deps-js && make generate-js

FROM golang:1.19.3-alpine3.16 as builder

ENV GO111MODULE on

COPY --from=website /opt/fleet/ /opt/fleet
RUN apk update && \
    apk add ca-certificates git bash gcc musl-dev make && \
    git config --global http.https://gopkg.in.followRedirects true && \
    cd /opt/fleet && \
    make deps-go && make generate-go && make

FROM alpine:3.16

COPY --from=builder /opt/fleet/build/fleet* /usr/bin/
RUN apk --no-cache add ca-certificates && \
    ln -s /tmp /.goquery

CMD ["fleet", "serve"]

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

zhumo commented 1 year ago

hey @rhuddleston That's awesome. We'd like to start off by including that in our documentation. Someone from fleet (@chiiph?) will reach out about that.

I also created an issue that tracks adding your contribution to our dockerhub. https://github.com/fleetdm/fleet/issues/8904

michalnicp commented 1 year ago

@rhuddleston I tested your Makefile and Dockerfile and it looks pretty good. Unfortunately, I think we would need to update our ci workflows and make some significant changes to our existing Makefile and Dockerfile so that the multi-platform image gets pushed to dockerhub. All of our existing deployments user dockerhub as the docker registry. We will do this in #8904.

RachelElysia commented 1 year ago

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

@rhuddleston we got the green light to add your solution to the Deploying section of our docs. Since you offered and it's your solution, would you like to open a PR to contribute where you see fit in Deploying? We want to include a caveat that we don't support deploying using arm officially, but this is the suggested way in the meantime. :)

Feel free to add me, @chiiph, and @chris-mcgillicuddy as reviewers, or let me know if you just want me to add it to the documentation instead.

rhuddleston commented 1 year ago

Can we just have the documnetation link it to a directory in the fleet source with these two files?

mike-j-thomas commented 1 year ago

Hey @zhumo, please can you give me a little more info about what you need from #g-website team for this? I'm not sure if you are asking for us to create a directory for files in the comment above

Glad to add it to the repo if there is a directory where it makes sense (maybe down tools)?

Or if you only want us to add a reference in the docs?

We'd like to start off by including that in our documentation.

mike-j-thomas commented 1 year ago

Hey @zhumo, just a little nudge about what you need for this ⬆️

zhumo commented 1 year ago

Shoot, sorry @mike-j-thomas I think I mis-labeled this. Not sure what I was thinking.

dherder commented 7 months ago

@noahtalerman The ability to deploy Linux agents on arm64 is becoming critical for testing linux script execution (and the development of linux script libraries). Can we re-examine this, if just to build fleetd arm64 compatible package creation?

alirezaghey commented 5 months ago

Do I understanding this correctly, that there is currently no way to add Linux arm64 hosts to fleet?

noahtalerman commented 5 months ago

@alirezaghey yes. We're working on adding support for Linux arm64.

noahtalerman commented 5 months ago

@mikermcneil heads up, I moved your original issue description here:

As a user of osquery via Orbit (aka Fleet-osquery), I want a release of it available for Linux arm64.

How?

noahtalerman commented 5 months ago

Hey @georgekarrv IIRC we spoke briefly about how we might accomplish this and you had a pretty good grasp of what we need to do.

If that's true, when you get the chance, can you please update the issue description w/ the required changes to enroll Linux on arm64 hosts?

dherder commented 4 months ago

more interest in arm64 support: https://macadmins.slack.com/archives/C0214NELAE7/p1708535372632249

noahtalerman commented 4 months ago

@dherder heads up, this user story is in the current design sprint (drafting) so we're targeting shipping this in the next 6 weeks. No need to bring it back to feature fest yet.

We'll update you if the story doesn't make the 3 week drafting timeline. At that point, we can bring it back to feature fest.

georgekarrv commented 4 months ago

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @jahzielv @mna @roperzh

dantecatalfamo commented 4 months ago

With regards to cross-compiling CGO, I would highly suggest taking a look at zig cc if this hasn't already been solved. I've used it for cross-compiling my own CGO projects, and it makes builds across multiple targets and architectures substantially easier. Uber already uses it for this purpose.

nonpunctual commented 2 months ago

Duplicates: https://github.com/fleetdm/fleet/issues/18532

noahtalerman commented 2 months ago

Hey @nonpunctual heads up, this story is in the current engineering sprint (has the :release label and 4.50-tentative milestone.

Removing this story from feature fest.

cc @georgekarrv

nonpunctual commented 2 months ago

@noahtalerman when this closes https://github.com/fleetdm/fleet/issues/18532 can also be closed as dupe. Thanks.

lucasmrod commented 2 months ago

fleetctl changes: Universal fleetctl binary for Linux (this might already exist)

~TIL about universal binaries for Linux. Couldn't we do the same for fleetd components? (That is, build orbit and fleet-desktop as fat linux executables so that we don't need more targets to publish on every release. If possible, this would keep releasing fleetd simpler)~

EDIT: It seems FatELF is not a thing (not going to be supported by Linux kernel). https://icculus.org/finger/icculus?date=2009-11-03&time=19-08-04.

NickBorgersOnLowSecurityNode commented 2 months ago

I ran into this today trying to enroll an arm64 Linux box.

Nice to see this is already being worked on and potentially being released soon.

valentinpezon-primo commented 2 months ago

Hi - We just ran into the the problem aswell, we were trying to build a linux installer on a amd64 machine and the fleetctl tool did not succeed, it was unable to create the linux package

Command used : fleetctl package --type="deb" --verbose --debug --enable-scripts --fleet-desktop --fleet-url="secret" --enroll-secret="secret"

Error : Error: open output file: open build/fleet-osquery_1.24.0_amd64.deb: no such file or directory

We are building the installer in the cloud, inside an AWS Fargate task, and this runs on Linux amd64 behind the scene.

Note: The command worked and I successfully managed to create a linux installer on my Mac M1.

If I understand correctly, this github issue is related and it should fix my problem ?

Thanks

cc @nonpunctual

alirezaghey commented 2 months ago

hey @valentinpezon-primo, this issue is related to enrolling arm64 linux devices. AFAIK, your issue is not related to this thread. On another note: I think you should first investigate how fargate handles your file output, since this could be related to the serverless structure of fargate, depending on how you set it up. Just a guess though, I'm in no way experienced with it.

valentinpezon-primo commented 2 months ago

Thanks for your input @alirezaghey

I tried to make it work on a local Docker on my MacBook pro and it also did not work, I can share the files if you want to reproduce

DockerFile : 

FROM fleetdm/fleetctl

ENV FLEET_URL "https://xxx.com"
ENV ENROLLMENT_SECRET "xxxx"

COPY entrypoint.sh /entrypoint.sh

RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

WORKDIR /tmp

entrypoint.sh :

#!/bin/bash

fleetctl package --type="deb" --verbose --debug --enable-scripts --fleet-desktop --fleet-url=$FLEET_URL --enroll-secret=$ENROLLMENT_SECRET

ls -lA

Commands & output :

docker build -t generate-linux-installer:latest path/to/folder/whith/dockerfile docker run --platform linux/amd64 --rm -ti generate-linux-installer:latest 

Generating your fleetd agent...
{"level":"debug","path":"/tmp/orbit-package2405643366","time":"2024-05-14T11:56:32Z","message":"created temp directory"}
{"level":"debug","error":"stat /tmp/orbit-package2405643366/root/opt/orbit/bin/osqueryd/linux/stable/osqueryd: no such file or directory","time":"2024-05-14T11:56:33Z","message":"stat file"}
{"level":"debug","error":"stat /tmp/orbit-package2405643366/root/opt/orbit/bin/orbit/linux/stable/orbit: no such file or directory","time":"2024-05-14T11:56:39Z","message":"stat file"}
{"level":"debug","error":"stat /tmp/orbit-package2405643366/root/opt/orbit/bin/desktop/linux/stable/desktop.tar.gz: no such file or directory","time":"2024-05-14T11:56:43Z","message":"stat file"}
{"level":"debug","data":"orbit={/tmp/orbit-package2405643366/root/opt/orbit/bin/orbit/linux/stable/orbit,1.24.0}, osqueryd={/tmp/orbit-package2405643366/root/opt/orbit/bin/osqueryd/linux/stable/osqueryd,5.12.1}","time":"2024-05-14T11:56:46Z","message":"updates initialized"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/etc/default/orbit","Destination":"/etc/default/orbit","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":384,"MTime":"2024-05-14T11:56:46.097691009Z","Size":334}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/bin/desktop/linux/stable/desktop.tar.gz","Destination":"/opt/orbit/bin/desktop/linux/stable/desktop.tar.gz","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:45.229691009Z","Size":16210606}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop","Destination":"/opt/orbit/bin/desktop/linux/stable/fleet-desktop/fleet-desktop","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:46.049691009Z","Size":37516008}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/bin/orbit/linux/stable/orbit","Destination":"/opt/orbit/bin/orbit/linux/stable/orbit","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:42.502691007Z","Size":40996236}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/opt/orbit/bin/orbit/linux/stable/orbit","Destination":"/opt/orbit/bin/orbit/orbit","Type":"symlink","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":134218221,"MTime":"2024-05-14T11:56:46.112790467Z","Size":0}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/bin/osqueryd/linux/stable/osqueryd","Destination":"/opt/orbit/bin/osqueryd/linux/stable/osqueryd","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:39.263691006Z","Size":86504304}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/certs.pem","Destination":"/opt/orbit/certs.pem","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":420,"MTime":"2024-05-14T11:56:46.099691009Z","Size":229654}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/osquery.flags","Destination":"/opt/orbit/osquery.flags","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":384,"MTime":"2024-05-14T11:56:46.098691009Z","Size":0}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/opt/orbit/tuf-metadata.json","Destination":"/opt/orbit/tuf-metadata.json","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":384,"MTime":"2024-05-14T11:56:33.359691003Z","Size":119821}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/tmp/orbit-package2405643366/root/usr/lib/systemd/system/orbit.service","Destination":"/usr/lib/systemd/system/orbit.service","Type":"","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":420,"MTime":"2024-05-14T11:56:46.086691009Z","Size":317}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"/opt/orbit/bin/orbit/orbit","Destination":"/usr/local/bin/orbit","Type":"symlink","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":134218221,"MTime":"2024-05-14T11:56:46.112796259Z","Size":0}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"","Destination":"/var/log/orbit","Type":"dir","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:46.104382467Z","Size":0}},"time":"2024-05-14T11:56:46Z","message":"added file"}
{"level":"debug","file":{"Source":"","Destination":"/var/log/osquery","Type":"dir","Packager":"","FileInfo":{"Owner":"root","Group":"root","Mode":493,"MTime":"2024-05-14T11:56:46.104377134Z","Size":0}},"time":"2024-05-14T11:56:46Z","message":"added file"}
Error: open output file: open build/fleet-osquery_1.24.0_amd64.deb: no such file or directory
total 0

See the error : Error: open output file: open build/fleet-osquery_1.24.0_amd64.deb: no such file or directory

Notes :

Edit :

valentinpezon-primo commented 2 months ago

Seems like the error comes from this line of code

https://github.com/fleetdm/fleet/blob/5af8392615d70afcc31c013c4dca59206b7f2cc1/orbit/pkg/packaging/linux_shared.go#L228

I do not understand what could cause this tho.

ksatter commented 2 months ago

I set up #18997 to track @valentinpezon-primo's issue.

nonpunctual commented 2 months ago

List of issues related to not supporting arm64:

https://github.com/fleetdm/fleet/issues/1031 https://github.com/fleetdm/fleet/issues/1845 https://github.com/fleetdm/fleet/issues/2466 https://github.com/fleetdm/fleet/issues/4420 https://github.com/fleetdm/fleet/issues/4430 https://github.com/fleetdm/fleet/issues/8257 https://github.com/fleetdm/fleet/issues/8904 https://github.com/fleetdm/fleet/issues/9047 https://github.com/fleetdm/fleet/issues/10864 https://github.com/fleetdm/fleet/issues/18532

zayhanlon commented 1 month ago

@georgekarrv @noahtalerman Can I get an update today on if this is selected for the 4.52.0 sprint? Thanks!

lucasmrod commented 1 month ago

In the description, the following should be changed from:

fleetctl changes: Universal fleetctl binary for Linux (this might already exist)

to:

fleetctl binary for arm64 linux

(there's no universal binary in Linux)

marko-lisica commented 3 weeks ago

@dantecatalfamo We reviewed this during the design review and came up with the following:

Add flag description to help (add it below --type)

$fleetctl package -h

--arch   Architecture of package to build (only available with '--type' deb or rpm) (default: amd64)

Error messages:

If wrong --type is specified: Error: can't use '--arch' with '--type <type>' -> is dynamic based on what user specified.

If wrong --arch is specified: Error: arch must be one of ('amd64', 'arm64')

dantecatalfamo commented 2 weeks ago

The PR for this issue is complete, it is just waiting on QA and getting the components pushed to the fleet TUF server

roperzh commented 1 week ago

@dantecatalfamo osqueryd for linux-arm64 is in edge now (only osqueryd) to unblock your testing, once that's confirmed please:

  1. ping all the people necessary to get that PR merged
  2. ping me and I'll release fleetd to edge, this will include a few fixes so it'll go through the normal QA process