Alpine Linux support

[FlorianHeigl]

Describe the enhancement: Agent support for Alpine Linux

Describe a specific use case for the enhancement or feature: I was testing Elastic Security (Endgame) and wanted to present a bit about it in a talk. The linux tar.gz agent currently doesn't install on Alpine. It would be great if you made the tar.gz work there or at least provided most minimal build instructions. I can help with creating an OS package manifest if that is desirable, but I saw i.e. apm-php does simply use fpm. Why would it matter? Alpine Linux has a very low attack surface, but any reasonable endpoint security setup would not be based on only partially supporting the environment

Alpine isn't a standard enterprise distro, but a good choice for a resilient infrastructure, and via docker it has become very widespread.

[nimarezainia]

thanks for raising this issue @FlorianHeigl - at the moment support for Alpine isn't on our development radar.

[gabrieligbastos]

+1

[gabrieligbastos]

+1

Also tried to run with bitnami And didnt work...

Is there any plan or workaround to it run with some smaller image that you support?

[fludo]

+1 would need it since its our base image for all deployments in K8s.

[alxproject]

+1

[mariocandela]

+1

[gianfrancodemarco]

+1! If direct support isn't on the roadmap, would you provide some hint on how to proceed? @nimarezainia

[cmacknz]

The agent is primarily developed in Go (except endpoint security) but has multiple C dependencies that need to be built specifically to target Alpine Linux.

For the agent would need to update our cross-compilation process to support Alpine which uses musl libc instead glibc which is what we link against today with our existing Debian based cross-compilation environment. We would also need to regularly test on Alpine Linux.

I don't think this is as simple as just running go build in an Alpine container unfortunately, although that might work for some configurations. It is definitely more complex than that to support endpoint security on Alpine since it isn't developed in Go.

[nimarezainia]

@gianfrancodemarco as mentioned above currently we are not planning on proving support for this OS.

[cmacknz]

There is a glibc compatibility library that may allow Beats and Agent to run on Alpine Linux without having to recompile them to specifically target Alpine Linux.

https://git.adelielinux.org/adelie/gcompat

We haven't tested this ourselves, but since there are a few requests for work arounds I wanted to link it here.

[FlorianHeigl]

I'll give it a shot asap and report back. It was on my mind it SHOULD be working like that but had no energy to try. I used to run glibc tools in a debian chroot on alpine with bind mounts, but for a security tool that is pointless.

Edit: I tried now.

At the moment, there's an issue with installing, but it seems a small one

[pid 18392] <... read resumed>"y\n", 4096) = 2
[pid 18392] futex(0x7fba2d363cb8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 18391] <... futex resumed>)        = 0
[pid 18392] readlinkat(AT_FDCWD, "/proc/self/exe",  <unfinished ...>
[pid 18391] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 18392] <... readlinkat resumed>"/lib/ld-musl-x86_64.so.1", 128) = 24
[pid 18391] <... nanosleep resumed>NULL) = 0
[pid 18392] newfstatat(AT_FDCWD, "/lib/elastic-agent",  <unfinished ...>
[pid 18391] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 18392] <... newfstatat resumed>0xc00022c518, 0) = -1 ENOENT (No such file or directory)
[pid 18391] <... nanosleep resumed>NULL) = 0
[pid 18392] write(2, "Error: failed to discover the so"..., 206 <unfinished ...>
Error: failed to discover the source directory for installation: missing elastic-agent
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.7/fleet-troubleshooting.html

from what I can see the system call newfstatat failed to return data. the good news is that it's supported by musl and would work on a kernel >2020. So, I think it's just some implementation issue in gcompat. We can report it there and than we can see how it goes.

If you can, please could you provide a link to the installer source where it calls this?

edit2: it is super odd.

localhost:/install/elastic-agent-8.7.0-linux-x86_64# ./elastic-agent -c elastic-agent.yml 
Error: could not read configuration file /lib/elastic-agent.yml: open /lib/elastic-agent.yml: no such file or directory
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.7/fleet-troubleshooting.html
localhost:/install/elastic-agent-8.7.0-linux-x86_64# pwd
/install/elastic-agent-8.7.0-linux-x86_64

anyone understand how we get a /lib in the path here? (edit: it's in path.config as default, but that's wrong)

A botched install can be achieved like this:

localhost:/install/elastic-agent-8.7.0-linux-x86_64# ./elastic-agent --path.home=$PWD --path.config=$PWD enroll --url=https://12.34.45.67:8220 --enrollment-token=sdlkfjasdlsdfjlkdfjsalfkj== --insecure
I also installed auditd to ensure some better OS integration.

The culprit now seems to be related to this whole thing of looking in /lib while we run the installer in /install/elastic-agent. I suppose there's some OS dependent bit that goes wrong. It would be good if this could get a quick glance from elastic.

That's for solving it. a good approach if you need a stable way of installing is probably like this:

1. install gcompat
2. install alien
3. download the dpkg agent and convert it to tarball
4. install THIS tar distribution to avoid going via the normal elastic-agent install. That would not work due to install not being supported in some conditions, the way that was solved means you'll be a hitting a brick wall.

[FlorianHeigl]

I also found this prior post, https://discuss.elastic.co/t/unable-to-install-fleet-managed-elastic-agent/274774 it's closed; but maybe you could put an update there in case this route works out well.

[cmacknz]

Setting --path.config should be how you control which directory the agent looks in to find its configuration file, -c just changes the name of the configuration file, not the path to it. https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-global-flags

It's not clear if you've recompiled the agent on an Alpine Linux host which is necessary for this to work. Alpine uses musl libc and the existing Linux builds of Elastic Agent are linked against glibc.

[FlorianHeigl]

@cmacknz hey, thanks for the reply. I am quite sure I tried with path.config and a few more. someone else will need to reproduce my results or make the next step as i'm error prone at times ;-)

1. i used the glib compat layer as suggested above by, errrr, you? to my understanding there is NOT a neccessity to rebuild, nor would I expect that build to just work.
2. From what I recall, some of the steps correctly look to path.config, but not all. it can just end up empty or not be propagated to the next step. There was a bug report I found regarding that, where people triggered it by running the wrong install method, but it was closed. There are likely just a few dead code paths in the handling of the variable.
3. Please don't ask me to look at it again, it was painful.
4. OTOH it is also certainly tiny issues so it'll be possible to get it solved over time.

It would be good though to know which is the best looking way. I think the one with gcompat is best since there are clear upstreams, one can easily look into what calls are implemented etc. Recompiling for alpine would be sweeter and also beat a path for FreeBSD, but we have ~1k community packages that need maintainers in Alpine, so this is not a wise approach (IMO)

[FlorianHeigl]

A pointer on Stackoverflow that seemingly used a different compatibility library and has apparently worked for the person that asked for help: https://stackoverflow.com/a/70025196/2949743

[FlorianHeigl]

@nimarezainia hi, does your development radar already show a faraway blip in the meantime?