search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.13k stars 4.91k forks source link

Decreasing Build Size and Increased Portability #19921

Open andrewstucki opened 4 years ago

andrewstucki commented 4 years ago

Describe the enhancement:

Shrink the size of our release binaries and make them more portable.

Describe a specific use case for the enhancement or feature:

So right now our beats are objectively large. our Elastic licensed stripped auditbeat build clocks in at roughly 79MB. Currently we build our beats using gcc linking against GPL glibc. Large go projects often produce smaller statically stripped binaries when using static linking, likely in part due to additional compiler optimizations that can occur due to the static link as well as getting rid of the huge dynamic symbol table left when linking dynamically.

The problem with static linking is that we can't statically link against glibc because then our binaries would need to be GPL-compatible. So I did a quick test with a musl libc-based build in an alpine container and here are the basic results:

/src/x-pack/auditbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-s'
/src/x-pack/auditbeat # ls -la auditbeat
-rwxr-xr-x    1 root     root      82012696 Jul 14 20:31 auditbeat
/src/x-pack/auditbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/auditbeat # ls -la auditbeat
-rwxr-xr-x    1 root     root      67057816 Jul 14 20:57 auditbeat

As an added benefit, we would now get portability across operating systems with incompatible system libc.

As a secondary test I ran the static binary through UPX just to see how well packing would affect the binary:

/src/x-pack/auditbeat # upx -9 auditbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  67057816 ->  19599108   29.23%   linux/amd64   auditbeat

Packed 1 file.

So, if we went with something like that we'd basically have roughly 1/4 of the binary size while being able to support more systems out of the box.

WDYT?

elasticmachine commented 4 years ago

Pinging @elastic/siem (Team:SIEM)

andrewstucki commented 4 years ago

For some more statistics, here's all the builds I tried (non-OSS was done just to get a feel for the size reduction)

Metricbeat

Before:

[root@localhost]# ls -la metricbeat
-rwxr-xr-x. 1 root root 124485568 Jun 14 18:23 metricbeat

After:

/src/x-pack/metricbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/metricbeat # ls -la metricbeat
-rwxr-xr-x    1 root     root     101149856 Jul 14 21:50 metricbeat
/src/x-pack/metricbeat # upx -9 metricbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
 101149856 ->  27441836   27.13%   linux/amd64   metricbeat

Packed 1 file.

static musl + stripped: 81% original size static musl + stripped + packed: 22% original size

Packetbeat

Before:

[root@localhost]# ls -la packetbeat
-rwxr-xr-x. 1 root root 77414552 Jun 14 17:49 packetbeat

After:

/src/x-pack/packetbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/packetbeat # ls -la packetbeat
-rwxr-xr-x    1 root     root      65854872 Jul 14 21:29 packetbeat
/src/x-pack/packetbeat # upx -9 packetbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  65854872 ->  19294968   29.30%   linux/amd64   packetbeat

Packed 1 file.

static musl + stripped: 85% original size static musl + stripped + packed: 25% original size

Filebeat

Before:

[root@localhost]# ls -la filebeat
-rwxr-xr-x. 1 root root 89226256 Jun 14 18:17 filebeat

After:

/src/x-pack/filebeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/filebeat # ls -la filebeat
-rwxr-xr-x    1 root     root      76930584 Jul 14 21:31 filebeat
/src/x-pack/filebeat # upx -9 filebeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  76930584 ->  22323776   29.02%   linux/amd64   filebeat

Packed 1 file.

static musl + stripped: 86% original size static musl + stripped + packed: 25% original size

Auditbeat

Before:

[root@localhost]# ls -la auditbeat
-rwxr-xr-x. 1 root root 78971184 Jun 14 18:13 auditbeat

After:

/src/x-pack/auditbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/auditbeat # ls -la auditbeat
-rwxr-xr-x    1 root     root      67057816 Jul 14 20:57 auditbeat
/src/x-pack/auditbeat # upx -9 auditbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  67057816 ->  19599108   29.23%   linux/amd64   auditbeat

Packed 1 file.

static musl + stripped: 84% original size static musl + stripped + packed: 25% original size

Heartbeat

Before:

[root@localhost]# ls -la heartbeat
-rwxr-xr-x. 1 root root 74062704 Jun 14 17:32 heartbeat

After:

/src/x-pack/heartbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/heartbeat # ls -la heartbeat
-rwxr-xr-x    1 root     root      63126904 Jul 14 21:36 heartbeat
/src/x-pack/heartbeat # upx -9 heartbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  63126904 ->  18286088   28.97%   linux/amd64   heartbeat

Packed 1 file.

static musl + stripped: 85% original size static musl + stripped + packed: 25% original size

Functionbeat

Before:

[root@localhost]# ls -la functionbeat
-rwxr-xr-x. 1 root root 74990832 Jun 14 17:53 functionbeat

After:

/src/x-pack/functionbeat # CC=/usr/bin/x86_64-alpine-linux-musl-gcc go build -mod=mod -ldflags '-extldflags "-static" -s'
/src/x-pack/functionbeat # ls -la functionbeat
-rwxr-xr-x    1 root     root      64279352 Jul 14 21:39 functionbeat
/src/x-pack/functionbeat # upx -9 functionbeat
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2020
UPX 3.96        Markus Oberhumer, Laszlo Molnar & John Reiser   Jan 23rd 2020

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  64279352 ->  18753196   29.17%   linux/amd64   functionbeat

Packed 1 file.

static musl + stripped: 86% original size static musl + stripped + packed: 25% original size

Journalbeat

Can't currently build on alpine due to lack of system-available systemd headers. I'm sure we could either port the headers or build a musl-based toolchain on another distro to get this to work.

andrewstucki commented 4 years ago

Just FYI I did a bit more experimentation and was able to cross-compile most architectures out of the box with compilers I found at https://musl.cc/ -- the exceptions: arm* due to the binutils in the toolchains not coming with the gold linker, which, for some reason the go compiler has an explicit check for on arm (so we'd need to modify/build a toolchain with gold if we'd want that to work). And ppc64, which is a known thing to not support CGO due to the go compiler's lack of external linking on ppc64.

The only other snafu I ran into was seeing that the auditbeat system/package module uses dlopen to dynamically link in different version of librpm for 1. licensing purposes (it's LGPL), and 2. portability. dlopen on musl is stubbed, so it wouldn't work with a static build even though it compiles fine. I wonder if a native go implementation might work? Something like https://github.com/cavaliercoder/go-rpm?

Here are final cross-compiled/packed metricbeats (notice no upx support for 64-bit mips or s390x architectures):

/src/x-pack/metricbeat/builds # ls -la
total 422492
drwxr-xr-x   10 root     root           320 Jul 15 12:43 .
drwxr-xr-x   25 root     root           800 Jul 15 12:43 ..
-rwxr-xr-x    1 root     root      25938880 Jul 15 12:42 metricbeat.386
-rwxr-xr-x    1 root     root      27449944 Jul 15 12:42 metricbeat.amd64
-rwxr-xr-x    1 root     root      22985804 Jul 15 12:42 metricbeat.mips
-rwxr-xr-x    1 root     root     103886880 Jul 15 12:42 metricbeat.mips64
-rwxr-xr-x    1 root     root     103886784 Jul 15 12:43 metricbeat.mips64le
-rwxr-xr-x    1 root     root      22935320 Jul 15 12:43 metricbeat.mipsle
-rwxr-xr-x    1 root     root      24749764 Jul 15 12:43 metricbeat.ppc64le
-rwxr-xr-x    1 root     root     100784064 Jul 15 12:43 metricbeat.s390x

For any other architecture where we can't fully statically link (like OSX) we can still likely run it through a packer and save space on disk if we want to, but there's less compelling of a reason to because if we just gzip our binaries for distribution we get about the same savings for the actual distribution as compressing sections with something like upx.

So, final thoughts and I'll let (hopefully) others chime in:

  1. It'd be nice to statically compile our linux builds with musl for portability, we can likely do this with other *nix distros if we have supported builds for them too with their native libc since most other libc are not GPL'd like glibc
  2. If we want to have smaller binaries we might consider using upx to pack them--the concern for this isn't as big since we only get a small win on distributing the binaries since we make tarballs/rpms/debs of most things for distribution anyway and all of these are already gzipped. It would, however, give us a cost savings on disk.

Pinging @urso / @kvch / @ph in case you want to chime in since I've seen you active in some of the build/toolchain work for beats recently.

adriansr commented 4 years ago

Great initiative @andrewstucki, would love to see our binaries losing some weight.

Is this intended for Linux only? I'm curious about the usage of upx for Windows binaries, whether it can cause some trouble with binary analyzers flagging us as malware or is that a thing of the past?

Also, do you think the usage of upx -9 can cause a significant increase in startup times? IIRC there was some effort in reducing Functionbeat's startup times, but I guess this is probably a drop in the ocean.

andrewstucki commented 4 years ago

@adriansr , so yeah those are valid points. Bringing this up in the Linux context primarily because I artificially joined the two issues of static Linux builds and code packing together--namely because both change the way we build out the binaries and have size implications. 😬

That said, with regard to packing. I ran a 64 bit dynamically linked mac build at the default upx packing level and saw an ~110MB drop in the file size of a metricbeat build. I'm fairly certain we'd see comparable gains for Windows binaries. The way upx generally works is that it goes through and compresses the sections of your binary and replaces your binary entrypoint with a little unpacking trampoline that decompresses the sections and then jumps to the real entrypoint--so as you might imagine, there is a bit of a startup cost, I would imagine part of this is correlated with the size of the binary (more code == more to decompress).

What that is in terms of Functionbeat start-up times I'm not entirely sure, but I'm sure we could attempt the compression at multiple levels and test out startup time with something simple like a time ./functionbeat --help > /dev/null.

With regard to Malware--I'm not entirely sure I can say. I know that things like section entropy are taken into consideration when doing malware analysis since sometimes malware authors do things like run their code through a packer and then modify the packer signatures to make it harder to reverse, but I'm pretty sure that with most AV this is only one of many considerations. Just for kicks I dropped the Linux and Mac packed builds into VirusTotal and nothing flagged them, but no guarantees on Windows until we tried it.

So yeah, I'm not entirely convinced on packing due to the above reasons and because of the fact that we already compress the files we distribute as tarballs, so we're really just saving bytes on disk (thought that is nice). That said, I think the static compilation on Linux would be a win since we'd no longer have to worry about building with old glibc for backwards compatibility across distros and/or lack of support on Linux flavors shipping with alternative libc implementations.

urso commented 4 years ago

The only other snafu I ran into was seeing that the auditbeat system/package module uses dlopen to dynamically link in different version of librpm for 1. licensing purposes (it's LGPL), and 2. portability. dlopen on musl is stubbed, so it wouldn't work with a static build even though it compiles fine. I wonder if a native go implementation might work? Something like cavaliercoder/go-rpm?

dlopen MUST function. It is not just librpm, but other C-libs as well. For example libs for Oracle Database we can't just link statically. There is always the chance that there is no go-native library. This is why we enable CGO in the past (originally Beats have been build with CGO disabled).

When statically linking MUSL, it becomes a hard dependency. Bugs and security issues in the MUSL version used will be Beats bugs as well. With dynamic linking we don't care, as libc security issues should be handled by the user/distro.

on stripping: The go compiler includes all debug symbols twice. For once in an internal section and a second timw as DWARF. delve and the go built in profile do not even use the DWARF. +1 on always stripping our builds.

on upx: We've discussed this a few times in the past and decided not to use it. Even on Linux I wonder how upx interacts with apparmor or even more aggressive system hardening. For saving network bandwith tar.bz2 should be good enough. Users can always opt to compress binaries locally via upx, but I don't think its worth it to package them out of the box.

glibc devs spend a huge amount of effort in optimizing performance. Do we have an idea if/how MUSL affects resource usage for Beats?

jlind23 commented 2 years ago

@nimarezainia is it something we should still think about?

elasticmachine commented 6 months ago

Pinging @elastic/security-scalability (Team:Security-Scalability)