christoph-zededa
commented
4 days ago use shared libraries for Go binaries
Open rucoder opened 4 days ago
christoph-zededa
commented
4 days ago use shared libraries for Go binaries
deitch
commented
3 days ago Are we comfortable with shared libraries vs static binaries? I guess within a single container it's fine, although we would like to break pillar apart.
Then again, does it matter, if most of it is a single pillar binary?
christoph-zededa
commented
3 days ago Then again, does it matter, if most of it is a single pillar binary?
There is f.e. also the vtpm binary
christoph-zededa
commented
3 days ago single container
containerd is 36MB, I guess it would also profit a bit from shared libraries
I think if we have the same (i.e. same hash sum) shared library in different containers, then squashfs can compress it very well.
deitch
commented
3 days ago It would be an interesting experiment to see what actually happens if we use shared libraries. We could be spending all of this time to discover that it makes little difference.
I know that if you launch two copies of the same executable, the read only memory consumption is once. Same for shared objects. If two distinct executables, each way 5MB, use a shared object of 3MB, then total memory is 52+3, and not (5+3)2, ie one copy of the shared object.
What about statically compiled? Is it intelligent enough to recognize the common bits? I would think no, but @rucoder would know much better.
rucoder
commented
3 days ago @deitch this is exactly why shared libraries were born. Static images cannot share their code. I would be nice if we can have all libs in /lib in 'on_init' section in linux kit. LK can do some magic to move all libs from all containers to /lib or /usr/lib and mount them into containers behind the scene
deitch
commented
3 days ago this is exactly why shared libraries were born
I know. I didn't know if in the years since, loaders had gotten smarter about recognizing identical binary chunks in different files.
Well, this thread isn't really about memory usage, is it? I was just checking that as as side question.
I would be nice if we can have all libs in /lib in 'on_init' section in linux kit. LK can do some magic to move all libs from all containers to /lib or /usr/lib and mount them into containers behind the scene
You mean to share libraries across containers? I don't see it. Nothing in containerd infrastructure supports it now, I haven't seen anyone else doing it.
I have seen people mounting some shared libs from a host filesystem into multiple containers, but it isn't all that common. Mainly because size of root filesystem doesn't matter all that much to most?
rucoder
commented
3 days ago @deitch this is exactly what we need to solve: the image size. I think it is not that hard to bind mount /hosfs/lib into /lib inside the container. We can even do it manually in build.yml but LK must move all libs from individual containers to host fs and de-duplicate them
christoph-zededa
commented
3 days ago Nothing in containerd infrastructure supports it now
If two containers share the same base layer, then in the overlayfs they have the same lower mount, don't they? Then they share the files.
rucoder
commented
3 days ago yes, if SHA is the same but all our containers are FROM scratch with different SHAs
deitch
commented
3 days ago this is exactly what we need to solve: the image size
Right, not memory usage. Agreed.
I think it is not that hard to bind mount /hosfs/lib into /lib inside the container. We can even do it manually in build.yml but LK must move all libs from individual containers to host fs and de-duplicate them
Doable? Yes. I am really concerned about the management and synchronization overhead. We would need to have something that makes it easier to use. Like we have our FROM eve-alpine system that ensures it all stays in sync, we would need something like that here. And then how do you have each container run on its own, when not mounted?
As a general rule, containers that cannot run on their own, that depend on some volume mount or bind mount just to get executables to run, goes against the grain.
In any case, are we really sure that shared libraries across containers is what will make the difference? As it is, pillar - the biggest one - is mostly a single binary anyways.
We should take a good hard look at a built eve and see where the sizes are big.
deitch
commented
3 days ago FWIW, I just built eve from latest master commit:
$ du -s * | sort -n
4 config
4 home
4 mnt
4 persist
4 proc
4 root
4 run
4 srv
4 sys
4 tmp
8 opt
12 dev
16 media
84 var
1292 EFI
1372 init
3844 etc
4180 sbin
4236 bin
22580 boot
127140 usr
234784 lib
585720 containersSo the only things that really matter:
Starting with the smallest:
$ du -s -h boot/*
4.0K boot/cmdline
14M boot/kernel
7.8M boot/ucode.img
1.2M boot/xen.gzKernel and code, not much else there.
$ du -s lib/* | sort -n
0 lib/libc.musl-x86_64.so.1
0 lib/libcom_err.so.2
0 lib/libfdisk.so.1
0 lib/libmount.so.1
0 lib/libpam.so.0
0 lib/libpam_misc.so.0
0 lib/libpamc.so.0
0 lib/libsmartcols.so.1
0 lib/libuuid.so.1
0 lib/libz.so
4 lib/mdev
4 lib/modules-load.d
8 lib/sysctl.d
16 lib/libcom_err.so.2.1
16 lib/libpam_misc.so.0.82.1
16 lib/pkgconfig
20 lib/libpamc.so.0.82.1
32 lib/libuuid.so.1.3.0
52 lib/resolvconf
60 lib/libpam.so.0.85.1
88 lib/libkmod.so.2
100 lib/libz.so.1
100 lib/libz.so.1.2.12
100 lib/libz.so.1.2.13
128 lib/libudev.so.1
180 lib/libapk.so.3.12.0
200 lib/libsmartcols.so.1.1.0
308 lib/libblkid.so.1
308 lib/libblkid.so.1.1.0
332 lib/libmount.so.1.1.0
400 lib/libfdisk.so.1.1.0
512 lib/libssl.so.1.1
556 lib/udev
592 lib/ld-musl-x86_64.so.1
592 lib/libssl.so.3
740 lib/security
996 lib/apk
2552 lib/libcrypto.so.1.1
3792 lib/libcrypto.so.3
65556 lib/modules
156420 lib/firmwarelib/firmware is 153MB. 202 lines in there, not going to post it all here. But most are .fw or .ucode files of 1.5-2MB, with a few really big ones:
$ du -s lib/firmware/* | sort -n
...
4236 lib/firmware/intel
6232 lib/firmware/cypress
8876 lib/firmware/display-t234-dce.bin
8996 lib/firmware/mrvl
9452 lib/firmware/ti-connectivity
13732 lib/firmware/brcm
19960 lib/firmware/ath10kDo we need all of those on every build?
The only other really big one in lib was:
$ du -s lib/modules/6.1.106-linuxkit-06a737b0f212/* | sort -n
0 lib/modules/6.1.106-linuxkit-06a737b0f212/build
12 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.order
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep
36 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.bin
40 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep.bin
164 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols
192 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.modinfo
196 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols.bin
316 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias.bin
320 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias
8288 lib/modules/6.1.106-linuxkit-06a737b0f212/extra
55928 lib/modules/6.1.106-linuxkit-06a737b0f212/kernelMoving on to usr:
$ du -s -h usr/*
104M usr/bin
3.8M usr/include
14M usr/lib
388K usr/libexec
20K usr/local
2.0M usr/sbin
720K usr/shareOK, mostly usr/bin:
$ du -s usr/bin/* | sort -n
...
11648 usr/bin/containerd-shim-runc-v2
13168 usr/bin/runc
14252 usr/bin/service
20324 usr/bin/ctr
37128 usr/bin/containerdOnly really big things there are containerd, runc and service.
Last /containers:
$ du -s containers/*/* | sort -n
28 containers/services/pillar
1444 containers/onboot/000-rngd
1884 containers/onboot/001-sysctl
6764 containers/services/watchdog
8272 containers/onboot/006-measure-config
8612 containers/onboot/003-kdump
11460 containers/services/wlan
13140 containers/services/memory-monitor
18592 containers/services/newlogd
18648 containers/onboot/005-apparmor
21460 containers/services/guacd
24936 containers/services/edgeview
28708 containers/onboot/002-storage-init
45268 containers/services/wwan
53160 containers/services/vtpm
88840 containers/services/debug
97104 containers/services/xen-tools
137388 containers/onboot/004-pillar-onbootMost of them are pretty small. The last bunch start getting really big.
$ du -s containers/onboot/004-pillar-onboot/lower/* | sort -n
0 containers/onboot/004-pillar-onboot/lower/containers
4 containers/onboot/004-pillar-onboot/lower/dhcpcd.conf
4 containers/onboot/004-pillar-onboot/lower/fscrypt.conf
4 containers/onboot/004-pillar-onboot/lower/home
4 containers/onboot/004-pillar-onboot/lower/init.sh
4 containers/onboot/004-pillar-onboot/lower/mnt
4 containers/onboot/004-pillar-onboot/lower/proc
4 containers/onboot/004-pillar-onboot/lower/root
4 containers/onboot/004-pillar-onboot/lower/run
4 containers/onboot/004-pillar-onboot/lower/srv
4 containers/onboot/004-pillar-onboot/lower/sys
4 containers/onboot/004-pillar-onboot/lower/tmp
12 containers/onboot/004-pillar-onboot/lower/dev
16 containers/onboot/004-pillar-onboot/lower/media
104 containers/onboot/004-pillar-onboot/lower/var
1604 containers/onboot/004-pillar-onboot/lower/etc
2320 containers/onboot/004-pillar-onboot/lower/bin
5068 containers/onboot/004-pillar-onboot/lower/sbin
7076 containers/onboot/004-pillar-onboot/lower/lib
52460 containers/onboot/004-pillar-onboot/lower/usr
68652 containers/onboot/004-pillar-onboot/lower/optSo usr and opt.
$ du -s containers/onboot/004-pillar-onboot/lower/usr/bin/* | sort -n
...
1032 containers/onboot/004-pillar-onboot/lower/usr/bin/sgdisk
1100 containers/onboot/004-pillar-onboot/lower/usr/bin/coreutils
2044 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-io
2100 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-img
2220 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-nbd
3064 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-storage-daemonusr/lib is 174 files, almost all in the hundreds of KB range. Just the last few slightly larger ones:
$ du -s containers/onboot/004-pillar-onboot/lower/usr/lib/* | sort -n
...
852 containers/onboot/004-pillar-onboot/lower/usr/lib/libisoburn.so.1.111.0
1064 containers/onboot/004-pillar-onboot/lower/usr/lib/libglib-2.0.so.0.7200.4
1084 containers/onboot/004-pillar-onboot/lower/usr/lib/libp11-kit.so.0.3.0
1660 containers/onboot/004-pillar-onboot/lower/usr/lib/libunistring.so.2.2.0
1732 containers/onboot/004-pillar-onboot/lower/usr/lib/libgio-2.0.so.0.7200.4
1892 containers/onboot/004-pillar-onboot/lower/usr/lib/libgnutls.so.30.34.1
2008 containers/onboot/004-pillar-onboot/lower/usr/lib/xtables
4164 containers/onboot/004-pillar-onboot/lower/usr/lib/libzpool.so.5.0.0and:
$ du -s containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/* | sort -n
...
8 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/onboot.sh
20 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/device-steps.sh
392 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/dnsmasq
5288 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/fscrypt
62916 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/zedboxzedbox is big.
You can keep going for the other big ones, xen-tools and debug.
rene
commented
3 days ago FWIW, CDI can also be used to provide a common software stack base to all containers, we could have a directory with the whole common Alpine base files and mount it inside all containers through a CDI spec.... but considering our ecosystem, I still don't see any "easy/medium implementation efforts" for this kind of approach....
deitch
commented
3 days ago Yeah, it gets pretty complicated. We don't want solutions that make it even harder to build and run EVE and its components.
In pillar, zedbox is big, but so are usr and lib. Do we need everything there?
eriknordmark
commented
1 day ago Why is onboot container so large? For instance, do we need qemu* in onboot?
And why does onboot need zedbox??
deitch
commented
1 day ago It's the same container image filesystem as services. It's just reused.
christoph-zededa
commented
1 day ago Breakdown of zedbox done with https://github.com/Zxilly/go-size-analyzer
root@ae3e50cac946:/src/zedbox# gsa -f text zedbox | tee gsa.txt
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ zedbox │
├─────────┬─────────────────────────────────────────────────────────────────────────────┬────────┬───────────┤
│ PERCENT │ NAME │ SIZE │ TYPE │
├─────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼───────────┤
│ 14.87% │ .rodata │ 14 MB │ section │
│ 9.40% │ github.com/google/gopacket │ 8.7 MB │ vendor │
│ 7.35% │ .strtab │ 6.8 MB │ section │
│ 7.33% │ github.com/lf-edge/eve/pkg/pillar │ 6.8 MB │ vendor │
│ 6.66% │ .dynstr │ 6.2 MB │ section │
│ 6.21% │ .symtab │ 5.7 MB │ section │
│ 5.77% │ .debug_info │ 5.3 MB │ section │
│ 4.71% │ github.com/aws/aws-sdk-go │ 4.4 MB │ vendor │
│ 4.36% │ .dynsym │ 4.0 MB │ section │
│ 3.77% │ .debug_loc │ 3.5 MB │ section │
│ 3.15% │ .debug_line │ 2.9 MB │ section │
│ 2.21% │ github.com/Azure/azure-storage-blob-go │ 2.0 MB │ vendor │
│ 2.11% │ net │ 2.0 MB │ std │
│ 2.00% │ google.golang.org/protobuf │ 1.8 MB │ vendor │
│ 1.91% │ github.com/containerd/containerd │ 1.8 MB │ vendor │
│ 1.66% │ runtime │ 1.5 MB │ std │
│ 1.59% │ crypto │ 1.5 MB │ std │
│ 1.48% │ github.com/lf-edge/eve-api/go │ 1.4 MB │ vendor │
│ 1.30% │ github.com/gogo/protobuf │ 1.2 MB │ vendor │
│ 1.23% │ google.golang.org/grpc │ 1.1 MB │ vendor │
│ 1.16% │ github.com/robertkrimen/otto │ 1.1 MB │ vendor │
│ 1.13% │ golang.org/x/net │ 1.0 MB │ vendor │
│ 1.11% │ │ 1.0 MB │ generated │
│ 1.01% │ .gnu.hash │ 935 kB │ section │
│ 0.94% │ .debug_ranges │ 872 kB │ section │
│ 0.87% │ google.golang.org/api │ 802 kB │ vendor │
│ 0.76% │ github.com/miekg/dns │ 704 kB │ vendor │
│ 0.75% │ .debug_frame │ 693 kB │ section │
│ 0.73% │ cloud.google.com/go │ 677 kB │ vendor │
│ 0.72% │ github.com/lf-edge/eve-libs │ 671 kB │ vendor │
│ 0.53% │ golang.org/x/crypto │ 494 kB │ vendor │
│ 0.53% │ github.com/docker/docker │ 491 kB │ vendor │
│ 0.51% │ github.com/klauspost/compress │ 470 kB │ vendor │
│ 0.49% │ github.com/xeipuuv/gojsonschema │ 451 kB │ vendor │
│ 0.45% │ github.com/google/go-containerregistry │ 414 kB │ vendor │
│ 0.44% │ github.com/godbus/dbus/v5 │ 409 kB │ vendor │
│ 0.42% │ reflect │ 388 kB │ std │
│ 0.41% │ github.com/vishvananda/netlink │ 383 kB │ vendor │
│ 0.40% │ github.com/prometheus/client_golang │ 374 kB │ vendor │
│ 0.40% │ github.com/prometheus/procfs │ 368 kB │ vendor │
│ 0.38% │ text/template │ 354 kB │ std │
│ 0.38% │ golang.org/x/text │ 353 kB │ vendor │
│ 0.36% │ .gnu.version │ 336 kB │ section │
│ 0.36% │ math │ 330 kB │ std │
│ 0.35% │ github.com/containerd/cgroups │ 321 kB │ vendor │
│ 0.34% │ github.com/google/go-cmp │ 319 kB │ vendor │
│ 0.34% │ go.opentelemetry.io/otel │ 312 kB │ vendor │
│ 0.33% │ gopkg.in/yaml.v3 │ 308 kB │ vendor │
│ 0.29% │ gopkg.in/yaml.v2 │ 271 kB │ vendor │
│ 0.27% │ github.com/pkg/sftp │ 251 kB │ vendor │
│ 0.26% │ github.com/go-playground/validator/v10 │ 244 kB │ vendor │
│ 0.26% │ os │ 242 kB │ std │
│ 0.26% │ github.com/google/s2a-go │ 240 kB │ vendor │
│ 0.25% │ encoding/gob │ 233 kB │ std │
│ 0.25% │ .data │ 229 kB │ section │
│ 0.24% │ github.com/gabriel-vasile/mimetype │ 221 kB │ vendor │
│ 0.22% │ regexp │ 205 kB │ std │
│ 0.20% │ go.opencensus.io │ 188 kB │ vendor │
│ 0.20% │ github.com/shirou/gopsutil │ 184 kB │ vendor │
│ 0.19% │ google.golang.org/genproto │ 178 kB │ vendor │
│ 0.19% │ encoding/json │ 177 kB │ std │
│ 0.18% │ vendor/golang.org/x/text/unicode/norm │ 170 kB │ std │
│ 0.18% │ golang.org/x/oauth2 │ 166 kB │ vendor │
│ 0.18% │ time │ 164 kB │ std │
│ 0.17% │ github.com/andrewd-zededa/go-libzfs │ 159 kB │ vendor │
│ 0.16% │ github.com/ti-mo/conntrack │ 149 kB │ vendor │
│ 0.16% │ internal/profile │ 144 kB │ std │
│ 0.15% │ log │ 143 kB │ std │
│ 0.15% │ github.com/google/go-tpm │ 139 kB │ vendor │
│ 0.15% │ html │ 137 kB │ std │
│ 0.15% │ github.com/packetcap/go-pcap │ 136 kB │ vendor │
│ 0.15% │ tags.cncf.io/container-device-interface │ 135 kB │ vendor │
│ 0.14% │ encoding/xml │ 130 kB │ std │
│ 0.13% │ oras.land/oras-go │ 118 kB │ vendor │
│ 0.13% │ github.com/jmespath/go-jmespath │ 116 kB │ vendor │
│ 0.12% │ .noptrdata │ 112 kB │ section │
│ 0.11% │ fmt │ 105 kB │ std │
│ 0.11% │ syscall │ 99 kB │ std │
│ 0.11% │ archive/tar │ 98 kB │ std │
│ 0.11% │ internal/poll │ 98 kB │ std │
│ 0.10% │ vendor/golang.org/x/net/dns/dnsmessage │ 96 kB │ std │
│ 0.10% │ internal/abi │ 91 kB │ std │
│ 0.10% │ github.com/sirupsen/logrus │ 91 kB │ vendor │
│ 0.10% │ github.com/gorilla/websocket │ 88 kB │ vendor │
│ 0.09% │ github.com/lf-edge/edge-containers │ 85 kB │ vendor │
│ 0.09% │ .typelink │ 84 kB │ section │
│ 0.09% │ github.com/Azure/azure-pipeline-go │ 82 kB │ vendor │
│ 0.09% │ mime │ 82 kB │ std │
│ 0.08% │ compress/flate │ 77 kB │ std │
│ 0.08% │ encoding/asn1 │ 72 kB │ std │
│ 0.08% │ github.com/golang/protobuf │ 72 kB │ vendor │
│ 0.08% │ strconv │ 71 kB │ std │
│ 0.07% │ golang.org/x/sys │ 67 kB │ vendor │
│ 0.07% │ vendor/golang.org/x/net/idna │ 66 kB │ std │
│ 0.07% │ github.com/mdlayher/netlink │ 63 kB │ vendor │
│ 0.07% │ sync │ 61 kB │ std │
│ 0.06% │ github.com/go-chi/chi/v5 │ 57 kB │ vendor │
│ 0.06% │ github.com/go-logr/logr │ 57 kB │ vendor │
│ 0.06% │ internal/reflectlite │ 57 kB │ std │
│ 0.06% │ strings │ 56 kB │ std │
│ 0.06% │ io │ 55 kB │ std │
│ 0.06% │ vendor/golang.org/x/crypto/chacha20poly1305 │ 55 kB │ std │
│ 0.06% │ archive/zip │ 54 kB │ std │
│ 0.06% │ bufio │ 52 kB │ std │
│ 0.06% │ vendor/golang.org/x/crypto/cryptobyte │ 51 kB │ std │
│ 0.06% │ github.com/jaypipes/ghw │ 51 kB │ vendor │
│ 0.05% │ slices │ 46 kB │ std │
│ 0.05% │ github.com/docker/distribution │ 46 kB │ vendor │
│ 0.05% │ encoding/binary │ 46 kB │ std │
│ 0.05% │ github.com/linuxkit/linuxkit/src/cmd/linuxkit │ 45 kB │ vendor │
│ 0.05% │ github.com/mdlayher/socket │ 45 kB │ vendor │
│ 0.05% │ context │ 45 kB │ std │
│ 0.05% │ github.com/googleapis/gax-go/v2 │ 44 kB │ vendor │
│ 0.05% │ flag │ 43 kB │ std │
│ 0.05% │ github.com/distribution/reference │ 42 kB │ vendor │
│ 0.04% │ github.com/prometheus/client_model │ 39 kB │ vendor │
│ 0.04% │ vendor/golang.org/x/net/http2/hpack │ 38 kB │ std │
│ 0.04% │ bytes │ 38 kB │ std │
│ 0.04% │ unicode │ 34 kB │ std │
│ 0.04% │ github.com/facebook/time │ 34 kB │ vendor │
│ 0.04% │ sort │ 33 kB │ std │
│ 0.03% │ github.com/ti-mo/netfilter │ 32 kB │ vendor │
│ 0.03% │ github.com/go-logr/stdr │ 32 kB │ vendor │
│ 0.03% │ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp │ 31 kB │ vendor │
│ 0.03% │ github.com/tatsushid/go-fastping │ 30 kB │ vendor │
│ 0.03% │ github.com/docker/go-metrics │ 30 kB │ vendor │
│ 0.03% │ path │ 29 kB │ std │
│ 0.03% │ sigs.k8s.io/yaml │ 28 kB │ vendor │
│ 0.03% │ .itablink │ 28 kB │ section │
│ 0.03% │ github.com/grandcat/zeroconf │ 27 kB │ vendor │
│ 0.03% │ github.com/opencontainers/go-digest │ 24 kB │ vendor │
│ 0.03% │ github.com/google/uuid │ 24 kB │ vendor │
│ 0.02% │ vendor/golang.org/x/text/unicode/bidi │ 23 kB │ std │
│ 0.02% │ github.com/pkg/errors │ 23 kB │ vendor │
│ 0.02% │ github.com/jackwakefield/gopac │ 21 kB │ vendor │
│ 0.02% │ github.com/golang-jwt/jwt │ 21 kB │ vendor │
│ 0.02% │ github.com/satori/go.uuid │ 19 kB │ vendor │
│ 0.02% │ github.com/cshari-zededa/eve-tpm2-tools │ 19 kB │ vendor │
│ 0.02% │ github.com/containerd/ttrpc │ 19 kB │ vendor │
│ 0.02% │ github.com/docker/go-events │ 18 kB │ vendor │
│ 0.02% │ github.com/anatol/smart.go │ 18 kB │ vendor │
│ 0.02% │ encoding/base64 │ 18 kB │ std │
│ 0.02% │ .text │ 16 kB │ section │
│ 0.02% │ github.com/leodido/go-urn │ 16 kB │ vendor │
│ 0.02% │ github.com/containerd/fifo │ 16 kB │ vendor │
│ 0.02% │ github.com/fsnotify/fsnotify │ 16 kB │ vendor │
│ 0.02% │ internal/bisect │ 16 kB │ std │
│ 0.02% │ debug/dwarf │ 15 kB │ std │
│ 0.02% │ vendor/golang.org/x/net/http/httpproxy │ 15 kB │ std │
│ 0.02% │ text/tabwriter │ 15 kB │ std │
│ 0.01% │ compress/gzip │ 13 kB │ std │
│ 0.01% │ github.com/googleapis/enterprise-certificate-proxy │ 12 kB │ vendor │
│ 0.01% │ github.com/lf-edge/go-qemu │ 12 kB │ vendor │
│ 0.01% │ hash/crc32 │ 12 kB │ std │
│ 0.01% │ github.com/containerd/typeurl │ 12 kB │ vendor │
│ 0.01% │ github.com/moby/sys/mountinfo │ 11 kB │ vendor │
│ 0.01% │ encoding/csv │ 10 kB │ std │
│ 0.01% │ gopkg.in/sourcemap.v1 │ 10 kB │ vendor │
│ 0.01% │ encoding/base32 │ 10 kB │ std │
│ 0.01% │ main │ 10 kB │ main │
│ 0.01% │ internal/godebug │ 9.9 kB │ std │
│ 0.01% │ container/list │ 9.6 kB │ std │
│ 0.01% │ expvar │ 9.4 kB │ std │
│ 0.01% │ .debug_loclists │ 9.3 kB │ section │
│ 0.01% │ .debug_str │ 9.3 kB │ section │
│ 0.01% │ internal/fmtsort │ 9.1 kB │ std │
│ 0.01% │ golang.org/x/sync │ 9.0 kB │ vendor │
│ 0.01% │ .eh_frame │ 9.0 kB │ section │
│ 0.01% │ encoding/pem │ 8.8 kB │ std │
│ 0.01% │ github.com/tklauser/go-sysconf │ 8.6 kB │ vendor │
│ 0.01% │ internal/bytealg │ 8.4 kB │ std │
│ 0.01% │ github.com/docker/go-connections │ 8.0 kB │ vendor │
│ 0.01% │ github.com/moby/sys/user │ 8.0 kB │ vendor │
│ 0.01% │ vendor/golang.org/x/crypto/chacha20 │ 7.8 kB │ std │
│ 0.01% │ github.com/hashicorp/go-envparse │ 7.7 kB │ vendor │
│ 0.01% │ .go.buildinfo │ 7.3 kB │ section │
│ 0.01% │ github.com/vishvananda/netns │ 6.6 kB │ vendor │
│ 0.01% │ github.com/xeipuuv/gojsonpointer │ 6.4 kB │ vendor │
│ 0.01% │ go/token │ 6.0 kB │ std │
│ 0.01% │ vendor/golang.org/x/sys/cpu │ 5.9 kB │ std │
│ 0.01% │ internal/cpu │ 5.8 kB │ std │
│ 0.01% │ github.com/vbatts/tar-split │ 5.7 kB │ vendor │
│ 0.01% │ golang.org/x/mod │ 5.7 kB │ vendor │
│ 0.01% │ internal/lazyregexp │ 5.6 kB │ std │
│ 0.01% │ errors │ 5.2 kB │ std │
│ 0.01% │ internal/intern │ 5.0 kB │ std │
│ 0.01% │ github.com/containerd/continuity │ 4.9 kB │ vendor │
│ 0.01% │ internal/singleflight │ 4.7 kB │ std │
│ 0.00% │ vendor/golang.org/x/net/http/httpguts │ 4.5 kB │ std │
│ 0.00% │ github.com/golang/groupcache │ 4.4 kB │ vendor │
│ 0.00% │ github.com/docker/go-units │ 4.3 kB │ vendor │
│ 0.00% │ encoding/hex │ 4.2 kB │ std │
│ 0.00% │ github.com/syndtr/gocapability │ 4.0 kB │ vendor │
│ 0.00% │ internal/chacha8rand │ 3.6 kB │ std │
│ 0.00% │ .rela.plt │ 3.6 kB │ section │
│ 0.00% │ .debug_abbrev │ 3.5 kB │ section │
│ 0.00% │ github.com/eriknordmark/ipinfo │ 3.4 kB │ vendor │
│ 0.00% │ github.com/tklauser/numcpus │ 3.4 kB │ vendor │
│ 0.00% │ container/ring │ 3.3 kB │ std │
│ 0.00% │ go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc │ 3.2 kB │ vendor │
│ 0.00% │ github.com/cenkalti/backoff │ 3.2 kB │ vendor │
│ 0.00% │ vendor/golang.org/x/crypto/internal/poly1305 │ 2.8 kB │ std │
│ 0.00% │ internal/syscall/unix │ 2.7 kB │ std │
│ 0.00% │ github.com/xeipuuv/gojsonreference │ 2.7 kB │ vendor │
│ 0.00% │ github.com/kr/fs │ 2.7 kB │ vendor │
│ 0.00% │ github.com/moby/locker │ 2.6 kB │ vendor │
│ 0.00% │ github.com/eshard/uevent │ 2.5 kB │ vendor │
│ 0.00% │ testing │ 2.5 kB │ std │
│ 0.00% │ github.com/cespare/xxhash/v2 │ 2.4 kB │ vendor │
│ 0.00% │ .plt │ 2.4 kB │ section │
│ 0.00% │ vendor/golang.org/x/text/secure/bidirule │ 2.4 kB │ std │
│ 0.00% │ github.com/digitalocean/go-libvirt │ 2.3 kB │ vendor │
│ 0.00% │ internal/testlog │ 2.3 kB │ std │
│ 0.00% │ vendor/golang.org/x/crypto/hkdf │ 2.2 kB │ std │
│ 0.00% │ github.com/prometheus/common │ 2.1 kB │ vendor │
│ 0.00% │ github.com/containerd/log │ 2.0 kB │ vendor │
│ 0.00% │ github.com/opencontainers/runtime-tools │ 2.0 kB │ vendor │
│ 0.00% │ .eh_frame_hdr │ 1.9 kB │ section │
│ 0.00% │ github.com/opencontainers/selinux │ 1.7 kB │ vendor │
│ 0.00% │ github.com/lithammer/shortuuid/v4 │ 1.6 kB │ vendor │
│ 0.00% │ github.com/opencontainers/image-spec │ 1.4 kB │ vendor │
│ 0.00% │ github.com/moby/sys/signal │ 1.3 kB │ vendor │
│ 0.00% │ .got │ 1.3 kB │ section │
│ 0.00% │ github.com/go-playground/locales │ 1.3 kB │ vendor │
│ 0.00% │ github.com/golang-design/lockfree │ 1.2 kB │ vendor │
│ 0.00% │ CGO C11 │ 1.2 kB │ cgo │
│ 0.00% │ internal/saferio │ 1.1 kB │ std │
│ 0.00% │ internal/itoa │ 777 B │ std │
│ 0.00% │ .debug_line_str │ 659 B │ section │
│ 0.00% │ .dynamic │ 528 B │ section │
│ 0.00% │ .shstrtab │ 527 B │ section │
│ 0.00% │ github.com/opencontainers/runtime-spec │ 450 B │ vendor │
│ 0.00% │ .debug_aranges │ 426 B │ section │
│ 0.00% │ regexp.(*onePassInst).regexp/syntax │ 411 B │ vendor │
│ 0.00% │ github.com/ghodss/yaml │ 408 B │ vendor │
│ 0.00% │ .debug_rnglists │ 379 B │ section │
│ 0.00% │ github.com/containerd/stargz-snapshotter/estargz │ 370 B │ vendor │
│ 0.00% │ github.com/jaypipes/pcidb │ 306 B │ vendor │
│ 0.00% │ database/sql/driver │ 278 B │ std │
│ 0.00% │ github.com/docker/cli │ 273 B │ vendor │
│ 0.00% │ github.com/josharian/native │ 147 B │ vendor │
│ 0.00% │ .gnu.version_r │ 128 B │ section │
│ 0.00% │ .rela.dyn │ 120 B │ section │
│ 0.00% │ .note.go.buildid │ 100 B │ section │
│ 0.00% │ .debug_gdb_scripts │ 46 B │ section │
│ 0.00% │ .comment │ 38 B │ section │
│ 0.00% │ .note.gnu.build-id │ 36 B │ section │
│ 0.00% │ .note.ABI-tag │ 32 B │ section │
│ 0.00% │ .note.gnu.property │ 32 B │ section │
│ 0.00% │ .interp │ 28 B │ section │
│ 0.00% │ .init │ 27 B │ section │
│ 0.00% │ .fini │ 13 B │ section │
│ 0.00% │ .fini_array │ 8 B │ section │
│ 0.00% │ .init_array │ 8 B │ section │
│ 0.00% │ github.com/docker/docker-credential-helpers │ 0 B │ vendor │
│ 0.00% │ github.com/go-playground/universal-translator │ 0 B │ vendor │
│ 0.00% │ github.com/mattn/go-ieproxy │ 0 B │ vendor │
│ 0.00% │ github.com/gorilla/mux │ 0 B │ vendor │
│ 0.00% │ github.com/beorn7/perks │ 0 B │ vendor │
│ 0.00% │ github.com/coreos/go-systemd/v22 │ 0 B │ vendor │
│ 0.00% │ CGO Language(32769) │ 0 B │ cgo │
│ 0.00% │ github.com/felixge/httpsnoop │ 0 B │ vendor │
│ 0.00% │ golang.org/x/time │ 0 B │ vendor │
│ 0.00% │ github.com/mitchellh/go-homedir │ 0 B │ vendor │
│ 0.00% │ github.com/lf-edge/eve/pkg/kube/cnirpc │ 0 B │ vendor │
├─────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼───────────┤
│ 100.00% │ Known │ 92 MB │ │
│ 100% │ Total │ 92 MB │ │
└─────────┴─────────────────────────────────────────────────────────────────────────────┴────────┴───────────┘Have we tried building pillar while removing the reflect package? It is included in 27 files (will paste here below). One of the interesting pieces of golang optimization is that in general, it strips out everything that is unused, e.g. structs that are not referenced, etc. If you use reflect package, because it does not know and cannot know in advance what you will referenced, it keeps everything. We have seen cases where it made a compiled binary 30-50% bigger in size.
$ grep -r -w '"reflect"' * | grep -v vendor
cipher/cipher.go: "reflect"
cmd/usbmanager/subscriptions.go: "reflect"
cmd/tpmmgr/tpmmgr.go: "reflect"
cmd/zedagent/attesttask.go: "reflect"
cmd/domainmgr/domainmgr_test.go: "reflect"
cmd/zedmanager/handlezedrouter.go: "reflect"
containerd/oci_test.go: "reflect"
dpcmanager/dpc_test.go: "reflect"
dpcreconciler/genericitems/resolvconf.go: "reflect"
dpcreconciler/linuxitems/wlan.go: "reflect"
dpcreconciler/linuxitems/bond.go: "reflect"
evetpm/tpm_test.go: "reflect"
hypervisor/hypervisor_test.go: "reflect"
netmonitor/mock.go: "reflect"
objtonum/map.go: "reflect"
pubsub/large_test.go: "reflect"
pubsub/subscribe.go: "reflect"
pubsub/util.go: "reflect"
pubsub/pubsub.go: "reflect"
pubsub/publish.go: "reflect"
types/dns.go: "reflect"
types/assignableadapters.go: "reflect"
types/errortime_test.go: "reflect"
types/dpc.go: "reflect"
types/errortime.go: "reflect"
utils/deserialize_test.go: "reflect"
utils/deserialize.go: "reflect"Actually, that would be an interesting strategy. We should check for reflect in all of our packages. I would start with pillar and see what a difference it makes, but if we can get rid of it and makes a big difference, we can add "reflect alerts" to CI.
deitch
commented
5 hours ago @mikem-zed @rene @milan-zededa any thoughts on this comment, especially the part about halfway through it:
lib/firmware is 153MB. 202 lines in there, not going to post it all here. But most are .fw or .ucode files of 1.5-2MB, with a few really big ones:
$ du -s lib/firmware/* | sort -n ... 4236 lib/firmware/intel 6232 lib/firmware/cypress 8876 lib/firmware/display-t234-dce.bin 8996 lib/firmware/mrvl 9452 lib/firmware/ti-connectivity 13732 lib/firmware/brcm 19960 lib/firmware/ath10k Do we need all of those on every build?
christoph-zededa
commented
4 hours ago Actually, that would be an interesting strategy. We should check for
reflectin all of our packages. I would start with pillar and see what a difference it makes, but if we can get rid of it and makes a big difference, we can add "reflect alerts" to CI.
Doesn't json marshalling depend on reflect?
Good, that I just added more use of it ... (https://github.com/lf-edge/eve/pull/4295 )
deitch
commented
4 hours ago Maybe I'll try next week to build pillar without any reflect and see what happens.
Not sure about json marshalling. I don't think it depends on it, but not sure.
christoph-zededa
commented
3 hours ago Maybe I'll try next week to build pillar without any reflect and see what happens.
Not sure about json marshalling. I don't think it depends on it, but not sure.
287 func (e *encodeState) marshal(v any, opts encOpts) (err error) {
288 defer func() {
289 if r := recover(); r != nil {
290 if je, ok := r.(jsonError); ok {
291 err = je.error
292 } else {
293 panic(r)
294 }
295 }
296 }()
297 e.reflectValue(reflect.ValueOf(v), opts)
298 return nil
299 } json/encode.go
For TinyGo they do it f.e. like this: https://github.com/json-iterator/tinygo
EVE rootfs is close t 250Mb in size. The obvious option is to increase a partition size but first let's try to get rid of unused files/tools/packages and strip debug information from binaries