Open dcathapermal opened 4 months ago
Can you share ldd cadvisor
output?
~/usr/bin# ldd cadvisor
linux-vdso.so.1 (0x00007ffdc295c000)
libpfm.so.4 => /lib/x86_64-linux-gnu/libpfm.so.4 (0x00007fe0483bd000)
libc.musl-x86_64.so.1 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe048194000)
/lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007fe04864d000)
Also just a note, this is an upstream of cadvisor so we didn't rebuild the binary with any flags or anything. But it seems that libpfm4 is already integrated in this version's upstream
~/usr/bin# ldd cadvisor linux-vdso.so.1 (0x00007ffdc295c000) libpfm.so.4 => /lib/x86_64-linux-gnu/libpfm.so.4 (0x00007fe0483bd000) libc.musl-x86_64.so.1 => not found libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe048194000) /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007fe04864d000)
Also just a note, this is an upstream of cadvisor so we didn't rebuild the binary with any flags or anything. But it seems that libpfm4 is already integrated in this version's upstream
https://github.com/google/cadvisor/blob/54dff2b8ccb147747d814b0ff3b4a3256dc569c0/perf/collector_libpfm.go#L269
This code cause the segment violation, not libpfm4 library. Can you check old version ldd
output and readelf -s xxx | grep free
output?
this is an ldd
of the cadvisor executable container running v0.47.2
docker exec 322ea45f166fb8533ca917817c2758cc444587381bce95b6af844bc431714552 ldd /usr/bin/cadvisor
/lib/ld-musl-x86_64.so.1 (0x7f5fae13a000)
libipmctl.so.4 => /usr/local/lib/libipmctl.so.4 (0x7f5faddd3000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f5fae13a000)
libndctl.so.6 => /usr/lib/libndctl.so.6 (0x7f5faddaa000)
libdaxctl.so.1 => /usr/lib/libdaxctl.so.1 (0x7f5fadd9d000)
libudev.so.1 => /lib/libudev.so.1 (0x7f5fadd7b000)
libuuid.so.1 => /lib/libuuid.so.1 (0x7f5fadd72000)
libkmod.so.2 => /lib/libkmod.so.2 (0x7f5fadd5b000)
libzstd.so.1 => /usr/lib/libzstd.so.1 (0x7f5fadcdd000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x7f5fadcba000)
libz.so.1 => /lib/libz.so.1 (0x7f5fadca0000)
libcrypto.so.1.1 => /lib/libcrypto.so.1.1 (0x7f5fada1e000)
I can't do a readelf because the container does not come with it installed, but you can see v0.47.2 doesn't have libpfm4as a dependency. So it gives the error 1 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
when trying to set the -perf_events_config
flag using this older version.
Do you know what kernel version the gcr.io/cadvisor/cadvisor:v0.49.1 image was built on? We have a suspicion that our 5.15.0-105-generic
is unable to support the perf counters if the binary was built on top of a 6.0 version''
EDIT: ^^This above issue is not the case. I tested the upstream gcr image on a 6.X machine and still have the same issue
~/usr/bin# ldd cadvisor linux-vdso.so.1 (0x00007ffdc295c000) libpfm.so.4 => /lib/x86_64-linux-gnu/libpfm.so.4 (0x00007fe0483bd000) libc.musl-x86_64.so.1 => not found libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe048194000) /lib/ld-musl-x86_64.so.1 => /lib64/ld-linux-x86-64.so.2 (0x00007fe04864d000)
Also just a note, this is an upstream of cadvisor so we didn't rebuild the binary with any flags or anything. But it seems that libpfm4 is already integrated in this version's upstream
I suspect the issue may be due to the use of the musl library not being found correctly, resulting in a SIGSEGV error.
Actually, I'm not sure which cadvisor binary I did ldd
on in the first comment I replied to, but I no longer see libc.musl-x86_64.so.1 => not found
when looking at the dependencies for v0.49.1. So I do not think it's a musl library issue..
/ # ldd /usr/bin/cadvisor
/lib/ld-musl-x86_64.so.1 (0x7fd0d860c000)
libpfm.so.4 => /usr/local/lib/libpfm.so.4 (0x7fd0d8337000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fd0d860c000)
/ #
This is based on the docker build here:
docker run -it --rm \
--volume=/etc/configs/perf/perf.json:/etc/configs/perf/perf.json \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--name=cadvisor-debug \
--privileged \
--entrypoint /bin/sh \
gcr.io/cadvisor/cadvisor:v0.49.1
Actually, I'm not sure which cadvisor binary I did
ldd
on in the first comment I replied to, but I no longer seelibc.musl-x86_64.so.1 => not found
when looking at the dependencies for v0.49.1. So I do not think it's a musl library issue../ # ldd /usr/bin/cadvisor /lib/ld-musl-x86_64.so.1 (0x7fd0d860c000) libpfm.so.4 => /usr/local/lib/libpfm.so.4 (0x7fd0d8337000) libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fd0d860c000) / #
This is based on the docker build here:
docker run -it --rm \ --volume=/etc/configs/perf/perf.json:/etc/configs/perf/perf.json \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --name=cadvisor-debug \ --privileged \ --entrypoint /bin/sh \ gcr.io/cadvisor/cadvisor:v0.49.1
I built cAdvisor on my local machine with glibc, and it ran successfully, so I suspect it may be a musl library link issue.
source build/config/libpfm4.sh
make build
ldd _output/cadvisor
#linux-vdso.so.1 (0x00007ffc316d2000)
#libpfm.so.4 => /usr/local/lib/libpfm.so.4 (0x000073cc9e000000)
#libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000073cc9dc00000)
#/lib64/ld-linux-x86-64.so.2 (0x000073cc9e4e8000)
Hi all,
Running with these specs:
I don't seem to have any issues spinning up a plain version of cadvisor v0.49.1.
However, when adding perf to the build
--perf_events_config=/etc/configs/perf/perf.json
, there seems to be a segfault errorIs this a known issue or bug with this version? I'm aware that older versions (v0.47.2 and before) do not come with perf built in, but it seems that v0.49.1 does, but it is unable to run?