[RFE] Add sysstat bashrc alias (done: bcc-tools)

dharmab commented 3 years ago

Current situation

There are a number of troubleshooting tools I commonly use on other Linux OSes from the sysstat and bcc-tools packages:

sar
iostat
vmstat
mpstat
pidstat
tcpretrans
tcpconnect
tcpaccept
biolatency
ext4slower
ext4dist

Currently, to run these tools you must use toolbox or run a privilged container in the global namespaces. This adds friction and delay to troubleshooting disk, network and other performance issues. In some cases, such as a disk is highly utilized/saturated, the toolbox or container image pull may fail.

Impact

It is harder to troubleshoot some types of resource saturation/utilization issues such as a saturated disk, saturated NIC or network degradation.

Ideal future situation

These tools could be installed in the base Flatcar OS, making them instantly available for troubleshooting.

**Implementation options

The sysstat package is a long-running project written in C. The binaries could therefore be compiled for Flatcar and included in the base OS.
bcc-tools are largely written as python 3 scripts that use the bcc and python-bcc libraries. Including these in the base Flatcar OS would require the inclusion of a python interpreter. Understandably, this may be outside the intended scope of Flatcar as an OS.

Additional information

SREcon presentation on how these tools can be used for troubleshooting

pothos commented 3 years ago

How about adding an alias entry for each tool to run the latest Docker container?

dharmab commented 3 years ago

That would be interesting- at the very least it would reduce the friction of trying to remember how to create a container in the necessary global namespaces.

pothos commented 3 years ago

There is an existing issue for this which also covers bpftrace: https://github.com/kinvolk/Flatcar/issues/198

pothos commented 3 years ago

This here is a working alias for execsnoop:

alias iovisor-execsnoop="docker run --rm -it -v /lib/modules:/lib/modules -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/fs/bpf:/sys/fs/bpf --privileged --net host --pid host quay.io/iovisor/bcc /usr/share/bcc/tools/execsnoop"

since for most commands is will be the same the alias value could be generated in a for loop.

Edit: For reference, this needs to go here: https://github.com/flatcar-linux/coreos-overlay/blob/main/app-shells/bash/files/dot-bashrc#L18

Edit2: Since there is no sysstat image a workaround could be this: docker run --rm -it --privileged --net host --pid host alpine sh -c 'apk -U add sysstat && pidstat' etc for the sysstat tools

aniruddha2000 commented 3 years ago

For the sysstat tools should we use ghcr.io for the images?

pothos commented 3 years ago

Aliases for some bcc tools were added in https://github.com/flatcar-linux/coreos-overlay/pull/1278 - for sysstat it's either waiting for a proper image or using the alpine workaround I pasted above

krishjainx commented 1 year ago

@pothos Would it be appropriate to include the aliases in https://github.com/flatcar/scripts/tree/main/sdk_container/src/third_party/coreos-overlay/app-shells/bash/files/dot-bashrc now that the coreos-overlay has been relocated to https://github.com/flatcar/scripts/tree/main/sdk_container/src/third_party? Thank you.

pothos commented 1 year ago

It would go into https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/misc-files/files/99-flatcar-bcc

krishjainx commented 1 year ago

Got it. Saw the changed location in this commit https://github.com/flatcar/scripts/commit/c2524a6d2dbd02a7aa94da281fb5b74b2d5690d1. Thanks

On Mon, Jun 5, 2023, 3:09 PM Kai Lüke @.***> wrote:

It would go into https://github.com/flatcar/scripts/blob/main/sdk_container/src/third_party/coreos-overlay/coreos-base/misc-files/files/99-flatcar-bcc

— Reply to this email directly, view it on GitHub https://github.com/flatcar/Flatcar/issues/498#issuecomment-1576773998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR4RDLOKMJCBA32RANE7DSLXJXLAVANCNFSM5DL6Y35Q . You are receiving this because you commented.Message ID: @.***>

krishjainx commented 1 year ago

@pothos Could we do the following?

for sysstat_tool in iostat sar vmstat mpstat pidstat; do
    alias "sysstat-${sysstat_tool}=docker run --rm -it -v /proc:/proc --privileged --net host --pid host ghcr.io/krishjainx/sysstat-docker:main /usr/bin/${sysstat_tool}"
done
unset -v sysstat_tool

pothos commented 1 year ago

Looks ok, we just would need to publish the container image to the flatcar org and it should be multiarch. We also need to save the Dockerfile somewhere… maybe the cleanest would be to create a new repository with a github action like https://github.com/flatcar/mantle/blob/flatcar-master/.github/workflows/docker.yml.

elafontaine commented 11 months ago

Hi, is there any traction on this ?

I would like to know as we are looking at doing some temporary setup that would use the sysstat to get some data for when we get an issue.

krishjainx commented 4 months ago

I am interested in this as well for what I'm working on: a sysstat container image published to the Flatcar organization. I maintain a multi-arch image here: https://github.com/krishjainx/sysstat-docker based on SUSE. Can we publish it to the Flatcar organization and have a GitHub repository for this? If this is acceptable, I can add aliases for it! cc/ @elafontaine @pothos @dongsupark

flatcar / Flatcar