Add support or guidelines on installing the linux tools `perf` tool

[darach]

Is your feature request related to a problem? Please describe

First of all. We ❤️ GitPod at tremor. Tip of the hat to all involved!

One of the tools we use for performance analysis is the perf linux tool. It isn't obvious how to install this as usual incantations via apt-get fail, and identifying the correct variant via uname -r doesn't help.

Describe the behaviour you'd like

A minimal guide for how to get perf installed so perf stat, perf record can be run. Our project happens to be primarily rust based, but we sometimes need to debug native libraries such as librdkafka.

Describe alternatives you've considered

Alternatives include using dedicated hardware ( which we use for lengthy benchmark runs and automated benchmarking via equinix metal ). However, having this within a gitpod workspace would be useful to show new contributors to the project how to get started with benchmarking and used to performance engineering tools.

Additional context

The following incantation may illustrate:

sudo apt-get install -y linux-tools linux-tools-`uname -r`
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package linux-tools is a virtual package provided by:
  linux-tools-virtual-hwe-20.04-edge 5.13.0.27.29~20.04.13
  linux-tools-virtual-hwe-20.04 5.13.0.30.33~20.04.17
  linux-tools-virtual 5.4.0.100.104
  linux-tools-oracle-lts-20.04 5.4.0.1064.64
  linux-tools-oracle-edge 5.13.0.1018.22~20.04.1
  linux-tools-oracle 5.13.0.1018.22~20.04.1
  linux-tools-oem-20.04d 5.14.0.1024.22
  linux-tools-oem-20.04 5.14.0.1024.22
  linux-tools-lowlatency-hwe-20.04-edge 5.13.0.27.29~20.04.13
  linux-tools-lowlatency-hwe-20.04 5.13.0.30.33~20.04.17
  linux-tools-lowlatency 5.4.0.100.104
  linux-tools-intel 5.13.0.1009.10
  linux-tools-ibm-lts-20.04 5.4.0.1015.16
  linux-tools-ibm 5.4.0.1015.16
  linux-tools-gkeop-5.4 5.4.0.1034.37
  linux-tools-gkeop 5.4.0.1034.37
  linux-tools-gke-edge 5.13.0.1013.16~20.04.1
  linux-tools-gke-5.4 5.4.0.1063.73
  linux-tools-gke 5.4.0.1063.73
  linux-tools-generic-hwe-20.04-edge 5.13.0.27.29~20.04.13
  linux-tools-generic-hwe-20.04 5.13.0.30.33~20.04.17
  linux-tools-generic 5.4.0.100.104
  linux-tools-gcp-lts-20.04 5.4.0.1065.75
  linux-tools-gcp-edge 5.13.0.1015.18~20.04.2
  linux-tools-gcp 5.13.0.1015.18~20.04.2
  linux-tools-aws-lts-20.04 5.4.0.1066.68
  linux-tools-aws-edge 5.13.0.1014.15~20.04.7
  linux-tools-aws 5.13.0.1014.15~20.04.7
You should explicitly select one to install.

E: Package 'linux-tools' has no installation candidate
E: Unable to locate package linux-tools-5.16.14-051614-generic
E: Couldn't find any package by glob 'linux-tools-5.16.14-051614-generic'
E: Couldn't find any package by regex 'linux-tools-5.16.14-051614-generic'

And, the result of uname -r

uname -r
5.16.14-051614-generic

The perf tool needs to target the correct kernel version so this was a good place to stop where auto discovering same wasn't successful.

Also - learning heaps from your Docker repo too 👍

[aledbf]

@darach you can instal the packages from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.16.14/ Something like

apt install -y libc6 debconf

wget -c http://mirrors.kernel.org/ubuntu/pool/main/o/openssl/libssl3_3.0.1-0ubuntu1_amd64.deb
dpkg -i libssl3_3.0.1-0ubuntu1_amd64.deb

wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.16.14/amd64/linux-headers-5.16.14-051614-generic_5.16.14-051614.202203111231_amd64.deb
wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.16.14/amd64/linux-headers-5.16.14-051614_5.16.14-051614.202203111231_all.deb
wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.16.14/amd64/linux-image-unsigned-5.16.14-051614-generic_5.16.14-051614.202203111231_amd64.deb
wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.16.14/amd64/linux-modules-5.16.14-051614-generic_5.16.14-051614.202203111231_amd64.deb

dpkg -i linux*.deb

[darach]

@aledbf Kudos for the quick response.

apt-get install linux-tools-generic --fix-missing

This gets one step closer:

sudo /usr/lib/linux-tools/5.4.0-100-generic/perf record ls
perf_event_open(..., PERF_FLAG_FD_CLOEXEC) failed with unexpected error 1 (Operation not permitted)
perf_event_open(..., 0) failed unexpectedly with error 1 (Operation not permitted)
Error:
No permission to enable cycles event.

You may not have permission to collect stats.

Consider tweaking /proc/sys/kernel/perf_event_paranoid,
which controls use of the performance events system by
unprivileged users (without CAP_SYS_ADMIN).

The current value is 2:

  -1: Allow use of (almost) all events by all users
      Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
>= 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
      Disallow raw tracepoint access by users without CAP_SYS_ADMIN
>= 1: Disallow CPU event access by users without CAP_SYS_ADMIN
>= 2: Disallow kernel profiling by users without CAP_SYS_ADMIN

To make this setting permanent, edit /etc/sysctl.conf too, e.g.:

        kernel.perf_event_paranoid = -1

As /proc is of course read only.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[GitMensch]

There is still no direct option to install working perf tools (and the workaround above does not work; it would likely work to build and install the matching perf tools manually, allowing the installation of linux-tools-generic to works, but that's of course "last resort").

As soon as that is available people could use it do do several things, including well timing with perf stat; it would be even much more useful if the docker config is adjusted as outlined in #9687.

... and here's the last resort, put into a downloadable gist: https://gist.github.com/GitMensch/b0021e3cfd0957eeed6fbb9ce2973d7d

[LeoDog896]

This also means you cannot install linux-headers, making it virtually impossible to create virtual cameras in /dev under gitpod. (As well as other tools that may depend on this.)