search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.62k stars 1.76k forks source link

BPF Session Recording fails on Linux 5.4.74 #5081

Open benarent opened 3 years ago

benarent commented 3 years ago

Description

What happened:

While Testing Cloud, I created a new instance and tried to setup BPF Session Recording. It didn't connect, it looks like it failed due to a kernel mis-match. 

Dec 09 06:20:53 ip-172-31-45-76.us-west-2.compute.internal teleport[3647]: INFO [PROC]      Connecting to the cluster asteroid-sun.teleport.sh with TLS client certificate. service/connect.go:127
Dec 09 06:20:53 ip-172-31-45-76.us-west-2.compute.internal teleport[3647]: INFO [NODE:1:CA] Cache "node" first init succeeded. cache/cache.go:574
Dec 09 06:20:53 ip-172-31-45-76.us-west-2.compute.internal teleport[3647]: WARN [PROC:1]    Teleport process has exited with error: incompatible kernel found, minimum supported kernel is 4.18.0 service:ssh.node service/supervisor.go:247
Dec 09 06:20:53 ip-172-31-45-76.us-west-2.compute.internal teleport[3647]: ERRO [PROC:1]    Critical service ssh.node has exited with error incompatible kernel found, minimum supported kernel is 4.18.0, aborting. service/signals.go:145
Dec 09 06:20:53 ip-172-31-45-76.us-west-2.compute.internal teleport[3647]: error: incompatible kernel found, minimum supported kernel is 4.18.0
Dec 09 06:21:01 ip-172-31-45-76.us-west-2.compute.internal systemd[1]: Stopping Teleport SSH Service...
Dec 09 06:22:31 ip-172-31-45-76.us-west-2.compute.internal systemd[1]: teleport.service stop-sigterm timed out. Killing.
Dec 09 06:22:31 ip-172-31-45-76.us-west-2.compute.internal systemd[1]: teleport.service: main process exited, code=killed, status=9/KILL
lines 1-34
Linux ip-172-31-45-76.us-west-2.compute.internal 5.4.74-36.135.amzn2.x86_64 #1 SMP Wed Nov 4 17:56:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
hoenirvili commented 3 years ago

My hunch is that AWS modifies the naming scheme in /proc/sys/kernel/osrelease totally different than what we are expecting. That's why maybe it fails to compare the versions correctly even if we explicitly require no kernel less than 4.18.0.

Can you run this command inside the instance where teleport runs? (and paste here the output?)

cat /proc/sys/kernel/osrelease

Or if you can't do this, I would happily spin an aws instance myself (If I can with the free account) and test it myself. Can you add more details like, what type of instance you used?

hoenirvili commented 3 years ago

Normally the cat /proc/sys/kernel/osrelease should return something like 5.4.74-36.135.amzn2.x86_64 which indeed the if version.LessThan(*minKernel) (here) will be always false but maybe is something inconsistent.

It would really help to also include in this trace the other kernel version parsed.

aaronmell commented 2 years ago

Ran into this same issue using amazon linux 2 AMI's. Mine returns, however the error specifies a minimum kernel of

cat /proc/sys/kernel/osrelease 5.4.204-113.362.amzn2.x86_64


riginal Error: *trace.BadParameterError incompatible kernel found, minimum supported kernel is 5.8.0
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/bpf/common.go:151 github.com/gravitational/teleport/lib/bpf.IsHostCompatible
        /go/src/github.com/gravitational/teleport/lib/bpf/bpf.go:129 github.com/gravitational/teleport/lib/bpf.New
        /go/src/github.com/gravitational/teleport/lib/service/service.go:1928 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSSH.func1
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:521 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:269 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: incompatible kernel found, minimum supported kernel is 5.8.0] service:ssh.node service/supervisor.go:274