gdb failure - "Operation not permitted."

[naseemkullah]

Hi, I cannot seem to get gdb to work with any pods. I get the following error. Any clue as to what might be going on? Please note pods run as non root (if that has anything to do with it).

(gdb) target remote localhost:54990
Remote debugging using localhost:54990
Reading /usr/bin/node from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
warning: "target:/usr/bin/node": could not open as an executable file: Operation not permitted.
Reading /usr/bin/node from remote target...
warning: `target:/usr/bin/node': can't open to read symbols: Operation not permitted.
warning: Could not load vsyscall page because no executable was specified

[mitchdraft]

Hi @naseemkullah it looks like you are debugging node.js with gdb, is that true?

[naseemkullah]

Hi @mitchdraft, in fact it is node js “wrapped” c++

[mitchdraft]

Interesting, I am not familiar with that paradigm. Could you provide an example? I'd like to give it a shot.

Are you able to use gdb on this target when running locally? If so, squash + gdb should work in cluster.

It's possible that you need to use a node js built with debug symbols. I don't think it's a non-root user issue, since squash-plank runs as root when opening a gdb connection: https://github.com/solo-io/squash/blob/master/cmd/plank/Dockerfile.gdb

[naseemkullah]

So an example of this paradigm would be: https://github.com/martinbark/nan-hello-world

I'd like to point out that after remove the securityContext of the workload, which included readOnlyFileSystem, the error dissapeared! https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#securitycontext-v1-core

Ideally we would be able to run the pods with the securityContext and still be able to use squash.

However I am now faced with the following error, any clues as to what may be causing this?

(gdb) target remote localhost:60009
Remote debugging using localhost:60009
warning: <threads>
<thread id="48c3a"/>
</threads>
: No such file or directory.
warning: Remote failure reply: E01

PS when you ask about running locally, do you mean the process running locally, or as a container or as a k8s workload on minikube?

[mitchdraft]

thanks for the link, that's really cool.

When I say "are you able to debug locally", I mean can you run the process on your computer (no kubernetes, minikube, or container) and debug it with gdb, also running locally. This is a good sanity check when testing squash in a new language/environment. If gdb works in that case, it should work through squash.

I'm glad you figured out why gdb couldn't open the file as an executable. I'm not sure if we can make it work while readOnlyFileSystem is in effect. It would be interesting to investigate.

For your new error, I don't think you need to use the remote commands. The way squash works is to run gdb on the same node as your target process and then attach directly. You should not need to specify a target once gdb is running. Could you try a different gdb command, something like list instead?

[naseemkullah]

Np for the link!

Ah I see now with regards to locally. Yes gdb does work with process running locally.

Yes or it would be nice if we could have a RO filesystem and juste have maybe /usr/bin/node as Read-Execute somehow, maybe securityContext's will evolve to permit this.

With regards to current error, just to be clear I get the following message when starting up squash:

Squash will create a debugger pod in your target pod's namespace.
Creating service account squash-plank in namespace squash-debugger

Creating cluster role squash-plank-cr

Creating cluster role binding squash-plank-crb

All squashctl permission resources created.

Pod creating
Forwarding from 127.0.0.1:52429 -> 33171
Forwarding from [::1]:52429 -> 33171
gdb debug port available on local port 52429.

If I just type list as a command I get the following:

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) list
No symbol table is loaded.  Use the "file" command.
(gdb)

[naseemkullah]

So I think using a remote target of localhost with the forwarded port is the right idea as per: https://github.com/solo-io/squash/blob/master/next_docs.md

Just guessing but might it have something to do with my gdb running on macOS and the container being Ubuntu based?

[mitchdraft]

ah yes, sorry, you were write about the invocation. I need to move those docs into the website.

I think you just need to tell gdb where to find the symbol table now. You should be able to do that with directory. You may also need to specify a substitute-path. Here's a link to the man pages https://sourceware.org/gdb/onlinedocs/gdb/Source-Path.html

I have used squash and gdb from mac to ubuntu containers without problems in the past.