search

facebookarchive / memory-analyzer

A memory analyzer for running python processes
MIT License
154 stars 14 forks source link

No symbol table is loaded. Use the "file" command - even with `root` user #20

Closed harit-h closed 3 years ago

harit-h commented 3 years ago

Hello,

My use case matches the recommendation as per the documentation, and I followed that as well.

The different in my situation (maybe) is that I am running this inside a docker container. The necessary dependencies are installed in Docker container

Step 8/9 : RUN pip install click attrs jinja2 prettytable pympler objgraph memory-analyzer

The process I am trying to debug (7) is listed below

root@005bbfb16790:/app# ps -aef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 10:41 pts/0    00:00:00 /bin/sh -c /usr/local/bin/runpython -m src.main
root         7     1  0 10:41 pts/0    00:00:15 python -m src.main # <-- This one
root        13     0  0 10:41 pts/1    00:00:00 bash
root        59     0  0 16:23 pts/2    00:00:00 bash
root       648    59  0 18:43 pts/2    00:00:00 ps -aef
root@005bbfb16790:/app#

When I run the command, I get error

root@005bbfb16790:/app# memory_analyzer run 7 --debug
Analyzing pid 7
Setting up GDB for pid 7
/usr/local/bin/python-dbg: No such file or directory.
ptrace: Operation not permitted.
+set directories /usr/local/lib/python3.6/site-packages/memory_analyzer/
+py sys.path.append("/usr/local/lib/python3.6/site-packages/memory_analyzer/")
+file_command /tmp/tmp8bp84tt7/rendered_template-0.py.out
Python Exception <class 'gdb.error'> No symbol table is loaded.  Use the "file" command.: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/memory_analyzer//gdb_commands.py", line 47, in <module>
    TEMPLATES_PATH=TEMPLATES_PATH, pid=pid
gdb.error: Error occurred in Python command: No symbol table is loaded.  Use the "file" command.
Gathering data...
ERROR: Analysis for 7 returned no data!  Try rerunning with --debug
ERROR: No results to report

I checked that I am running it as a root user

root@005bbfb16790:/app# id
uid=0(root) gid=0(root) groups=0(root)

I am using Python 3 as shown here

root@005bbfb16790:/app# python --version
Python 3.6.10
root@005bbfb16790:/app#

and I have python3-dbg installed as well

root@005bbfb16790:/app# find / -name "python3-dbg"
/usr/bin/python3-dbg
/usr/share/doc/python3-dbg

Your help is greatly appreciated. Thank you

harit-h commented 3 years ago

As a next step, I did the following

root@032164243d89:/app# ln -s /usr/bin/python3-dbg  /usr/local/bin/python-dbg

and ran the process again, and get a different error

root@032164243d89:/app# memory_analyzer run 6 --debug
Analyzing pid 6
Setting up GDB for pid 6
ptrace: Operation not permitted.
+set directories /usr/local/lib/python3.6/site-packages/memory_analyzer/
+py sys.path.append("/usr/local/lib/python3.6/site-packages/memory_analyzer/")
+file_command /tmp/tmpdpbn23s7/rendered_template-0.py.out
Python Exception <class 'gdb.error'> You can't do that without a process to debug.: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/memory_analyzer//gdb_commands.py", line 47, in <module>
    TEMPLATES_PATH=TEMPLATES_PATH, pid=pid
gdb.error: Error occurred in Python command: You can't do that without a process to debug.
Gathering data...
ERROR: Analysis for 6 returned no data!  Try rerunning with --debug
ERROR: No results to report
thatch commented 3 years ago

ptrace: Operation not permitted.

is a fatal error; are you doing syscall filtering perhaps? Try something like gdb $(which python) -p 6 and see if you can attach outside of memory_analyzer at all.

harit-h commented 3 years ago

Hi @thatch, I am not doing syscall filtering at all. I tried what you said and it looks like

root@2415b0a98595:/app# gdb $(which python) -p 6
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/python...done.
Attaching to program: /usr/local/bin/python, process 6
ptrace: No such process.
(gdb)

Now, I am not sue what to do next. So, I did the following

Is my understanding correct? Can I use memory_analyzer inside (gdb) shell? Kindly advice

thatch commented 3 years ago

memory_analyzer is basically scripting gdb, so yes you can... but it's probably better to figure out the ptrace problem.

It looks like your pid is 7 now (the "python -m src.main"); and I gave you a bad gdb command line, the -p <num> needs to come first. Try gdb -p 7 $(which python)

harit-h commented 3 years ago

No worries, @thatch, you've been a great person, I am learning a lot this way. With the updated command, I see

root@2415b0a98595:/app# ps -aef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Dec04 pts/0    00:00:00 /bin/sh -c /usr/local/bin/runpython -m src.main
root         7     1  0 Dec04 pts/0    00:00:04 python -m src.main
root        13     0  0 Dec04 pts/1    00:00:00 bash
root        24     0  0 Dec04 pts/2    00:00:00 bash
root        59    13  0 Dec04 pts/1    00:00:00 gdb /usr/local/bin/python -p 6
root        78     0  1 00:32 pts/3    00:00:00 bash
root        83    78  0 00:32 pts/3    00:00:00 ps -aef
root@2415b0a98595:/app# gdb -p 7 $(which python)
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/python...done.
Attaching to program: /usr/local/bin/python, process 7
ptrace: Operation not permitted.
(gdb)

However, I still see ptrace: Operation not permitted.. What do you think should be my next step? Sorry for asking, because I don't know much about my next steps. Thanks

harit-h commented 3 years ago

Hello @thatch, I wanted to update you that it is now resolved. I fiddled on google and found this stackoverflow answer. With this, I modified my docker-compose.yml file, to add the following snippet under services: section

    cap_add:
      - SYS_PTRACE

Then, I restarted the process to recreate the container using the following command

docker-compose up --build

With this, when I run the following command, I am now able to see the screen

root@08c55b23e820:/app# memory_analyzer run 7
Screen Shot 2020-12-04 at 4 49 47 PM
harit-h commented 3 years ago

I would like to explicitly call out @thatch for your help on this issue. THANK YOU!