Python interpreter has debug symbols but libpython.so doesn't

[manopapad]

I am trying to debug a C++ application that embeds a python interpreter, and would like to use gdb's python extensions to interpret python frames (py-bt, py-up etc.). The version of gdb I get from conda-forge (conda install gdb) is already properly configured to do this, and works for scripts executed in a standalone python interpreter. However, it doesn't work for applications that invoke the python interpreter through libpython.so, and this is likely because the version of libpython.so I get from conda-forge does not seem to contain debug symbols (see command log below). Is it possible to update the recipes to include debug symbols for libpython.so as well?

Note that I can get debug symbols on both the standalone and embedded interpreter cases if I use the debug builds from Ubuntu package python-dbg.

Commands to reproduce:

``` prm-login:~> conda create --name temp python=3.7 Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /gpfs/fs1/mpapadakis/anaconda3/envs/temp added / updated specs: - python=3.7 The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-0_gnu ca-certificates conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0 certifi conda-forge/linux-64::certifi-2020.6.20-py37hc8dfbb8_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.34-h53a641e_7 libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007 libgcc-ng conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2 libgomp conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2 libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2 ncurses conda-forge/linux-64::ncurses-6.1-hf484d3e_1002 openssl conda-forge/linux-64::openssl-1.1.1g-h516909a_0 pip conda-forge/noarch::pip-20.1.1-py_1 python conda-forge/linux-64::python-3.7.6-cpython_h8356626_6 python_abi conda-forge/linux-64::python_abi-3.7-1_cp37m readline conda-forge/linux-64::readline-8.0-h46ee950_1 setuptools conda-forge/linux-64::setuptools-49.2.0-py37hc8dfbb8_0 sqlite conda-forge/linux-64::sqlite-3.32.3-hcee41ef_1 tk conda-forge/linux-64::tk-8.6.10-hed695b0_0 wheel conda-forge/noarch::wheel-0.34.2-py_1 xz conda-forge/linux-64::xz-5.2.5-h516909a_1 zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1006 Proceed ([y]/n)? y Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate temp # # To deactivate an active environment, use # # $ conda deactivate prm-login:~> conda activate temp prm-login:~> cat symtest.py import time def foo(x): # give us enough time to Ctrl-C time.sleep(60) return x def bar(x): return foo(x+1) def qux(x): return bar(x+1) def main(): x = qux(0) print("done") return x if __name__ == "__main__": main() prm-login:~> cat call.c #define PY_SSIZE_T_CLEAN #include int main(int argc, char *argv[]) { PyObject *pName, *pModule, *pFunc; PyObject *pArgs, *pValue; int i; if (argc < 3) { fprintf(stderr,"Usage: call pythonfile funcname [args]\n"); return 1; } Py_Initialize(); pName = PyUnicode_DecodeFSDefault(argv[1]); /* Error checking of pName left out */ pModule = PyImport_Import(pName); Py_DECREF(pName); if (pModule != NULL) { pFunc = PyObject_GetAttrString(pModule, argv[2]); /* pFunc is a new reference */ if (pFunc && PyCallable_Check(pFunc)) { pArgs = PyTuple_New(argc - 3); for (i = 0; i < argc - 3; ++i) { pValue = PyLong_FromLong(atoi(argv[i + 3])); if (!pValue) { Py_DECREF(pArgs); Py_DECREF(pModule); fprintf(stderr, "Cannot convert argument\n"); return 1; } /* pValue reference stolen here: */ PyTuple_SetItem(pArgs, i, pValue); } pValue = PyObject_CallObject(pFunc, pArgs); Py_DECREF(pArgs); if (pValue != NULL) { printf("Result of call: %ld\n", PyLong_AsLong(pValue)); Py_DECREF(pValue); } else { Py_DECREF(pFunc); Py_DECREF(pModule); PyErr_Print(); fprintf(stderr,"Call failed\n"); return 1; } } else { if (PyErr_Occurred()) PyErr_Print(); fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]); } Py_XDECREF(pFunc); Py_DECREF(pModule); } else { PyErr_Print(); fprintf(stderr, "Failed to load \"%s\"\n", argv[1]); return 1; } if (Py_FinalizeEx() < 0) { return 120; } return 0; } prm-login:~> gcc -o call call.c -I$CONDA_PREFIX/include/python3.7m -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -ffunction-sections -pipe -isystem $CONDA_PREFIX/include -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto -fwrapv -g -O0 -L$CONDA_PREFIX/lib -l:libpython3.7m.so -lcrypt -lpthread -ldl -lutil -lrt -lm prm-login:~> gdb --args python symtest.py GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from python...done. (gdb) r Starting program: /gpfs/fs1/mpapadakis/anaconda3/envs/temp/bin/python symtest.py [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". ^C Program received signal SIGINT, Interrupt. 0x00007ffff78dc0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2c80) at ../sysdeps/unix/sysv/linux/select.c:41 41 ../sysdeps/unix/sysv/linux/select.c: No such file or directory. (gdb) bt #0 0x00007ffff78dc0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2c80) at ../sysdeps/unix/sysv/linux/select.c:41 #1 0x00005555556bee63 in pysleep (secs=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/timemodule.c:1843 #2 time_sleep () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/timemodule.c:371 #3 0x000055555569d72d in _PyMethodDef_RawFastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:648 #4 0x00005555556d55d0 in _PyCFunction_FastCallKeywords (kwnames=, nargs=, args=0x7ffff6c1efd0, func=0x7ffff6b3a8c0) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:734 #5 call_function () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4568 #6 0x000055555571b762 in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3093 #7 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6c1ee50) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547 #8 function_code_fastcall (globals=, nargs=, args=, co=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283 #9 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408 #10 0x00005555557172b5 in call_function (kwnames=0x0, oparg=, pp_stack=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616 #11 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124 #12 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6b66050) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547 #13 function_code_fastcall (globals=, nargs=, args=, co=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283 #14 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408 #15 0x00005555557172b5 in call_function (kwnames=0x0, oparg=, pp_stack=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616 #16 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124 #17 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6c20a00) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547 #18 function_code_fastcall (globals=, nargs=, args=, co=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283 #19 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408 #20 0x00005555557172b5 in call_function (kwnames=0x0, oparg=, pp_stack=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616 #21 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124 #22 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x555555926c80) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547 #23 function_code_fastcall (globals=, nargs=, args=, co=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283 #24 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408 #25 0x00005555557172b5 in call_function (kwnames=0x0, oparg=, pp_stack=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616 #26 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124 #27 0x000055555566d505 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6b74450) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547 #28 _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3930 #29 0x000055555566e8f3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=, globals=, _co=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3959 #30 PyEval_EvalCode (co=, globals=, locals=) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:524 #31 0x000055555577b692 in run_mod () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:1035 #32 0x00005555557857be in PyRun_FileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:988 #33 0x00005555557859ab in PyRun_SimpleFileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:429 #34 0x0000555555786a61 in pymain_run_file (p_cf=0x7fffffff3730, filename=0x5555558be540 L"symtest.py", fp=0x555555924de0) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:428 #35 pymain_run_filename (cf=0x7fffffff3730, pymain=0x7fffffff3840) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:1607 #36 pymain_run_python (pymain=0x7fffffff3840) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:2868 #37 pymain_main () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:3029 #38 0x0000555555786aec in _Py_UnixMain () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:3064 #39 0x00007ffff77e6b97 in __libc_start_main (main=0x555555645ab0

, argc=2, argv=0x7fffffff3998, init=, fini=, rtld_fini=, stack_end=0x7fffffff3988) at ../csu/libc-start.c:310 #40 0x000055555572b73d in _start () at ../sysdeps/x86_64/elf/start.S:103 (gdb) quit A debugging session is active. Inferior 1 [process 7007] will be killed. Quit anyway? (y or n) y prm-login:~> PYTHONPATH=. gdb --args ./call symtest main GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./call...done. (gdb) r Starting program: /home/mpapadakis/call symtest main [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". ^C Program received signal SIGINT, Interrupt. 0x00007ffff698e0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2fc0) at ../sysdeps/unix/sysv/linux/select.c:41 41 ../sysdeps/unix/sysv/linux/select.c: No such file or directory. (gdb) bt #0 0x00007ffff698e0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2fc0) at ../sysdeps/unix/sysv/linux/select.c:41 #1 0x00007ffff7b7997a in time_sleep () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #2 0x00007ffff7bb1b5c in _PyMethodDef_RawFastCallKeywords () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #3 0x00007ffff7bb2606 in _PyCFunction_FastCallKeywords () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #4 0x00007ffff7adb3ff in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #5 0x00007ffff7ad910a in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #6 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #7 0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #8 0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #9 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #10 0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #11 0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #12 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #13 0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #14 0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #15 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #16 0x00007ffff7bb248f in _PyFunction_FastCallDict () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0 #17 0x00000000004010e3 in main (argc=3, argv=0x7fffffff39b8) at call.c:40 (gdb) quit A debugging session is active. Inferior 1 [process 7286] will be killed. Quit anyway? (y or n) y ```

Environment (conda list):

``` $ conda list # packages in environment at /gpfs/fs1/mpapadakis/anaconda3/envs/temp: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 0_gnu conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge certifi 2020.6.20 py37hc8dfbb8_0 conda-forge ld_impl_linux-64 2.34 h53a641e_7 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgomp 9.2.0 h24d8f2e_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge openssl 1.1.1g h516909a_0 conda-forge pip 20.1.1 py_1 conda-forge python 3.7.6 cpython_h8356626_6 conda-forge python_abi 3.7 1_cp37m conda-forge readline 8.0 h46ee950_1 conda-forge setuptools 49.2.0 py37hc8dfbb8_0 conda-forge sqlite 3.32.3 hcee41ef_1 conda-forge tk 8.6.10 hed695b0_0 conda-forge wheel 0.34.2 py_1 conda-forge xz 5.2.5 h516909a_1 conda-forge zlib 1.2.11 h516909a_1006 conda-forge ```

Details about conda and system ( conda info ):

``` $ conda info active environment : temp active env location : /gpfs/fs1/mpapadakis/anaconda3/envs/temp shell level : 3 user config file : /home/mpapadakis/.condarc populated config files : /home/mpapadakis/.condarc conda version : 4.8.3 conda-build version : 3.18.9 python version : 3.7.4.final.0 virtual packages : __glibc=2.27 base environment : /gpfs/fs1/mpapadakis/anaconda3 (writable) channel URLs : https://conda.anaconda.org/rapidsai-nightly/linux-64 https://conda.anaconda.org/rapidsai-nightly/noarch https://conda.anaconda.org/nvidia/linux-64 https://conda.anaconda.org/nvidia/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /gpfs/fs1/mpapadakis/anaconda3/pkgs /home/mpapadakis/.conda/pkgs envs directories : /gpfs/fs1/mpapadakis/anaconda3/envs /home/mpapadakis/.conda/envs platform : linux-64 user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-62-generic ubuntu/18.04.3 glibc/2.27 UID:GID : 38092:30 netrc file : None offline mode : False ```

[mingwandroid]

We don't provide debug symbols as they tend to be large and most people do not need them.

You should build the interpreter yourself. The recipe has tweak parameters for making debug builds but this stuff is new and seldom used.

I would support and effort to split debug info into separate packages for important packages though but we don't really have the tooling in place for that.

[SelvamArul]

@mingwandroid I am trying to build the python interpreter myself but build_locally.py does not support building natively on ubuntu. Could you please suggest some resources related to building the python interpreter on ubuntu?

[nikhilweee]

@mingwandroid I'm in the same boat and would like to have debug symbol support as well. Can you point us to any resources to help build the package locally?