search

conda-forge / gdb-feedstock

A conda-smithy repository for gdb.
BSD 3-Clause "New" or "Revised" License
17 stars 62 forks source link

Conflict with Python 3.12 #66

Open pitrou opened 1 year ago

pitrou commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

gdb and Python 3.12 cannot be installed together currently. Excerpt from our CI:

#9 [3/5] RUN mamba install -q -y         --file arrow/ci/conda_env_python.txt         gdb         python=3.12         nomkl &&     mamba clean --all
#9 24.98 Could not solve for environment specs
#9 24.98 The following packages are incompatible
#9 24.98 ├─ gdb   is installable with the potential options
#9 24.98 │  ├─ gdb [10.2|9.2] would require
#9 24.98 │  │  └─ python_abi 3.6.* *_cp36m, which can be installed;
#9 24.98 │  ├─ gdb [10.2|11.1|11.2|9.2] would require
#9 24.98 │  │  └─ python_abi 3.7.* *_cp37m, which can be installed;
#9 24.98 │  ├─ gdb [10.2|11.1|11.2|12.1|9.2] would require
#9 24.98 │  │  └─ python_abi 3.8.* *_cp38, which can be installed;
#9 24.98 │  ├─ gdb [10.2|11.1|11.2|12.1] would require
#9 24.98 │  │  └─ python_abi 3.9.* *_cp39, which can be installed;
#9 24.98 │  ├─ gdb [11.1|11.2|12.1] would require
#9 24.98 │  │  └─ python_abi 3.10.* *_cp310, which can be installed;
#9 24.98 │  ├─ gdb [11.2|12.1] would require
#9 24.98 │  │  └─ python_abi 3.11.* *_cp311, which can be installed;
#9 24.98 │  ├─ gdb [7.11|7.12|...|8.3.1] would require
#9 24.98 │  │  └─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
#9 24.98 │  ├─ gdb [7.11|7.12] would require
#9 24.98 │  │  └─ python 3.4* , which can be installed;
#9 24.98 │  ├─ gdb [7.11|7.12|...|8.2] would require
#9 24.98 │  │  └─ python [3.5* |>=3.5,<3.6.0a0 ], which can be installed;
#9 24.98 │  ├─ gdb [7.12.1|8.0|8.1] would require
#9 24.98 │  │  └─ python 3.6* , which can be installed;
#9 24.98 │  ├─ gdb [8.1|8.2|8.3|8.3.1] would require
#9 24.98 │  │  └─ python >=3.6,<3.7.0a0 , which can be installed;
#9 24.98 │  ├─ gdb [8.2|8.3|8.3.1] would require
#9 24.98 │  │  └─ python >=3.7,<3.8.0a0 , which can be installed;
#9 24.98 │  └─ gdb 8.3.1 would require
#9 24.98 │     └─ python >=3.8,<3.9.0a0 , which can be installed;
#9 24.98 └─ python 3.12**  is uninstallable because there are no viable options
#9 24.98    ├─ python 3.12.0 would require
#9 24.98    │  └─ python_abi 3.12.* *_cp312, which conflicts with any installable versions previously reported;
#9 24.98    └─ python 3.12.0rc3 would require
#9 24.98       └─ _python_rc  , which does not exist (perhaps a missing channel).

Installed packages

Unavailable (this is on CI), but I don't think it matters here.

Environment info

Unavailable (this is on CI), but I don't think it matters here.
pitrou commented 1 year ago

By the way, gdb could probably use its own private Python interpreter (which needn't be the same as the Python being tested). The main downside would be a bigger disk footprint when both are installed. However, it would probably be much better for people who want only gdb and not Python

pitrou commented 1 year ago

Sidenote: it doesn't feel right that libpython.py is bundled with gdb. libpython.py is meant to debug a Python interpreter running inside gdb, so it should really be part of the Python interpreter package, not of the gdb package.

phil-blain commented 8 months ago

Hi @pitrou !

Sidenote: it doesn't feel right that libpython.py is bundled with gdb. libpython.py is meant to debug a Python interpreter running inside gdb, so it should really be part of the Python interpreter package, not of the gdb package.

The GDB feedstock was created specially for making it easy to debug a Python interpreter, that is why libpython.py is bundled in the feedstock, such that it works outside the box. See 7c7fa9a and 881360d.

Looking at CPython's Makefile, I see that the build does install libpython.py as python-gdb.py and so in theory it should work outside the box correctly, if the installed GDB has Python support. I think this was not a given when the GDB feedstock was created, which was one of the reason for its creation.

phil-blain commented 8 months ago

By the way, gdb could probably use its own private Python interpreter (which needn't be the same as the Python being tested). The main downside would be a bigger disk footprint when both are installed. However, it would probably be much better for people who want only gdb and not Python

I think for this conda-forge package, it would only be possible if we built GDB with a static version of libpython, such that the installed package does not need to depend on the conda-forge python package, which provides the dynamic libpython.so.

However, a Python-enabled GDB that is distributed without the accompanying Python stack used to build it is usable but not easily extendable (since there is no accompanying pip to install packages that you might want to use in the Python scripts you are writing to extend GDB).

I think some versions of Intel's gdb-oneapi are distributed that way and I recall having a very hard time installing additional packages so that the built-in interpreter recognizes them.

jakirkham commented 8 months ago

Would split packages be helpful here?

Maybe we could have a slimmed down GDB package for users that are uninterested in Python portions and then have a batteries included version that ships with the other pieces

If that doesn't work for some reason, another option might be building two variants of the package. One with Python and one without

pitrou commented 8 months ago

Hmm. I would not be satisfied with a gdb without Python support (that would break a lot of stuff, including pretty-printing for libstdc++ types). What I would like is a gdb whose embedded Python interpreter is dissociated from the Python interpreter I use for other purposes.

Also, I don't think that would require static linking.

jakirkham commented 8 months ago

Unfortunately think that comes with the tradeoffs that Philippe enumerated above

Plus that fact that static linking doesn't always play well in packaging ecosystems (hard to track version used in binaries, tricky to identify/fix CVEs, doesn't get migrations, etc.)

There's also another question that comes up. Namely which Python version should it build against?

Maybe a better approach would be to install GDB in its own environment separate from other things. Then stack the Conda environments

Edit: This isn't to static linking is not an option. Just some things we need to be mindful of here

pitrou commented 8 months ago

I didn't know about stacked environments. The problem is it doesn't work if the Python version is not the same, because the libpython.py that's shipped with the gdb package is not compatible.

Example:

  1. create and activate environments

    $ mamba create -n tgdb gdb
[...]
$ mamba create -n tpy312 python=3.12
[...]
$ mamba activate tgdb
$ mamba activate --stack tpy312

  2. Check that gdb uses Python 3.11 with the right path

    $ gdb
(gdb) python
>import sys; print(sys.version, sys.path)
>quit
3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 15:07:28) [GCC 12.3.0] ['/usr/share/gcc/python/', '/home/antoine/mambaforge/envs/tgdb/share/gdb/python', '/home/antoine/mambaforge/envs/tgdb/lib/python311.zip', '/home/antoine/mambaforge/envs/tgdb/lib/python3.11', '/home/antoine/mambaforge/envs/tgdb/lib/python3.11/lib-dynload', '/home/antoine/.local/lib/python3.11/site-packages', '/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages']
(gdb) quit

  3. Now debug an actual crashing Python program, using the activated Python 3.12

    
$ gdb --args python -c "import faulthandler;faulthandler._sigabrt()"

(gdb) run Starting program: /home/antoine/mambaforge/envs/tpy312/bin/python -c import\ faulthandler\;faulthandler._sigabrt() [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGABRT, Aborted.

4. Try to print the C backtrace? The loaded libpython.py should annotate it with useful information from the Python interpreter...
```console
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350387520) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737350387520) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737350387520, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7cb8476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c9e7f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x000055555570664a in faulthandler_sigabrt (self=<optimized out>, args=<optimized out>)
    at /usr/local/src/conda/python-3.12.1/Modules/faulthandler.c:1091
#6  0x0000555555768593 in cfunction_vectorcall_NOARGS (
    func=<built-in method _sigabrt of module object at remote 0x7ffff7125710>, args=<optimized out>, 
    nargsf=<optimized out>, kwnames=0x0) at /usr/local/src/conda/python-3.12.1/Include/cpython/methodobject.h:50
#7  0x000055555577b93f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x7ffff7fb0078, 
    callable=<built-in method _sigabrt of module object at remote 0x7ffff7125710>, 
    tstate=0x555555bff108 <_PyRuntime+459656>) at /usr/local/src/conda/python-3.12.1/Include/internal/pycore_call.h:92
#8  PyObject_Vectorcall (callable=<built-in method _sigabrt of module object at remote 0x7ffff7125710>, 
    args=0x7ffff7fb0078, nargsf=9223372036854775808, kwnames=0x0) at /usr/local/src/conda/python-3.12.1/Objects/call.c:325
#9  0x00005555556676ab in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb0020, throwflag=<optimized out>)
    at Python/bytecodes.c:2706
#10 0x000055555582f17e in PyEval_EvalCode (co=co@entry=<code at remote 0x7ffff72c0f30>, globals=globals@entry=Traceback (most recent call last):
  File "/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages/libpython.py", line 1592, in to_string
    return pyop.get_truncated_repr(MAX_OUTPUT_LEN)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages/libpython.py", line 225, in get_truncated_repr
    self.write_repr(out, set())
  File "/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages/libpython.py", line 805, in write_repr
    pyop_key.write_repr(out, visited)
  File "/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages/libpython.py", line 1440, in write_repr
    proxy = self.proxyval(visited)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/antoine/mambaforge/envs/tgdb/lib/python3.11/site-packages/libpython.py", line 1384, in proxyval
    if not int(state['ready']):
               ~~~~~^^^^^^^^^
gdb.error: There is no member named ready.

Ouch! As you can see libpython.py for Python 3.11 is not compatible with 3.12 (case in point).

This is why shipping a libpython.py with gdb for a randomly-chosen Python version is not right. It will work for a subset of users, out of pure luck.

jakirkham commented 8 months ago

Think we want to stack in the other order

Though maybe errors still occur. Would be interested to know

Another option would be just to add the GDB environments bin to the `PATH manually. They should be nearly the same as activating environments, but is possible Conda does something more, which is leading to the problems we are seeing

Last option (which is pretty hacky admittedly) would be to use conda remove --force to remove the python from the GDB environment after that Conda environment is created and before stacking. This would ensure there is only one python used and it is from the environment outside GDB's

jakirkham commented 8 months ago

cc @gmarkall (as we discussed this issue briefly earlier)

pitrou commented 8 months ago

Think we want to stack in the other order

What would that change? gdb would still use a libpython.py meant for 3.11, which wouldn't work properly when debugging a non-3.11 Python interpreter.

The solution is for gdb to stop shipping its own libpython.py, and for each package of python to ship the corresponding libpython.py in a (hopefully) well-known place of gdb autoloading.