Include python-dbg in Anaconda distribution

[parautenbach]

We are trying to debug a segmentation fault while running unit tests with nosetests on Anaconda. For that, we require python-dbg. This is available to the system Python (for us, under Ubuntu 12.04). One can compile this python-dbg yourself with the correct flags, but we deem it an error-prone and large task to compile Anaconda and all its dependencies ourself. Would it be possible to include this executable in the Anaconda distribution?

We'd be willing to try any suggestions, if any.

[asmeurer]

Do you also need to recompile any extension modules, like numpy?

[parautenbach]

I'm not sure (and I don't quite know how to find out), but I guess that a recompile would be required for all code that's not pure Python (e.g. Cython code, etc.).

[asmeurer]

Yes apparently extension modules are incompatible (https://wiki.ubuntu.com/PyDbgBuilds).

[parautenbach]

Thanks for confirming.

[asmeurer]

So an option for you would be to try to build your own conda package for python-dbg. See http://conda.pydata.org/docs/build.html. If you go this route, I suggest using the features flag.

We are contemplating add this feature to our paid product, Anaconda Server.

[parautenbach]

Thank you for the feedback and the suggestions. We'll try that.

On 10 February 2014 18:06, Aaron Meurer notifications@github.com wrote:

So an option for you would be to try to build your own conda package for python-dbg. See http://conda.pydata.org/docs/build.html. If you go this route, I suggest using the features flag.

We are contemplating add this feature to our paid product, Anaconda Serverhttp://www.continuum.io/anaconda-server.

Reply to this email directly or view it on GitHubhttps://github.com/ContinuumIO/anaconda-issues/issues/80#issuecomment-34648461 .

[pitrou]

A pydebug build of Python would be very useful to improve the debugging experience for C extensions, including Numba and llvmpy (this also includes chasing memory leaks).

See https://docs.python.org/3/c-api/intro.html#debugging-builds for the (incomplete) list of features that come with a pydebug Python build. See https://github.com/numba/numba/pull/607 for a Numba pull request that would benefit greatly from a pydebug build of Python.

Given the requirement to recompile (most) C extensions, a pydebug build of Python is really like a separate version, e.g. 3.3 or 3.4. This would probably double the number of Python versions handled by Anaconda.

[csoja]

@ilanschnell I believe this has been done for some time and we can close this issue?

[ilanschnell]

No, this has not been done, I don't consider it an issue either.

[leifwalsh]

I do consider this an issue, not being able to use gdb on python programs significantly hampers usability from a developer perspective. Will you consider reopening it, or justify why you don't consider it an issue?

[mcgibbon]

Somewhat related to this issue: https://github.com/conda/conda/issues/5780

[msarahan]

This is available if you conda-install it as an add-on after installing Anaconda. We can't include it in the primary Anaconda installer for space reasons. You do need network access to conda-install python-dbg, or you need some scheme for a locally-hosted channel.

[msarahan]

PS: there are two very different things here (on windows):

1. .pdb files, which provide debugging capability. We can, and do ship these on Windows. We can think about not stripping binaries on other platforms so that these might be available.
2. debug builds, which do not do any optimization.

The latter is a much bigger thing to ask for. It may be possible in the future with coming conda and conda-build features (building package variants and the ability to select variants at install time) to build debug versions of many programs and libraries. This also depends on a much more automated system than we have right now, to deal with the additional builds required. We're working on that.

[mcgibbon]

When you say "provide debugging capability", do you mean as in this guide?

[msarahan]

Sorry, I have no idea exactly what this requires (perhaps our own gdb build?) and also no time to investigate this with any depth. If anyone is willing to figure out exactly what we need to do to support this, I'd gladly investigate and work on integrating it into our provided packages.

[mcgibbon]

I'm also not entirely sure what it requires, but it seems to have been implemented already in the "debug" package. However, it is broken, resulting in this issue. At the least I know it breaks numpy and conda. It seems to completely replace "python" with "python-dbg", rather than give a python-dbg binary, based on the behavior I got (no python-dbg and the behavior of python and ipython changed), but I'm not entirely sure.

[rainwoodman]

a full python debug build is binary incompatible because two additional fields are added to PyObject. But what about a partial debug build (e.g. a debug-info builde) that includes all of the symbols and disable the optimizations? From the discussion here it appears the debug-info is really the most relevant issue.

[rainwoodman]

@mcgibbon : why do you say the debug package is relevant to python-debug / python-dbg?

[mcgibbon]

@rainwoodman I don't really know what I'm talking about, but as far as I could tell the "debug" package was supposed to install python-debug. I could be wrong. All it seems to do is replace the Python binary, but with what I can't say for sure (I guess it could be something other than python-debug).

[mcgibbon]

@rainwoodman the debug package also broke many other packages for me, to the point that "conda" couldn't run, so the idea that debug is putting in python-debug and the idea that python-debug is binary incompatible might not be mutually exclusive.

[mingwandroid]

You can of course build your own debug-symbol-containing Python using our recipes (minor modifications will be needed). If you do those modifications cleanly in a way that is easy to disable I would have no issue with merging them. You could even use split packages so the debug symbols get put in a separate package, and I would definitely support releasing that on the default channel.

Full Py_DEBUG builds of all Python packages are not something we can handle at present due to the combinatorial explosion due to binary incompatibility. Maybe in future we'll be able to go there.

[rainwoodman]

I think this file may have killed all debugging information in the current python package from anaconda. Currently I only see function names when python is loaded in gdb.

https://github.com/AnacondaRecipes/python-feedstock/blob/bdee59bab73daeb7fa284a94f23ad64f46e4ea52/recipe/0007-Do-not-pass-g-to-GCC-when-not-Py_DEBUG.patch#L1

On Linux can we achieve a similar file size reduction with -g build + strip?

[bombs-kim]

I followed @asmeurer advice and I succeed to install python-dbg in my conda environment. Here is how I did it. https://gist.github.com/bombs-kim/c9848c2b09962f2fd753b48b6d2cd87f

[nikhilweee]

@bombs-kim has the process changed in 4 years? I would really like if there were an easier way to install python-dbg with conda.