Add a debugging howto

[merwok]

BPO	12913
Nosy	@akuchling, @terryjreedy, @ezio-melotti, @merwok, @roseman, @davidmalcolm, @rovitotv

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/merwok' closed_at = None created_at = labels = ['type-feature', 'docs'] title = 'Add a debugging howto' updated_at = user = 'https://github.com/merwok' ``` bugs.python.org fields: ```python activity = actor = 'eric.araujo' assignee = 'eric.araujo' closed = False closed_date = None closer = None components = ['Documentation'] creation = creator = 'eric.araujo' dependencies = [] files = [] hgrepos = [] issue_num = 12913 keywords = [] message_count = 13.0 messages = ['143615', '143789', '143830', '143908', '143921', '143925', '143946', '146091', '146093', '173127', '173129', '280158', '302308'] nosy_count = 9.0 nosy_names = ['akuchling', 'terry.reedy', 'ezio.melotti', 'eric.araujo', 'markroseman', 'dmalcolm', 'Todd.Rovito', 'tshepang', 'Ramchandra Apte'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue12913' versions = ['Python 2.7', 'Python 3.3', 'Python 3.4'] ```

[merwok]

I found a few blogs posts that explained how to use pdb. It appears from the comments that such introductory material is very useful to a lot of users. Instead of just expanding the pdb module docs, I propose to add a debugging howto covering basic troubleshooting techniques and giving a pdb tutorial. I think a howto has more visibility, as people unaware of the existence of a thing named pdb may however look for “debugging”.

To educate myself on pdb, I have started writing a document. Before investing too much time, I’d like to know if people agree with the principle of adding such a howto.

[terryjreedy]

If you write 'How to debug Python code' rather than just "How to use pdb", I would start with the use of print statements and binary search. Have short sections on using trace and profile. Very useful would be a list of error messages and possible un-obvious to beginners but common causes. The following example comes up regularly on python-list.

TypeError: 'int' object is not callable Look in the previous line to see what you called. If it is a builtin name, perhaps you re-assigned the name to something else. Example:

list = 3
<many lines later>
list(1,2,3)
Traceback (most recent call last):
  File "<pyshell#7>", line 1, in <module>
    list(1,2,3)
TypeError: 'int' object is not callable 

Another one we see occasionally ("Is module x broken?") is something like: user runs script.py

import random
x = random.random()
Traceback...
NameError: name 'random.random' is not defined

Solution: user has a file random.py in the same directory

[ezio-melotti]

FWIW I almost always use "print" for debugging, and find the use of a real debugger overkill in Python in most of the cases. People coming from other languages often feel the need of using a debugger because that's what works best with the other languages, but it's not necessarily true for Python.

I'm not sure focusing on pdb is the best idea then. While it should be mentioned and a quick overview should be presented, a more extensive guide should probably go alongside the pdb doc, or possibly in a pdb-specific howto.

Mentioning unittest and coverage as a way to find errors earlier might also be a good idea (and reply to the "it's easier to debug compiled code because if I do something wrong the compiler gives me an error immediately" argument).

But all this might not be what you had in mind :) I'm wondering if, for this kind of work, it's better to set up a clone with the document and its outline, in order to allow different persons to work on it before pushing it. Rietveld works well for corrections, but a clone is better if you actually want to add something (and faster if you want to make corrections).

[merwok]

[Terry]

If you write 'How to debug Python code' rather than just "How to use pdb", That is my intention.

I would start with the use of print statements You, Ezio and I concur :)

and binary search. I’m not familiar with that term. Is it hg bisect?

Have short sections on using trace and profile. I haven’t thought about these, thanks.

Very useful would be a list of error messages and possible un-obvious to beginners but common causes. These would be great additions to the programming FAQ, and my document could link to it.

[Ezio]

People coming from other languages often feel the need of using a debugger because that's what works best with the other languages, but it's not necessarily true for Python. I’ll be sure to make this clear, thanks.

I'm not sure focusing on pdb is the best idea then. My initial message did not do justice to my outline. The part about pdb is probably the one that’ll take me more time, because I nearly don’t know it at all, but it won’t be the focus of the document. A quick “getting started with pdb” section is what I have in mind, something more newbie-level than the pdb module docs (which seem to assume basic knowledge of how a Unix debugger program works).

Mentioning unittest and coverage as a way to find errors earlier might also be a good idea Hadn’t though about coverage. Will mention it and add a link to the devguide part that talks about it.

I don’t think I’ll have much time for this in the following few weeks, but as your replies appear to accept the idea, I will set up a clone to let us work on this.

Here’s my initial outline:

=== Introduction

• running a script in a terminal or IDLE
• getting a traceback (file and line)
• making sense of the exception message

=== Simple Troubleshooting Techniques

• print debugging

• using facilities provided by modules used

  • increasing logging verbosity
  • httplib.debuglevel
  • your web framework's debug helper

• going into interactive mode

• Using lint tools to catch mistakes early

• pyflakes first

• pylint -E

• pychecker?

• Getting more information out of failing tests

• using the most specialized methods to get diffs

• msg argument for boolean-returning methods

• print! suspend! All is fair in debugging.

• Automating troubleshooting

• emacs/vim quickfix (mode that runs tests and open faulty files at the right line)

=== Using a Debugger

• pdb intro that does not duplicate what’s in library/pdb.rst
• definitions (breakpoint, etc.)

=== Debugging C code

(just a few pointers, this could have a whole howto in itself)

• running Python in gdb
• using faulthandler

Links to add somewhere: traceback module, cgitb

[terryjreedy]

Binary search with print is done manually. If error not obvious from quick read, in a 20 line function, add print around line 10. If ok there, look down and add print later in function. If not, look up and add print earlier in function. This is not exact at all, but quick reruns (F5 with IDLE) mean no need to load up entire function with prints (that will have to be removed again) all at once (as might do with compiled C 20 years ago). Same idea as binary search through revisions to find buildbot breaker. I am sure you know this, even if not by terms I used.

[terryjreedy]

More time: read outline, good start. On syntax errors, IDLE put up message box and OK returns to window with apparent error hi-lited and cursor just after.

[ezio-melotti]

Hadn’t though about coverage. Will mention it and add a link to the devguide part that talks about it.

Is devguide/coverage.html#using-coverage-py generic enough? We don't have to duplicate the coverage documentation though, mentioning the tool and what it does, provide a couple of simple example and a link to the coverage doc should be enough.

Talking about unittest and coverage is not strictly about debugging (you don't have any bug yet), but it's useful because it might reveal bugs earlier and avoid debugging sessions later. This might also go with pyflakes and friends, since they both provide a way to detect bugs earlier.

• running Python in gdb

This is somewhat orthogonal, but the devguide/gdb page doesn't say how to start running Python in gdb (it might be obvious to people used to use gdb, but it should still be mentioned).

If error not obvious from quick read, in a 20 line function, add print around line 10.

I usually print variables in order of suspiciousness, i.e., I usually have an idea about where the problem might be, if it's not there I move to the next suspect. This also applies when I know which variable is "wrong" but I don't know where it got wrong: I just add the prints around the most suspicious function that might have changed.

[merwok]

[Terry]

Binary search with print is done manually. If error not obvious from quick read, in a 20 line function, add print around line 10. If ok there, look down and add print later in function. [...] Okay, so it’s what I thought only I didn’t had the name :)

More time: read outline, good start. Thanks! I’ll publish a repo when I start writing.

[Ezio]

Hadn’t though about coverage. Will mention it and add a link to the devguide part that talks about it.

Is devguide/coverage.html#using-coverage-py generic enough? I had a quick look and it looks good.

We don't have to duplicate the coverage documentation though, mentioning the tool and what it does, provide a couple of simple example and a link to the coverage doc should be enough. Yep, I was thinking about mentioning, nothing more.

> - running Python in gdb This is somewhat orthogonal, but the devguide/gdb page doesn't say how to start running Python in gdb (it might be obvious to people used to use gdb, but it should still be mentioned). If the devguide is updated, I will be content with just one line containing one link in my howto. Currently we have this wiki page: http://wiki.python.org/moin/DebuggingWithGdb I also remember a ubuntu wiki page with more useful info but can’t find the bookmark.

FTR, I will use these resources: https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/ and http://www.jeetworks.org/node/99

[davidmalcolm]

>> - running Python in gdb > This is somewhat orthogonal, but the devguide/gdb page doesn't say how to start running Python > in gdb (it might be obvious to people used to use gdb, but it should still be mentioned). If the devguide is updated, I will be content with just one line containing one link in my howto. Currently we have this wiki page: http://wiki.python.org/moin/DebuggingWithGdb I also remember a ubuntu wiki page with more useful info but can’t find the bookmark.

FTR, I will use these resources: https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/ and http://www.jeetworks.org/node/99

There's also: http://docs.python.org/devguide/gdb.html

[a2c68251-5f3b-4972-91b1-b78401c27569]

I think this is an excellent idea. How about putting in some advanced debugging with IDLE? For example I have read somewhere that IDLE lets you set break points but the documentation for IDLE OS not that clear on debugging.

[918f67d7-4fec-4a8d-93e3-6530aeb1e57e]

Right-click to open a menu in which you can set a breakpoint. I don't think debugging in IDLE needs a tutorial. (except the setting breakpoint thing should be documented.

[akuchling]

Éric Araujo: did you ever make any progress on this, such as producing a draft version?

[merwok]

Hello! I didn’t produce anything for this apart from the initial outline. Looking at it now, it looks like a talk outline! Maybe I should run with it, present at my local user group, and only write it up after collecting feedback from real beginner/intermediate programmers.