Improve inline breakpoint experience to be similar to TypeScript's

[luabud]

This is coming from a conversation with @tonybaloney where he raised some issues with inline breakpoints with Python. First, inline breakpoints aren't very discoverable at the moment, it would be nice if the possible inline breakpoints were highlighted just like it's done for TypeScript:

Inline breakpoints that are added to Python code also don't seem to be validated when added:

[luabud]

fyi @paulacamargo25 @karthiknadig

[karthiknadig]

@fabioz @int19h This data should be sent from the debugger right?

[brettcannon]

@luabud are you asking for the community to vote on this as you forgot the canned response to ask for votes?

[tonybaloney]

I've looked at the DAP docs and compared it with the typescript implementation. This whole implementation needs to happen in debugpy and the ptvsd handler.

I hacked together a quick fork with the missing properties and implemented the callback and it works in VS Code without any changes needed to the extension.

If this isn't too tricky I'll submit a PR to debugpy

[tonybaloney]

https://github.com/microsoft/debugpy/pull/1094

[tonybaloney]

Further digging shows that even though it is possible to set the breakpoints, neither debugpy or pydevd supports setting of breakpoints any more granular than a line. Python's tracing functionality is by line or by opcode, so you could work out by the opcode where in the line it was, but this would require more work in pydevd than just updating the protocol.

[tonybaloney]

This is the original implementation in the IDE https://github.com/microsoft/vscode/issues/14784

[fabioz]

Further digging shows that even though it is possible to set the breakpoints, neither debugpy or pydevd supports setting of breakpoints any more granular than a line. Python's tracing functionality is by line or by opcode, so you could work out by the opcode where in the line it was, but this would require more work in pydevd than just updating the protocol.

Well, you're right, this will definitely need more work in the debugger side, but I don't think we should go to the opcode tracing (because it's much, much slower).

I'd only envision really implementing this when/if PEP 669 is implemented (doing it in the approach where we change the bytecode to add programmatic breakpoints -- a.k.a: frame evaluation -- we have now this isn't really feasible because this mode needs to fallback to the line tracing after a breakpoint is hit due to the go to line feature -- besides, we don't have a way to change the bytecode after code is being executed in a frame, so, breakpoint changes in an executing frame wouldn't be possible).

[tonybaloney]

+1 on opcode tracing being too slow (even with Cythonize)

Looking more at how this would be used, setting a breakpoint in VS Code on line 1 will fire 10 times, once for the assignment (line event) and then it will match the line on each call event for the function object inside the comprehension.

x = [i for i in range(10)]

This seems to be a side-effect of how pydevd works out whether a breakpoint has been hit by also supporting DAP's function call breakpoint.

The same applies to lambdas.

The debugger could be adapted to show a column breakpoint at the start of the list/set/dict comprehension, or lambda function so that the user can optionally break inside the call. But considering it does this now anyway, the feature would be more that:

1. It's obvious where the breakpoint has hit (ie inside the lambda/comprehension and not on the surrounding line)
2. You can choose to only break once on assignment
3. You can set a conditional on the breakpoint in the comprehension/lambda

[fabioz]

This seems to be a side-effect of how pydevd works out whether a breakpoint has been hit by also supporting DAP's function call breakpoint.

That's not really the case, it's more that the debugger just checks for filename/lineno to determine whether to stop and Python is producing those new line events on this case because it actually creates a new frame for execution in the list comprehension.

Now, this is good. In this case we could make a distinction where there's a separate code inside code with matches in the same line where we have a clear differentiation on the scope (this would work close to the way that we do a step into right now -- in that case it determines whether to stop based on the current bytecode in the parent frame, this case would be even a bit less complex as it'd just need to determine if it's the proper line/col + inner frame name).

For instance, in the case of a list comprehension (see dis below), we could check that we're inside of a <listcomp> and break only in that case -- or vice versa (as long as we have a new line event when we enter the related code, this should be implementable).

Python 3.8.1 (default, Jan  8 2020, 15:55:49) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> def foo():
...     return    [i for i in range(10)]
...
>>> dis.dis(foo)
  2           0 LOAD_CONST               1 (<code object <listcomp> at 0x000001C4F3F1EA80, file "<stdin>", line 2>)
              2 LOAD_CONST               2 ('foo.<locals>.<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (range)
              8 LOAD_CONST               3 (10)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

Disassembly of <code object <listcomp> at 0x000001C4F3F1EA80, file "<stdin>", line 2>:
  2           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                 8 (to 14)
              6 STORE_FAST               1 (i)
              8 LOAD_FAST                1 (i)
             10 LIST_APPEND              2
             12 JUMP_ABSOLUTE            4
        >>   14 RETURN_VALUE

We'd be constrained on getting the real source code to obtain the AST to determine column locations as column locations are not available just from the bytecode -- making the mapping is a bit tricky though and in some corner cases it may not be possible to make a proper mapping if there's more than one match in the same line.

i.e.: in the code below it wouldn't be possible to distinguish the first lambda from the second one (likewise for the list comprehension).

def foo():
    return lambda: [i for i in range(10)], lambda: [x for x in range(20)]

@karthiknadig this issue should be moved to debugpy (I think vscode-python doesn't really need to do anything here, the work is all on the debugger side).

[fabioz]

On a 2nd thought, maybe with https://peps.python.org/pep-0657/ we could be able to differentiate the columns on Python 3.11 (so, we'd stop at a given scope based on the context name as well as the bytecode position of the parent), in which case we could differentiate among the lambas and list comprehensions in the same line.

[tonybaloney]

See #1099 as well

[Ashark]

I tried to place inline breakpoint with Shift + F9, but it is treated as a usual line breakpoint. For example, I have such code:

def func1():
    return 3

def func2(arg):
    arg = arg + 5
    return arg

def func():
    return func1() * func2(4) + func1()

print(func())
pass

I have placed the breakpoint before the func2(4) invocation.

This is how it looks:


When I start debugging, the debugger is actually stopped at the beginning of line, so when I step in, it actually steps in the func1, but not in the func2.

Any way to get this working in python 3.12?

[luabud]

@ashark yes unfortunately our inline breakpoint experience doesn't support that at the moment, this is part of the feature request we're tracking here.