Rework template settings

[int19h]

We should see if it's possible to reliably auto-detect the templating engines used, e.g. by probing sys.modules, so that Django, Jinja etc "just work" without the need to set any properties in the debug configuration.

Regardless of whether this is feasible, we should also tidy up the template settings themselves, and group them in a single option. Something like:

"templates": "auto"

"templates": ["django", "jinja"]

[int19h]

@fabioz, what do you think about auto-detection? I know we've had a bad experience with Qt because of that, but that code was actually trying to import the module. If the probing looks at what's already in sys.modules instead, it should be side-effects-free. But I'm not sure there's an existing hook in which we could do such a check without a significant perf overhead?

[fabioz]

Well, we actually always try to import django right now in the django plugin -- and so far I don't know of any issue in doing so... the issue with Qt is mostly related to its binary dependencies (which is not the case for django).

This issue is mostly related to the performance overhead when plugins are enabled (since in theory both the django and jinja plugins could be always enabled without any issues), but I haven't done much performance analysis in these cases, so, maybe the first point would be doing that to then check on the best approach...

Some things that come to mind:

• plugins aren't currently cythonized (which should enable good speedups when available).
• file extensions for templates aren't well standardized, so, right now, any breakpoint for a non .py file would enable a plugin breakpoint (so, if django or jinja modes are enabled a breakpoint in a .cpp would trigger it), but maybe we could say that we handle a bunch of extensions such as .html, .djhtml, .jinja, .jinja2 -- and possibly others and for line breakpoints we'd automatically enable both plugins in this case... which means mostly that we check some additional conditions to decide if a context should be traced or not... although in the case of jinja, I can see cases where even a breakpoint in a .cpp could trigger breaks, so, it should be customizable).
• breakpoint exceptions are a bit trickier in this case as there are cases when users want them or not... the current approach is that a handled exception breakpoint + django option would start to handle django exceptions too (we could potentially want something more fine-grained when needed).

So, I think it should be possible to have plugins enabled by default (which would only kick if some breakpoint is added anyways or on handled exceptions), but to do that some work is probably needed to optimize that use case more than right now.

If everything would be changed to be enabled by default, we could potentially have something as the api below for the user to selectively enable/disable it or change the extensions we consider to be templates.

"templates": {
    "django": {
        "exception-breakpoints": "disable",
        "line-breakpoints": "enable",
        "extensions": [".custom_dj_file", ".html"],
    }
    "jinja2": {
        "exception-breakpoints": "enable",
        "line-breakpoints": "disable",
    }
}

[int19h]

(@luabud, FYI)

With respect to exceptions, I don't think we've had any requests for that degree of control so far. But if we decided to do it, I think we'd want to tie it into exception filters somehow, so that in e.g. VSCode, the user could check/uncheck "Django exceptions" at runtime. Better yet, if DAP gets a way for the adapter to enumerate specific exception types, we could have a separate top-level category for Python, Django, Jinja etc (with duplicated exception hierarchy underneath).

It's a good point about template file extensions. To clarify, does pydevd need to know which plugin is going to handle that particular extension? Or does it only need to know that some plugin will, and then it can figure out which plugin "owns" that file? So e.g. if somebody ends up with both Django and Jinja in the same project, would it be able to handle breakpoints in .html files regardless of which template framework is used to render them? Or would it be restricted to one framework?

[fabioz]

It's a good point about template file extensions. To clarify, does pydevd need to know which plugin is going to handle that particular extension? Or does it only need to know that some plugin will, and then it can figure out which plugin "owns" that file? So e.g. if somebody ends up with both Django and Jinja in the same project, would it be able to handle breakpoints in .html files regardless of which template framework is used to render them? Or would it be restricted to one framework?

Internally it's quite flexible in that breakpoints can be added as regular breakpoints, django breakpoints or jinja breakpoints and adding a breakpoint for a given file to one plugin does not preclude from adding another breakpoint of the same file with another plugin (so, it's not restricted to one framework and it should be possible to debug both django and jinja templates at the same time).

[int19h]

But can it figure out the right framework just from the filename / line number? Or apply them for all applicable frameworks at the same time?

[fabioz]

It can apply a breakpoint to all frameworks at the same time...

i.e.: what's available is an api close to:

add_breakpoint(breakpoint_type, filename, lineno, ...)

So, it's possible to call:

add_breakpoint('jinja2-line', 'my.html', 1, ...)
add_breakpoint('django-line', 'my.html', 1, ...)
add_breakpoint('python-line', 'my.html', 1, ...)

it doesn't really do any validation on whether that's valid for a given framework (after such a breakpoint is added it'll start to trace the related context to match breakpoints, otherwise, until such a breakpoint is added this overhead isn't added during the tracing).

For exceptions it's also similar, there's a add_python_exception_breakpoint(...) and add_plugins_exception_breakpoint(breakpoint_type, ...).

The actual semantic is up to the client (for instance, on Eclipse/PyDev that's actually dictated by the editor... if it opens a Python editor it adds python line breakpoints, if it adds a django editor it adds django line breakpoints and if it opens the jinja editor it adds jinja line breakpoints... for exceptions the options are the list of python exceptions and a toggle for django and another for jinja) -- for the DAP the current heuristic chosen is that .py files will add a regular breakpoint and for any other breakpoint it's decided based on whether django or jinja is selected in the launch configuration.

[int19h]

No traction on this to date; closing until we decide we actually need it.