Provide improved support for django

[rchiodo]

MVP idea:

• Completion for all related keys generated and or specified #3704 (maybe through dataclass_transform)
• Extended completions for filterby etc based on key names generated by Django https://github.com/microsoft/pylance-release/issues/3704#issuecomment-1335592027
• Goto definition on models referenced in Settings.py #3700
• Completions on models referenced in Settings.py #3700

MVP Python Core?

• Manage.py shortcuts (especially so you could map them to keyboard shortcuts)

[diego351]

1. Ability to run task of the manage.py utility
2. Ability to goto definition models referenced by string by'<app_name>.<model_name>'when model sits in '<app_name>.models.<model_name>'

[rchiodo]

Manage.py could

• have tasks.json added for each operation
• have commands for each operation
• have more specialized commands for different operations (like manage.py runserver should be on the 'run' tab)

All of these are probably better suited for the core python extension though. Not really language server related. See this documentation: https://code.visualstudio.com/docs/python/tutorial-django#_create-a-debugger-launch-profile

Which could be improved, but probably better to do that in the python extension.

[rchiodo]

Views.py

• Auto fill in types for any function that has a request parameter

def index(request):
   pass

becomes

def index(request: HttpRequest) -> HttpResponse:
   pass

automatically (line completions maybe?)

[rchiodo]

Urls.py

• Auto line complete all of the exported views from views.py (and compute their path?)

[rchiodo]

https://github.com/microsoft/pylance-release/issues/3704

[rchiodo]

Fields with required values should flag an error if those values are missing. Example:

class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField()
    votes = models.IntegerField(default=0)

choice_text is missing max_length

[rchiodo]

What @diego351 said in this issue: #3700

Settings.py basically takes a list of modules. It would be nice if they could be treated that way when entering them.

Like by doing

• Autocompletion on them
• Goto def on them

[rchiodo]

Could completions for models be driven by the data in the 'migrations' folder?

[diego351]

@rchiodo It's actually interesting how Jetbrains inferes that. Might do some tests.

btw, @rchiodo do you get notifications for closed issues?

[rchiodo]

btw, @rchiodo do you get notifications for closed issues?

Only if I had already responded to the item or you mention me by github id.

Is there another item we already closed that's about this topic?

[diego351]

@rchiodo Not about that topic, but don't know how to contact you another way, didn't wanted to create duplicate ticket https://github.com/microsoft/pylance-release/issues/3678

[diego351]

  "python.analysis.packageIndexDepths": [
    {
      "name": "django",
      "depth": 3,
      "includeAllSymbols": true
    },
    {
      "name": "rest_framework",
      "depth": 3,
      "includeAllSymbols": true
    }
]

Just this regular django setup hits Workspace indexing has hit its upper limit: 2000 files. In production it's much more needed. We need https://github.com/microsoft/pylance-release/discussions/2608 to make vscode competitive django IDE.

[debonte]

Created an issue to think about whether dataclass_transform can help with Django models: https://github.com/microsoft/pylance-release/issues/3720

[rchiodo]

Some ideas from the work done here: https://youtu.be/Lz4I7rFmFdY?t=1233

Generate type stubs on the fly for django classes

[rchiodo]

At this point in the video you can see the presenter use what @diego351 talked about in #3704

[rchiodo]

Comment here makes it seem like Django functions have data automatically sent to them (like pytest fixtures): https://github.com/microsoft/pylance-release/discussions/3187#discussioncomment-3421854

Type inference would be specific to Django I'd think. Much like pytest fixtures. Needs more investigation to see how this works.

[rchiodo]

https://github.com/microsoft/pylance-release/issues/3704#issuecomment-1335592027

Looking here: https://reinout.vanrees.org/weblog/2015/06/03/05-lookups-transforms-expressions.html

Shows that Django completions for parameter names can include a LOT of stuff. It would be cool to include at least some of these (if not all) in completions for parameters. (much like Pycharm does)

[rchiodo]

An example of why the 'generated' keys are useful.

Suppose we have these two models (from the Django tutorial):

class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

class Choice(models.Model):
    question = models.ForeignKey(Question, on_delete=models.CASCADE)
    choice_text = models.CharField(max_length=200)
    votes = models.IntegerField(default=0)

Both of these objects have hidden (generated) fields.

• Both have an id
• Question has a 'choice_set' generated by the ForeignKey relationship
• Both have a primary key called 'pk' which by default maps to 'id', but can be overridden

Example usage:

q = Question.objects.get(pk=1)
q.choice_set.create(choice_text='Not Much', votes=0)

In this example, pylance would have to know

• The implicit fields
• The types of those fields
• The implicit arguments for methods on those fields

[rchiodo]

Another idea: Query pattern matching.

Urls might have something like so:

urlpatterns = [
    path("", views.index, name="index"),
    path("<int:question_id>/", views.detail, name='detail'),
    path("<int:question_id>/results", views.results, name='results'),
    path("<int:question_id>/vote", views.vote, name='vote'),
]

See here for more information: https://docs.djangoproject.com/en/4.1/topics/i18n/translation/#translating-url-patterns

It might be possible to have a general regular expression support? Doesn't seem to be asked for, but maybe as a separate extension. Patterned after what regex101.com does.

[rchiodo]

Another obvious example, intellisense in Django templates. This would likely require an entirely different parser.

Yeah definitely. Django's template language is not python.

Example, this is how you concat two strings:

{{ str_1|add:str2 }}

[rchiodo]

There are predefined globals that can be set that have special meaning.

It would be nice if they:

• Came up in completions
• Had type information

Example - a Urls.py:

from django.urls import path
from . import views

app_name = 'polls'

urlpatterns = [
    path("", views.index, name="index"),
    path("<int:question_id>/", views.detail, name='detail'),
    path("<int:question_id>/results", views.results, name='results'),
    path("<int:question_id>/vote", views.vote, name='vote'),
]

app_name and urlpatterns are globals that django looks for.

https://docs.djangoproject.com/en/4.1/topics/http/urls/#how-django-processes-a-request

[rchiodo]

Warning in forms that don't use the csrf_token tag

[rchiodo]

@diego351, @luabud pointed this extension out to me:

https://marketplace.visualstudio.com/items?itemName=tonybaloney.python-task-provider

That provides commands for manage.py. Thought that might be useful for you.

[diego351]

Thanks! How the initial support is going? Please let me know if you need anything django related

[rchiodo]

@diego351 Honestly, we're still in the investigation phase, trying to figure out what to do first. Any more suggestions of things that you'd like to see or any other features you've seen elsewhere would be a big help. Thanks for asking.

[rchiodo]

Another idea: https://github.com/microsoft/pylance-release/issues/3874#issuecomment-1405665334

Supporting other languages inside of strings. Might also how we implement DTL support. Ship a separate language server just for the DTL parts.

[dakotahorstman]

@rchiodo Perhaps off-topic, but intellisense for the django.conf.settings object, which includes user settings.py variables would also be phenomenal.

[rchiodo]

1. Ability to run task of the manage.py utility
2. Ability to goto definition models referenced by string by'<app_name>.<model_name>'when model sits in '<app_name>.models.<model_name>'

@dakotahorstman do you mean like option 2 that @diego351 mentioned?

[dakotahorstman]

Albeit, I split my settings.py out into multiple settings/*.py files depending on which mode my server is running in. For example: settings/base.py, settings/dev.py, settings/prod.py, settings/test.py if that makes much of a difference.

[dakotahorstman]

@rchiodo I mean all settings, not just model definitions. Like regular intellisense, if I were to type settings., it would suggest DEBUG, INSTALLED_APPS, DATABASES, etc.

[rchiodo]

@dakotahorstman can you share your source? It would help to have an example.

[dakotahorstman]

@rchiodo Unfortunately, I can't due to it being a closed-source application. The most bare-bones Django project can be used though. Here's a screenshot of my application's structure if it's any benefit:

settings/local.py and settings/production.py import everything from settings/base.py where all shared settings exist (like INSTALLED_APPS or DATABASES)

All my request is, if I import from django.conf import settings in networking.py, intellisense should suggest all of the module-level variables that exist in these settings/*.py files. For example, if settings/base.py has a FOO="bar" variable, intellisense will suggest FOO when I type settings. in networking.py. Think of the behavior being no different if I were to import from dtautomation.settings import base as settings and did the same thing.

I can try to provide a GIF if that would be more beneficial?

[rchiodo]

If I'm understanding correctly, I'm not sure why those variables are not already visible as completions?

Example:

file: test.py

import test2
test2.b

file test2.py

b = list()

Completions when in test.py do show test2.b

[rchiodo]

Oh no wait I see what you mean. You want the 'settings' object from django to dynamically have these other 'settings' on it.

[dakotahorstman]

@rchiodo Exactly, yes! Plus the ones Django already ships with if possible.

[rchiodo]

More Feedback

• Query list completions should work : See https://github.com/microsoft/pylance-release/issues/3704#issuecomment-1335592027
• Being able to see object tables and data inside them somehow
• Providing snippets like (https://djangosnippets.org/)
• Add data as show in the django debug toolbar (https://pypi.org/project/django-debug-toolbar/)
• Show more information about models and other classes (like MRO list)
• Tables are hard to make in html, can there be an easier way?
• Data sent to views is hard to get correct. Is there a way to flag an error when it's wrong?

[rchiodo]

Solution Ideas for previous comment:

Query list completions

• Mergeable type stubs and generating dynamic fields for models
• Custom logic in a completion extension that finds all model based classes and understands the connections between them

Being able to see object tables

• Hover provider to show current state of database (we could run the sql)
• Reuse data wrangler or data frame viewer (jupyter) to show different tables
• Custom graph solution?

Snippets for django related stuff

• Automatic inclusion of the top 30 djangosnippets? Maybe prefix them with #django or something
• Better completion for the things that the top 30 snippets do

Django debug toolbar

• Provide more of this data in a custom view while debugging/running
• Provide another view ourselves that queries this data at runtime with specific hooks
• Provide methods to easily add this to an app (as there's a lot of stuff to install, setup correctly)

Show more information about models and other classes

• Implement this: https://github.com/microsoft/pylance-release/issues/2846
• provide filtering in completions to only show methods of a specific class?

Tables are hard to make in html

• Specific snippets for making tables out of django data
• Snippets for just plain html

Data sent to view is hard to get correct

• Generate types for functions based on if they are used in views or not. Special case type generation for django specific things
• Make pyright's type stub generator usable
• Generate type annotations from call sites (for return types too if we know the return value is used as a parameter to another call)

[debonte]

Related issue where we could provide IntelliSense (similar to pytest extension) where all of the model's attributes are mapped to kwargs -- https://github.com/microsoft/pylance-release/issues/4370

[Viicos]

Mergeable type stubs and generating dynamic fields for models

For anyone interested, I started django-autotyping, which can generate dynamic stubs based on the current application (providing support for foreign field set and get types, and later support for lookup queries, etc).

This article goes into more detail.

[ldeluigi]

Has anyone mentioned support for the auto-generated "id" field?

[rchiodo]

Has anyone mentioned support for the auto-generated "id" field?

I don't believe the 'id' field is mentioned explicitly but it generally follows under this item: https://github.com/microsoft/pylance-release/issues/3704. I believe @Viicos work on autotyping would be relevant too. We're discussing doing something similar to what @Viicos has done but generating the type stubs inside of pylance as you type in your code.

[Viicos]

We're discussing doing something similar to what @Viicos has done but generating the type stubs inside of pylance as you type in your code.

Nice to hear your are also planning on exploring this solution. I did face some challenges though, do not hesitate to let me know if you want to hear more details about it