Feature suggestion: run Django unittests

[DonJayamanne]

From @jvaesteves on January 16, 2017 22:28

Django unittest extends from unittests, but you can't run as the former on this extension. For this, you need to "manage.py test", but it would be awesome if there were those same shortcuts.

Copied from original issue: DonJayamanne/pythonVSCode#648

[DonJayamanne]

From @guhuajun on January 19, 2017 6:51

I did something like following. I think it's supported now.

[DonJayamanne]

From @jvaesteves on January 19, 2017 12:13

I tried your suggestion, copying the same parameters on the settings.json file, but running the "Run Unit Test Method..." command still doesn't work. The Output Console gives me the following message:

ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

And then, when I configure this environment variable to my global_settings.py path, the message change to: ImportError: Import by filename is not supported.

[DonJayamanne]

From @guhuajun on January 20, 2017 0:22

https://github.com/DonJayamanne/pythonVSCode/wiki/unitest-(Standard-Unit-testing)-Framework#test-discovery Please try to place a __init__.py.

Yes, missing DJANGO_SETTINGS_MODULE is annoying. I encountered this issue when writing tests for a django-cms project.

[DonJayamanne]

@jvaesteves Will be closing this issue as @guhuajun has a solution.

[DonJayamanne]

@jvaesteves please confirm @guhuajun suggestion works, so I can close this issue.

[DonJayamanne]

From @jvaesteves on February 2, 2017 10:46

No, it didn't.

[DonJayamanne]

From @james-perretta on March 5, 2017 4:35

It looks to me like the suggestion from @guhuajun gets us part of the way there for running Django test cases. Unless I'm mistaken, you can't really use unittest to run tests for any serious Django project. When you run python manage.py test, it first does general setup (figuring out which settings module to use and calling django.setup()). After that, it hands the rest off to a test-runner class that does even more important things like creating and destroying the databases that the test cases need to operate on.

I did a bit of digging, and although I'm not particularly familiar with the implementation details of your plugin, I think this is a starter summary of what it would require to add this:

• Add client modules similar to those for nose and pytest (I have almost zero knowledge of how this part works)
• Add handling for testFx == 'django' in testlauncher.py. This would involve code similar to what you see here in the runtests.py snippet.
• Possibly add a workspace settings option that lets the user specify the value for DJANGO_SETTINGS_MODULE. That environment variable needs to be set before calling django.setup()
• Certain manage.py test options should always be passed, such as --no-input
• Add support for other django-specific unit test options, such as --tag, --exclude-tag, and maybe --parallel

Thoughts?

[DonJayamanne]

From @PlugaruT on October 3, 2017 11:15

Because I have different settings files for each environment (production, local, test), I can't run tests because with this plugin because DJANGO_SETTINGS_MODULE is not set when trying to run the tests. I would be nice if I could set this env variable in in the workspace.json or somewhere else.

[DonJayamanne]

@PlugaruT Environment variables can be set for the workspace. Just create a file named .env in your workspace root and add the necessary variables in there.

[wsantos]

Any update here?, I have a bunch of tests file, I'm not sure where to put django.setup() if that is a real solution.

[Kurara]

You don't need to configure DJANGO_SETTINGS_MODULE if you use the option: "--settings < filesetting >" Just saying because maybe we can make thah unittest plugin doesn't need env variable, just tell him the setting and if not, use default.

And just one question about my closed issue.

My problem is not implementation of new feature for django. But tests are not DISCOVERED at all. I could make them run at least in part, but if it doesn't discover them I can't. When I use python -m unittest it discovers the tests, so the structure must be ok.

[yardensachs]

Any decision on this? This is somewhat of a small ask, but it will make dev work better for those who use django and not pytest/nose/etc.

[brettcannon]

@yardensachs not at the moment (we're tied up with other things that impact all Python developers). If anyone wants to submit a PR to introduce a fix for this we would happily review it.

[evenicoulddoit]

I came across this issue, not because I actually need Django support, but rather the ability to run some arbitrary test command (in my case using Flask with a test database setup/teardown, similar to Django application).

I've recently moved over from Sublime, where I use the very useful Anaconda IDE plugin to run my tests. Might be a useful pointer for anyone looking to solve this. Their config (though not documented particularly well) allows for complete extensibility, e.g:

{
  "test_command": "./test.sh",
  "test_delimeter": "::",
  "test_filepath_patterns": true,
  "test_method_delimeter": "::",
  "test_virtualenv": "~/.virtualenvs/my-venv"
}

[Cetlan]

As a workaround for anybody still looking for django support for tests, I've had some pretty good luck with using pytest as the test runner and pytest-django in place of native django test. I don't know if there are places where they won't be entiriely compatible, but so far seems to be working for the several projects I've tried.

[jmtb28]

When you guys get started on working on this issue, please add the automatic setup of the test environment and test databases like the python manage.py test would.

[evenicoulddoit]

I thought I'd just follow up to my previous comment re: having a pre-test setup. By doing things the "pytest way" I was actually able to achieve exactly what I wanted. Just as a pointer:

• By leveraging pytest fixtures with a session scoping, we were able to set-up and teardown a database as necessary
• In addition, we used the following plugins to get us to a good state
  • pytest-env we were able to automatically set the necessary testing environment variables
  • pytest-flask-sqlalchemy allowed us to run tests within transactions
  • pytest-xdist allowed us to run tests multi-threaded

[AndreaCitrolo]

Honestly, I think forcing pytest use for django is not the right way to go. Indeed, unittest is the standard and the one "officially" supported by django. I think a better solution would be adding an option in the python extension to use a custom test command such python.customTestCommand or python.djangoTestCommand (this could be set to "python manage.py test"). The output could be parsed using the existing unittest bindings.

[slavpetroff]

Its a shame that for 3 years no one of the Microsoft team decided to put some effort in making the VSCode way more popular than PyCharm by simply making it possible to run Django tests. Which i am sure is no more than 1 week job ( at most ). Long time ago ive adapted a plugin to simply call django.setup() on the test start and tear down on test end. It took me 1 day. Cant imagine how long it will take you guys.

[brettcannon]

@slavpetroff Please be careful about making assumptions about how long it would take to make such a change. There's UX, testing, etc. that would have to go into this which does not make it a simple one-liner fix. But if you would like to try and come up with a design for this and a subsequent PR then you are welcome and we will review it.

[Cetlan]

I took a glance at this over the weekend, and can definitely confirm it's not a one-line change. Besides the new test runner objects, and the hundred or so places it looks like I'd have to connect the new objects into the existing code, I think the django test might have to work differently from any of the existing test runners in terms of launching (I don't really understand all the code yet, but I think the existing ones don't need to run an external python script), but also django tests can use different runners (django's unittest based one, nose, or pytest) which will definitely need some extra design and/or UX.

This looks much bigger than I would have assumed before looking.

Is there some good documentation somewhere on the design of the test managers? Or just digging through the code like I've been doing to figure it out?

[brettcannon]

@Cetlan unfortunately there's no specific docs on the design, but you can feel free to ask here or on Discord at https://aka.ms/python-discord-invite.

[Cetlan]

I managed to find a little more time last weekend to look into this a bit further. I see that the unittest version of the testing code runs via visualstudio_py_testlauncher.py, and routes the test information back via sockets. Does this imply that output from python -m unittest -v [...] isn't enough to display proper test results and details?

If that's the case, then I think to make this work with basic django tests we'd need a custom django test runner. It should be possible to run manage.py from the vcode-python testing framework without too many changes, and django_nose.NoseTestSuiteRunner might be able to work mostly out-of-the-box because it can produce xunit output, which the vscode-python nose runner already deals with. But for the standard django tests, we'd need either a replacement for manage.py test similar to how visualstudio_py_testlauncher.py replaces python -m unittest, or a custom runner that provides the hooks like visualstudio_py_testlauncher.py, or xunit output, etc, but can be injected into the tests with mange.py test --testrunner <..>

[kavdev]

@Cetlan this might be useful: https://github.com/xmlrunner/unittest-xml-reporting#django-support. I had to tweak the settings to output a single xml file for compatibility with our ci/cd pipeline, but it pretty much works right out of the box.

[dalys]

I'm also eager for a real integration to run Django tests! In the meanwhile, if anyone just wants to run Django tests without the need for parsing the results, I use this VS Code extension with great success: https://github.com/Pachwenko/VSCode-Django-Test-Runner

Just remember to set up the prefix command setting to cd into the correct directory.

These are my settings. With the --keepdb, this extension is a great enabler for my workflow as it makes it really fast to execute a single or a few tests. I never run the whole test suite locally anyway because it takes too much time.

Hope this can help someone.

[theoctober19th]

For anyone who stumbles upon this, I managed to get around this issue by adding import django and django.setup() at the top of all my test cases.

[niccolomineo]

Any news about officially supporting Django unittest?

[djbrown]

Maybe the Python Extension testing revamp helped? Couldn't test that yet myself...

[niccolomineo]

@djbrown I am afraid those changes didn't fix the Django testing experience.

[brettcannon]

No news I'm afraid. We are still working through more low-level/fundamental issues in our testing code before we can tackle something like this.

[niccolomineo]

@brettcannon I hope you will place this request in the backlog, as you already provide explicit support to Django in the launch.json.

[b-long]

Note: The StackOverflow discussion here has workarounds for Bash (macOS/Linux) and Git-Bash (Windows): https://stackoverflow.com/questions/56858808/how-do-i-debug-individual-django-tests-in-vscode .

Quoting that discussion

Niels writes:

If you're on a Mac or Linux, the following launch config should work for a single unit test executed by Django:

    {
        "name": "Python: Django Debug Single Test",
        "type": "python",
        "request": "launch",
        "program": "${workspaceFolder}/manage.py",
        "args": [
            "test",
            "`echo -n ${relativeFileDirname} | tr \/ .`.${fileBasenameNoExtension}"
        ],
        "django": true
    },

Arenwino writes:

If you're using Windows and the git bash, the launch config with tr will also work, but you'll need to quote the variable substitution and quote the double backslashes.

    {
        "name": "Python: Django Debug Single Test",
        "type": "python",
        "request": "launch",
        "program": "${workspaceFolder}/manage.py",
        "args": [
            "test",
            "`echo -n \"${relativeFileDirname}\" | tr \\\\ .`.${fileBasenameNoExtension}"
        ],
        "django": true
    },

[Josephkready]

Here is generic way to get Django tests to run with full vscode support

1. Configure python tests
   1. Choose unittest
   2. Root Directory
   3. test*.py
2. Then each test case will need to look like the following:

   
   from django.test import TestCase

class views(TestCase):

@classmethod
def setUpClass(cls):
    import django
    django.setup()

def test_something(self,):
    from user.model import something
    ...

Any functions you want to import **have** to be imported inside the test case (like shown). The setUpClass runs before the test class is setup and will setup your django project. Once it's setup you can import functions inside the test methods. If you try to import models/views at the top of your script, it will raise an exception since django isn't setup. If you have any other preinitialization that needs to run for your django project to work, run it inside `setUpClass` 

[midouest]

Note: The StackOverflow discussion here has workarounds for Bash (macOS/Linux) and Git-Bash (Windows): https://stackoverflow.com/questions/56858808/how-do-i-debug-individual-django-tests-in-vscode .

This workaround has stopped working for me on macOS (just noticed today). It seems like VSCode now escapes the pipe character, but I don't remember exactly what the launcher command output looked like when it was working.

/usr/bin/env /path/to/venv/python /path/to/.vscode/extensions/ms-python.python-2022.6.2/pythonFiles/lib/python/debugpy/launcher 53076 -- /path/to/manage.py test --settings example "`echo -n path/to/example \| tr /.`.test_example"
System check identified no issues (0 silenced).
E
======================================================================
ERROR: path/to/example | tr /  (unittest.loader._FailedTest)
----------------------------------------------------------------------
<TRACEBACK OMITTED>
ImportError: Failed to import test module: path/to/example | tr /

[dmastylo]

@midouest running into the same issue as well.

[amirsalaar]

Here is generic way to get Django tests to run with full vscode support

1. Configure python tests

   1. Choose unittest
   2. Root Directory
   3. test*.py
2. Then each test case will need to look like the following:

from django.test import TestCase

class views(TestCase):

    @classmethod
    def setUpClass(cls):
        import django
        django.setup()

    def test_something(self,):
        from user.model import something
        ...

Any functions you want to import have to be imported inside the test case (like shown). The setUpClass runs before the test class is setup and will setup your django project. Once it's setup you can import functions inside the test methods. If you try to import models/views at the top of your script, it will raise an exception since django isn't setup. If you have any other preinitialization that needs to run for your django project to work, run it inside setUpClass

This no longer works since VSCode for some reason doesn't perform the bash inline string manipulation anymore!

[kalsky]

Note: The StackOverflow discussion here has workarounds for Bash (macOS/Linux) and Git-Bash (Windows): https://stackoverflow.com/questions/56858808/how-do-i-debug-individual-django-tests-in-vscode .

This workaround has stopped working for me on macOS (just noticed today). It seems like VSCode now escapes the pipe character, but I don't remember exactly what the launcher command output looked like when it was working.

/usr/bin/env /path/to/venv/python /path/to/.vscode/extensions/ms-python.python-2022.6.2/pythonFiles/lib/python/debugpy/launcher 53076 -- /path/to/manage.py test --settings example "`echo -n path/to/example \| tr /.`.test_example"
System check identified no issues (0 silenced).
E
======================================================================
ERROR: path/to/example | tr /  (unittest.loader._FailedTest)
----------------------------------------------------------------------
<TRACEBACK OMITTED>
ImportError: Failed to import test module: path/to/example | tr /

I documented another workaround in the comments of that SO page (https://stackoverflow.com/questions/56858808/how-do-i-debug-individual-django-tests-in-vscode/72666662#72666662). (in short, handling the relative file name inside manage.py)

BTW, I don't think it's the pipe that is escaped, even using something simple as `echo -n \"${relativeFileDirname}\"` will result in an empty value, so I think it's the tick that is being handled differently now.

[yulqen]

Any update? It would be great not to have to rely on including import django; django.setup() in all my test classes, nor requiring pytest as a test runner.

[brettcannon]

No update. We will post here when there's something to share.

[yulqen]

Thank you.

[mh-firouzjah]

@PlugaruT Environment variables can be set for the workspace. Just create a file named .env in your workspace root and add the necessary variables in there.

Hi there, I've created the .env file and added this DJANGO_SETTINGS_MODULE="my_project.settings" it's working but not correctly. as it will find part of the test files not all of them. I've found that for the apps that their tests aren't found there is an error:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
so is there anything else that I had to put in .env or anywhere else?

[Cetlan]

@mh-firouzjah, you also have to make sure that django.setup() is called before your tests run. That's what loads the apps into the django app registry, which django requires before using any of the models. I don't think anyone really has a good solution for calling that, which is why some people have suggested adding it to the test files, I've suggested a workaround with pytest-django, and this feature request exists in the first place.

The only other idea I've had meanwhile is to maybe hack a django.setup in through sitecustomize - but that'll force it to run all the time, which django itself might have a problem with, since it's not meant to be called more than once.

If you did already call django.setup, then I'm as stumped as you are.

[mh-firouzjah]

@Cetlan I did call django.setup() but this wasn't enough. again some of tests for some apps still not found and the error of apps aren't loaded was shown. I think it maybe important where to place it.

[DeeeeLAN]

I was able to get my tests partially working by calling django.setup() at the top of my setUpClass method. However, I attempt to initialize a couple models in setUpTestData() and that is pulling tables from the live database instead of the test database when run through VS Code. When I call it through manage.py it works properly. I call super().setUpClass() after django.setup(), which is what triggers the setUpTestData() call. I wonder if there is another method I need to call to get the database properly cloned.

Edit: Actually, I just examined further and it looks like all the model imports are grabbing the production db, even inside explicit test methods. So this solution is only viable for non-db tests.

[zachdrever]

I spent quite a bit of time on this in the last couple of days and I think I've got an alright solution. I wanted the same Django behaviour when running tests from the VSCode test explorer (setup the testing environment / create and destroy the database). Couple of other things - I didn't want to run setup / teardown methods on a class basis(via setUpClass() or other) and I also didn't want to alter the behaviour of Django manage.py test. unittest 3.1 provides hooks for injecting code on when any test is about to run or after all tests are executed. Additionally, Django exposes utilities for setting up / tearing down the test environment and database.

In my project, I have multiple apps with separate test dirs. At the root of every test dir I included the following in the root __init__.py:

from django.conf import settings

if not settings.configured:
    import unittest
    import os
    from django import setup

    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "example.settings")
    setup()

    from django.test.utils import (
        setup_test_environment,
        teardown_test_environment,
        teardown_databases,
        setup_databases,
    )

    VERBOSITY = 1
    INTERACTIVE = False

    def djangoSetUpTestRun(self):
        setup_test_environment()
        self._django_db_config = setup_databases(VERBOSITY, INTERACTIVE)

    def djangoTearDownTestRun(self):
        teardown_databases(self._django_db_config, VERBOSITY)
        teardown_test_environment()

    setattr(unittest.TestResult, "startTestRun", djangoSetUpTestRun)
    setattr(unittest.TestResult, "stopTestRun", djangoTearDownTestRun)

Unfortunately - this will still run the setup and teardown for every sub module. Wondering if leveraging load_tests protocol would allow discovery / packaging of all tests into a single unittest.TestSuite so that we could run setup and teardown only once per test package in VSCode test runner?

[DeeeeLAN]

What's the drawback of running the setup/teardown methods on a class basis? If that is the only drawback and otherwise you have the system functional inside vscode, that sounds like a worthwhile tradeoff to me.

[zachdrever]

For large test suites, running setup and teardown methods per module potentially adds a lot of overhead, depending on how many modules you have. It is possible to pass keepdb=True to the Django test setup utilities, so that would limit some of the overhead.

But yes, it does work in VSCode (forgive my failing test case).

[chatne]

I have finally figured out a way to make unittests work with VSCode and Django with only one python file and without any additional configuration or setup.

VSCode test runner does test discovery every time, regardless of the amount of tests that are about be run. We can leverage this discovery by running Django test setup during this phase. File or directory name for the setup file must be set so that the discovery finds it first. I use a directory in my project root called _unittest.

This is only tested on Ubuntu, also you must update filepaths to reflect your environment.

_unittest/__init__.py

"""
Hacks to make unittest work desirably.

VSCode + Python + Django + unittest integration is currently broken.
Django tests can be run using `python3 manage.py test` but this is not enough as VSCode unittest
runner only uses `python3 -m unittest`.

The purpose of this file is to simulate what the manage script is doing before any tests are run.
The trick is to make this load first when unittest does test discovery, hence the path `_unittest`.

This clearly cannot work when directly pointing to a test since test discovery is not used,
for example `python3 -m unittest server.component.test`.
However VSCode points to target tests differently and discovery still gets run making this work.

This all hopefully becomes obsolete if Django support is added to VSCode Python.
The issue: https://github.com/microsoft/vscode-python/issues/73
"""

import inspect
import os

import django
from django.conf import settings
from django.test.utils import get_runner

def _check_stack() -> bool:
    """
    Dig stack for the filepath of the script that imports this file.

    Return True if improting and invoking scripts are as expected.
    """
    # Skip first two calls which are this module and this function
    for frame in inspect.stack()[2:]:
        # Skip internal libraries which start with <
        if frame.filename[0] != "<":
            importing_script = frame.filename
            break
    else:
        raise RuntimeError("Importing script not found")

    # Unittest loader should be used
    if importing_script != "/usr/lib/python3.11/unittest/loader.py":
        return False

    # Django manage script uses the same loader, so check for that as well and skip if used
    first_invoking_script = inspect.stack()[-1].filename
    if first_invoking_script == "/workspace/backend/manage.py":
        return False

    return True

# This script should only be imported, so ignore direct execute
if __name__ == "__main__":
    pass

# Only allow this hack to work with unittest loader
elif _check_stack():
    # Setup Django for testing
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.testing")
    django.setup()
    settings.ALLOWED_HOSTS += ["testserver"]

    # Setup test database for testing
    get_runner(settings)(interactive=False).setup_databases()

[carltongibson]

Hi. I raised this on mastodon, and @brettcannon suggested to post here.

I'm trying work out what we're missing from Django in order for this to work. The django-admin test command wraps a DiscoverRunner (src) that's based on (or uses) unittest, so the thought is it should work, and I wasn't able to see exactly what's missing.

I asked:

Is this because we need a --discover flag to the management command or something...?

…to which Brett pointed out that you're calling unittest API via Python, and not the CLI, so it becomes, roughly, what API are we missing from DiscoverRunner that would enable test discovery to work here? Presumably we need to be able to call into build_suite…🤔 — I guess my question is, what would this take from VS Code's end? What does it need to look like? If it's just exposing the right API, then surely we can get this working — or is there a structural issue we'd need to resolve first?

Any insight welcomed. Thanks.

[mh-firouzjah]

maybe it would be possible to patch the extension so it could check if a manage.py file is present in the root folder, then use it to run the tests else do the old way(current way)