Running mypy on doctests

sobolevn commented 5 years ago

Hi!

I am trying to make a custom tool to run mypy on doctests inside my project. Because I want to be sure that my examples are correct. And since pytest / doctest modules do it with ease I am seeking the similar solution but for types. Examples, where I need this:

So, I have started with using api. It works fine, but requires to create temp files with extracted doctest code contents. This is a working solution, but the only ugly part is that it is based on temporary files. And reports incorrect filenames, locations, etc.

I have tried to go deeper and pass just the source code / ast parts. But, it does not look like it works this way. Is there a way to make it work without touching a lot of protected / private APIs?

Maybe this can be implemented in mypy itself at sometime?

Thanks!

JukkaL commented 5 years ago

Using a mypy plugin hook for this may be an option, though no existing hook seems to fit the bill.

The existing API may be enough to get the right kind of output however. One idea would be to include empty lines to align line numbers. Another idea would be to have a postprocessing step that translates the files names in mypy output. It's a bit hacky, but it has the benefit of working right now.

posita commented 3 years ago

I did both (align numbers and translate file names) here. It ain't great (it still uses temporary files), but it works. Bear in mind that I only spent a couple hours hacking this together, so reserve judgment. Improvements are likely many and obvious. Suggestions are invited.

% cat -n doctest.txt
     1  Ready to have your mind blown? Check *this* out!
     2
     3  >>> a = 0
     4  >>> a = "zero"  # <- typing error here
     5  >>> a
     6  'zero'
     7
     8  *Crazy*, right?!
% ./helpers/mypy-doctests.py doctest.txt
doctest.txt:4: error: Incompatible types in assignment (expression has type "str", variable has type "int")
Found 1 error in 1 file (checked 1 source file)

Help is built in:

% ./helpers/mypy-doctests.py --help
usage: mypy-doctests.py [-h] [-a ARG] [-A] [--exclude-dir-names PATTERN]
                        [--include-file-names PATTERN] [--log-level LEVEL]
                        [--tmp-file-suffix SUFFIX] [-k] [-K]
                        PATH [PATH ...]

Extract doctests and check them with mypy.

positional arguments:
  PATH                  paths to check for doctests; PATH can be a file or
                        directory

optional arguments:
  -h, --help            show this help message and exit
  -a ARG, --mypyp-arg ARG
                        append ARG to mypy command
  -A, --clear-mypy-args
                        clear any mypy command arguments
  --exclude-dir-names PATTERN, --exclude PATTERN
                        exclude directories matching PATTERN from inspection
                        (default is "\A(\..*|__pycache__|node_modules|site-
                        packages)\Z")
  --include-file-names PATTERN
                        include files matching PATTERN (default is
                        "\.(md|py|pyi|rst|txt)\Z")
  --log-level LEVEL     set logging verbosity to LEVEL (default is WARNING)
  --tmp-file-suffix SUFFIX
                        use SUFFIX when creating temporary files (default is
                        ".doctest.py")
  -k, --keep-temp-files
                        keep temporary files on exit (e.g., for debugging or
                        inspection)
  -K, --no-keep-temp-files
                        remove temporary files on exit (default)

python / mypy

Running mypy on doctests #7966