`tool.pysen.lint.source` may not work

[xuzijian629]

Problem

I could not exclude a specific path when I run mypy. I think I can configure this by setting tool.pysen.lint.source properly, but it didn't work. I checked the generated pyproject.toml which linters use, and found that the information I specified in tool.pysen.lint.source didn't exist there.

How to reproduce

1. Create a pyproject.toml following the example on https://github.com/pfnet/pysen#configuration-model

   
   [tool.pysen]
   version = "0.10"

[tool.pysen.lint] enable_black = true enable_flake8 = true enable_isort = true enable_mypy = true mypy_preset = "strict" line_length = 88 py_version = "py37" isort_known_third_party = ["numpy"] isort_known_first_party = ["pysen"] mypy_ignore_packages = ["pysen.generated.*"] mypy_path = ["stubs"] [[tool.pysen.lint.mypy_targets]] paths = [".", "tests/"]

[tool.pysen.lint.source] includes = ["."] include_globs = ["*/.template"] excludes = ["third_party/"] exclude_globs = ["*/_grpc.py"]

[tool.pysen.lint.mypy_modules."pysen.scripts"] preset = "entry"

[tool.pysen.lint.mypy_modules."numpy"] ignore_errors = true

2. Run `pysen generate hoge`

Then, `hoge/pyproject.toml` looks like

[tool.black] # automatically generated by pysen line-length = 88 target-version = ["py37"]

[tool.isort] # automatically generated by pysen default_section = "THIRDPARTY" ensure_newline_before_comments = true force_grid_wrap = 0 force_single_line = false include_trailing_comma = true known_first_party = ["pysen"] known_third_party = ["numpy"] line_length = 88 multi_line_output = 3 use_parentheses = true

and `hoge/setup.cfg` looks like

[flake8]

automatically generated by pysen

e203: black treats : as a binary operator

e231: black doesn't put a space after ,

e501: black may exceed the line-length to follow other style rules

w503 or w504: either one needs to be disabled to select w error codes

ignore = E203,E231,E501,W503 max-line-length = 88 select = B,B950,C,E,F,W

[mypy]

automatically generated by pysen

check_untyped_defs = True disallow_any_decorated = False disallow_any_generics = False disallow_any_unimported = False disallow_incomplete_defs = True disallow_subclassing_any = True disallow_untyped_calls = True disallow_untyped_decorators = False disallow_untyped_defs = True ignore_errors = False ignore_missing_imports = True mypy_path = stubs no_implicit_optional = True python_version = 3.7 show_error_codes = True strict_equality = True strict_optional = True warn_redundant_casts = True warn_return_any = True warn_unreachable = True warn_unused_configs = True warn_unused_ignores = False

[mypy-numpy]

automatically generated by pysen

ignore_errors = True

[mypy-pysen.generated.*]

automatically generated by pysen

follow_imports = skip ignore_errors = True

[mypy-pysen.scripts]

automatically generated by pysen

disallow_untyped_calls = False disallow_untyped_defs = False warn_return_any = False

In the above example, `[tool.pysen.lint.source]` information is lost.

## Environment
- macos 12.0.1
- pysen version 0.10.1

```bash
$ python
Python 3.8.5 (default, Sep  8 2020, 17:55:39) 
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

[bonprosoft]

Thank you for creating the issue!

As you know, pysen generates a configuration file for each linter. However, since some parameters for linters are only accepted via command line, pysen also executes linters with the appropriate command-line arguments when pysen run lint is called; tool.pysen.lint.source is one of the parameters. Thus, it doesn't exist in generated configuration files.

I think I can configure this by setting tool.pysen.lint.source properly

Unfortunately, mypy doesn't support tool.pysen.lint.source setting because of the limitation of mypy(*), while most tools (isort, black, flake8, and other pysen plugins) support. Please use tool.pysen.lint.mypy_targets setting instead to configure the target source for mypy.

Here is an example:

[[tool.pysen.lint.mypy_targets]]
  paths = ["packageA/src/", "packageA/setup.py"]
[[tool.pysen.lint.mypy_targets]]
  paths = ["packageB/src/", "packageB/setup.py"]

# ignore errors of modules under `packageB/src/generated` directory
[tool.pysen.lint.mypy_modules."packageB.generated"]
  ignore_errors = true

(*) mypy does not take each module path but the whole package directory in the command line to guess the type information properly.

[xuzijian629]

Thank you for the clear explanation! I see [[tool.pysen.lint.mypy_targets]] do the work.

One more question: Is there any way to exclude files from mypy targets? This is useful when I have a tree like below.

.
├── a.py
├── b.py
├── c.py
├── build/
└── src/

If I use [[tool.pysen.lint.mypy_targets]], then I have to write

[[tool.pysen.lint.mypy_targets]]
  paths = ["src/", "a.py", "b.py", "c.py"]

and this seems not very good because whenever I add some file like d.py, e.py at the root, I have to add these files to mypy setting.

Instead, I would like to configure mypy to just exclude build/.

I found from mypy docs how to exclude files from mypy targets like

[tool.mypy]
exclude = [
  "^file1\\.py$",  # TOML's double-quoted strings require escaping backslashes
  '^file2\.py$',  # but TOML's single-quoted strings do not
]

, but I don't know how to integrate this setting into pysen's one.

[bonprosoft]

I'm so sorry for my late response.

We prefer convention over configuration, so I'd rather not create a new pysen configuration as much as possible.

In your case, it might be clearer to create a new directory like scripts/ for such modules and use the simple pysen config:

[[tool.pysen.lint.mypy_targets]]
  paths = ["src/", "scripts/"]

If it doesn't work for you, I think we have three options:

1. Create a new pysen plugin to configure the tool.mypy.exclude items
2. Use tool.pysen-cli to merge your custom mypy config into the generated one by pysen

   [tool.pysen-cli]
   settings_dir = "."

3. Create a new pysen config to customize tool.mypy.exclude item

Do you have any opinion? @sergeant-wizard @Hakuyume

[xuzijian629]

Hi @bonprosoft,

In your case, it might be clearer to create a new directory like scripts/ for such modules and use the simple pysen config

This sounds like a good solution!

[bonprosoft]

Thank you! I'll close this issue, but feel free to reopen if you have any further issues :)