(🎁) Improve usability of `--install-types`

[KotlinIsland]

Feature

mypy --install-types requirements.txt will install type stubs of all dependencies in the file.

Pitch

I can't see a generic way to set up an environment ahead of time.

[JelleZijlstra]

Another useful addition could be to make mypy emit the libraries it would install, so you can do something like mypy --types-to-be-installed >> requirements.txt.

[mikepurvis]

Just to add to this, the --install-types flag only seems to work after there was a previous run to populate the mypy cache. This leads to the unfortunate situation in CI of having to basically do:

yes | mypy src --install-types || true
mypy src

Which seems super gross and leads to a jumbled and confusing console output with errors followed by a clean run.

IMO if --install-types doesn't have a cache to go on, and can't discover the needed types using the package dependencies or a requirements file, it would be great to have a CI-friendly mode where it can run only the dependency-tracing and not emit any errors (and not need a y prompt to pip!).

[JukkaL]

I wouldn't recommend running --install-types as it currently works in CI, since in the worst case it can almost double the mypy runtime. It also produces noisy output, as mentioned above.

Right now this is possible (as a one-time thing -- commit the changes to requirements.txt):

mypy --install-types
pip freeze | grep '^types[-]' >> requirements.txt

This isn't very intuitive, however. I can see how some projects would also prefer not to maintain stub requirements explicitly and are happy to use the latest versions of all available/known stub packages.

I'm trying to summarize the ideas above below. I can see several somewhat different but related use cases. The option names are open to bike shedding.

Use case 1: Use requirements.txt

Infer types from requirements.txt instead from import dependencies. This could by supported via mypy --install-types -r requirements.txt, for example (or --requirement requirements.txt). This would still produce an interactive prompt by default.

This could be used both as a one-off action and as part of every CI run. This would always install the latest stubs.

Example:

$ cat requirements.txt
requests==x.y.z
$ mypy --install-types -r requirements.txt
Installing stub packages:
python3 -m pip install types-requests

Install? [yN]

Use case 2: Install types non-interactively

Unconditionally install type packages and don't ask for confirmation. If type checking (i.e. not using requirements.txt), this could also silence normal error output about missing stubs. This could be be supported via mypy --install-types --non-interactive, for example, possibly together with -r requirements.txt.

Use case 3: Generate requirements output

Instead of installing stubs, produce output suitable for requirements.txt. This could be supported via mypy --types-requirements. This option would imply --non-interactive. This would also support -r.

Here's how this could look like (note no error output about missing stubs):

$ mypy --types-requirements src/
types-emoji>=0.1.0
types-requests>=0.1.0

I'm not sure whether this should look up the latest versions of stub packages and use types-<foo>==<latest_version>. We shouldn't perhaps include any type packages that are already installed.

[dcfranca]

Is it possible to run mypy install-types on a CI environment? I have tried, but it gets stuck on the interactive part... is there workaround available or we will have to wait for the --non-interactive flag to be implemented?

[mikepurvis]

In the absence of a recommendation to use --install-types in CI, is the suggestion to add the types packages to your main requirements.txt? Or a separate mypy-requirements.txt or types-requirements.txt? Or do you make them setuptools install requirements? Or add them to an extra_requires like types?

It'd be great to have some guidance from the project on this.

[dcfranca]

@mikepurvis we have added to the requirements.txt, just wondering if there is other solution to not have to do this on all our projects.

[JukkaL]

My recommendation is to do one of these (each has different tradeoffs):

1. Add type dependencies to your main requirements.txt
2. Add type dependencies to a separate mypy-requirements.txt or types-requirements.txt (this is preferred if you already have a dedicated requirements file for mypy)
3. Create a types-requirements.txt file and use -r types-requirements.txt to include it in another requirements file (or many files across multiple projects)

The --non-interactive flag should be easy to implement. Since this seems to impact a lot of projects, I'm leaning towards making a 0.910 release with the --non-interactive flag within the next week or so.

If/when 0.910 is out, you'd also have the option of running something like mypy --install-types --non-interactive src/ ... in your CI scripts before you invoke mypy to actually type check your code. This would simplify the maintenance of your dependencies, as you'd always get the latest stubs, but it would slow down your CI at least a little since you'd need to run mypy twice. Also your build could start failing because of changes to stubs, but if you are already not pinning to a particular mypy version you probably don't care about this much.

Please let me know if none of the above options work for you. Generating type requirements from your main requirements file (use case 1 above) is more effort to implement, so we may not have it available soon, unless somebody would like to contribute it.

[rafaellehmkuhl]

The --non-interactive seems that would solve most the problems.

[j616]

I agree. Non-interactive would return to something along the lines of the original behaviour, admittedly with the interface change in the form of the new flags, and would work for my team's workflows. We've currently had to pin back mypy on all of our repos because the lack of a non-interactive mode. It's broken CI for us.

[chorner]

I'd like to have some confidence that the package matches the type definition i'm downloading. Does typeshed not provide a mechanism for mapping package versions to type package versions?

With version pinning, it seems inevitable that if it isn't automated the versions will one day fall out of sync... crippling the value of type checking. So i'd say making it non-interactive isn't enough, it also needs to generate the version of the type packages it is going to install.

[JukkaL]

--non-interactive is now supported on git master. It would be great if some of you could try it out before I make the 0.910 release.

@chorner Typeshed supports defining the target library version that stubs support (in METADATA.toml). It's reflected in the version of the types package on PyPI. Since it hasn't been filled in for most stubs, it's not very useful yet. This is still better than what we used to have before, as previously there was no support for specifying the supported package versions.

Once typeshed has more dependable version information, at least the proposed variant of --install-types that looks at target packages in a requirements file could be able to make sure the installed type package is compatible with the installed version of the package.

For example, if we have versions 1.5.2 and 2.0.4 of types-foobar on PyPI, and you have foobar==1.4.2 in your requirements.txt, mypy --install-types -r requirements.txt would install version 1.5.2 of types-foobar. But if you have foobar>=2.0, we'd install version 2.0.4.

[mikepurvis]

Is the expectation that separate types stub packages are a long term thing? I kind of assumed they were a 12-24 month bridge and the hope in the end is that most popular dependencies would supply their own type information, at least at the API boundaries. I suppose the lesson of Python 3 is not to assume that any hack is a short-term thing.

[JukkaL]

I expect that types stub packages will be around for a long time, but hopefully they will be needed much less frequently in the future. It's not really something we can control, since the decision to bundle stubs or include inline annotations is up to individual project maintainers.

Making the workflows not suck is thus pretty important.

[larroy]

Seems install-types doesn't work when there's no cache directory:

mypy --install-types
Error: no mypy cache directory (you must enable incremental mode)

[john-bodley]

Somewhat related per https://github.com/pre-commit/mirrors-mypy/issues/50 is there merit in having the --install-types run pre-execution (rather than post per here) so it could be run once (inlining the installation), i.e.,

mypy --install-types --non-interactive program.py

rather than via two-steps which is non-viable when using with pre-commit, i.e.,

mypy --install-types --non-interactive 
mypy program.py

[asottile]

I expect that types stub packages will be around for a long time, but hopefully they will be needed much less frequently in the future

I kinda have the opposite hope -- take for example setting up a separate type checking environment, I'd rather install a handful of text files (~order of KB) than the actual libraries (~order of MB) especially for libraries with native extensions.

inline types also aren't possible for py_modules based distributions given PEP 561 requires folders. so without a PEP improving that separate stubs will be required indefinitely to satisfy that usecase

[RouquinBlanc]

Same as @larroy , does not work asking for incremental mode, even if explicitly asked (which should be the default mode?)

$ /private/tmp/.tox/mypy/bin/mypy --install-types --non-interactive
Error: no mypy cache directory (you must enable incremental mode)

$ /private/tmp/.tox/mypy/bin/mypy --install-types --non-interactive --incremental
Error: no mypy cache directory (you must enable incremental mode)

[JukkaL]

@larroy @RouquinBlanc Did you actually have some files for mypy to type check? If you just use mypy --install-types without passing any files or directories, mypy will try to use the results of the previous run, which are stored in the cache directory. You can also use files=... in your config file. Alternatively, the directory where you run mypy might be write-only, and mypy can't create cache files.

In any case, the error message is confusing.

[RouquinBlanc]

Hi @JukkaL,

The use case is running mypy with tox in a container. The mypy section was configured as follows:

[testenv:mypy]
basepython=python3.8
deps=mypy
commands=python -m mypy -p {posargs:mypackage}
skip_install=true

After following this ticket I naively tried to modify it like this:

[testenv:mypy]
basepython=python3.8
deps=mypy
commands=
    python -m mypy --install-types --non-interactive
    python -m mypy -p {posargs: mypackage}
skip_install=true

But as you say, because it has not run yet once, it fails... if I manually call mypy --install-types --non-interactive again afterward, then it works, but that's not a desirable way of working to just start by failing.

For those tests we skip installation, and do not have a requirements.txt to work with, and do not require one at that place for various reasons (not telling we do not have one elsewhere, just that for those test we rely on setup.cfg install_requires, and work with bleeding edge).

A very short term quick-fix is to manually define the list of packages which require external types:

[testenv:mypy]
basepython=python3.8
deps=mypy
commands=
    # TODO replace this
    python3 -m pip install types-PyYAML
    python -m mypy -p {posargs:mypackage}

But that's only a quick fix to me... What if tomorrow another dependency needs external types as well?

In that sense, there are 2 propositions above which would make sense for our scenario:

• the option of @john-bodley to run it as a one-liner, installing types on the fly
• or, although less optimum for that setup, the option to pre-generate the list of typing requirements, so that we can install them prior to running mypy as suggested by @JelleZijlstra

[JukkaL]

Hmm it looks like the current behavior still seems somewhat problematic.

What if we'd change --install-types to run the type check again after installing stubs? So mypy --install-types --non-interactive src/ would both install types and produce type checking results, similar to mypy src/ in earlier Python versions.

Currently two runs are needed for the same results:

1. mypy --install-types --non-interactive src/
2. mypy src/

[john-bodley]

@JukkaL I don't fully grasp why the change in how types were installed in Mypy 0.900, but would there be merit in making --install-types and --non-interactive the default (and provide the inverse --no--install-types and --interactive flags respectively) for consistency with versions < 0.900?

Personally I'm a fan of simplify and this means tools like pre-commit et al. (and the numerous repos which have mentioned this issue) would work off the shelf for all versions of Mypy without the need for having to modify their CI.

[larroy]

@larroy @RouquinBlanc Did you actually have some files for mypy to type check? If you just use mypy --install-types without passing any files or directories, mypy will try to use the results of the previous run, which are stored in the cache directory. You can also use files=... in your config file. Alternatively, the directory where you run mypy might be write-only, and mypy can't create cache files.

In any case, the error message is confusing.

Definitely I have a working configuration. The issue only happens when install-types is requested and the project is cleaned up with "git clean -ffdx" or similar which removes the .mypy_cache dir

[JukkaL]

@john-bodley:

but would there be merit in making --install-types and --non-interactive the default (and provide the inverse --no--install-types and --interactive flags respectively) for consistency with versions < 0.900?

I'm reluctant to do this, because of three main reasons.

First, I don't think that installing packages to the current environment is something we should do by default. This should continue to be opt-in behavior, as it's a somewhat unusual result from running a type checker. It would be a pretty drastic change if mypy started automatically installing packages by just updating to 0.9xx.

Second, --install-types --non-interactive is not going to be the best option for many projects, and it's unclear if it's even the best option for most projects. By requiring projects to make a decision about this, instead of providing a default path of least resistance, I hope that more users will choose do something that's best for them. In particular, pinning to particular type package versions is something I'd recommend most projects of significant size to at least seriously consider.

Third, --install-types --non-interactive is also different from pre-0.900 behavior in another important sense: the results are not stable over time. As new versions of type packages are released, output from mypy may vary. Previously installing a specific version of mypy would produce repeatable type checking results, but in 0.900 and later also type package versions should be pinned, not just mypy, for repeatable results.

For many projects using --install-types --non-interactive may be the best option, but I'd rather not make the decision for them -- any default option will likely be adopted by the majority of projects, and may be considered by many as the recommended/blessed way.

I'm aware that this change causes at least one-time friction to many (if not most) projects. I've tried to make the switch as painless as possible (e.g. via --install-types --non-interactive that will be available soon), but a change of this magnitude can't be made completely seamlessly. This should make future mypy updates smoother, since dealing with stub issues will be easier, and I believe that this will be a net win in the long term.

[j616]

I'm slightly confused by this. For mypy to work correctly, won't the matching type packages always have to be installed? Surely a non-matching type package version may lead to incorrect results? Not having type packages installed at all results in failure. If the only way to have mypy work correctly is to have very specific packages installed, I think it makes sense to have mypy verify the correct packages are installed at the correct version. And to install them if not.

I can see an argument for prompting before install by default and having --non-interactive for CI. But for --install-types, I see less of a point. It's trivial to do the correct thing automatically every time. The alternative is have a human do the job/roll their own CI and run the risk of them doing the wrong thing, like let type package and main package versions get out of step.

Am I missing something? I can't see any other mode of operation beyond always making sure you have the matching type packages installed one way or another. You may as well have mypy handle it all the time.

It's also worth remembering that not all python packages follow semantic versioning fully. Even mypy isn't. I know 0.x is often used as pre-release. But in the strictest sense, the interface has been broken in a minor version bump here. The only safe thing to do if you manage type packages in the user's project is make sure that the type and main packages match to the patch release. Otherwise there's a race condition where you pin to e.g. a minor version and you get a different patch version for both the main and type package on install. That's going to be really inconvenient and make mypy more effort than its worth for some. And encouraging people to pin to patch releases will result in many projects falling behind on security updates.

[j616]

I've just found the relevant PEP here. Apparently this is handled by stubs specifying the versions of the runtime package they support in the dependency metadata. So the dep resolver in the package manager should figure out a set that works. https://www.python.org/dev/peps/pep-0561/#stub-only-packages . Though it still leaves the question of if not having these packages installed is an error condition to start with, it seems sensible for mypy to try and install them with a check before installing if necessary. If a project decides to pin these stubs in their requirements, then that step would presumably become a no-op anyway.

[JukkaL]

For mypy to work correctly, won't the matching type packages always have to be installed?

Not really. Users often decide to ignore missing stubs. There are many reasons to do this, such as:

1. The stubs target the wrong version of the package
2. The stubs are incomplete and are missing some definitions
3. The stubs have incorrect types for some definitions, resulting in false positives
4. The user is not aware that stubs exist

All missing third-party type packages can be ignored. For the "legacy" packages that used to have bundled stubs, this needs to be done in the config file in a per-module section, and other packages can be ignored also using --ignore-missing-imports. Even in earlier mypy versions mypy shipped with only a subset of available stubs (those in typeshed). Many other stubs are available outside typeshed. The new behavior makes typeshed stubs less special.

Surely a non-matching type package version may lead to incorrect results?

Yes, but usually library interfaces are pretty stable and it's not a problem to use a somewhat out-of-date (or too recent) version of a type package. Even a matching type package can lead to incorrect results if there are issues in the stubs. In practice type packages aren't updated as often as implementations, so some deviations are common. Still, this is usually not a major problem, in my experience.

If the only way to have mypy work correctly is to have very specific packages installed, I think it makes sense to have mypy verify the correct packages are installed at the correct version. And to install them if not.

Mypy can't always install the correct versions currently, since many type packages are missing version metadata. As I discussed above, this is less of a problem than you might think -- but it can still be a problem. Once we have more complete version metadata, inferring type package version automatically may become feasible.

Right now mypy suggests some type packages to install, but it can't know whether they are compatible with what the user has installed. This is one reason why we don't install them automatically.

However, maybe --install-types should still be the default. It would be with a prompt, since installing packages by default seems like too much. This wouldn't work in CI, where --non-interactive would still be required. Does somebody dislike this option? We'd have --no-install-types for the current default behavior.

Making mypy interactive by default wouldn't work if the output is piped to another process, for example. We might need an isatty() check.

[septatrix]

I would also very much like if mypy defaults to pip install --user if it itself is installed in the user directory:

sh-5.1$ which mypy  # mypy was installed with pip install --user
/home/septatrix/.local/bin/mypy
sh-5.1$ mypy --install-types
Installing missing stub packages:
/usr/bin/python -m pip install types-requests

Install? [yN] N
mypy: Skipping installation
sh-5.1$ /usr/bin/python -m  pip install --user types-requests  # this should be suggested instead

[mitar]

I think this issue shows why having --install-types is probably a case of putting too much logic into one program. Now that --install-types is there, more options had to be added, like --non-interactive. And much more probably will have to (overriding a particular typedefs package, etc.). I think this whole direction should be deprecated and instead only one option should be made: --types-requirements generating a requirements.txt file (option 3 suggested by @JukkaL above) and then you can use whichever tool you like to install those packages, you can modify it, you can use local caches, private package registries, etc.

Also it suggests a stable use pattern: commit this file into your repository so that you get reproducible results. Now this pattern is hard to do (you have to grep from pip3 freeze after installing them). I think it should be the opposite. This pattern should be easy to do and then if you want to make everything automatic, you would just do:

mypy --types-requirements src > types-requirements.txt
pip3 install -r types-requirements.txt
mypy src

inside your CI.

[Psychokiller1888]

Well, afteer reading a few topics about it, the only thing I can conclude with, is that this change completly broke CI support because the stubs don't ship anymore. So I have about 180 repositories to update their CI, but first find a solution to this....

[JakeSummers]

Related issue: #14663

[FeryET]

Is there any way to suppress the "error" or "warning" messages when running mypy --install-types? I want it to install the types in a CI job and don't want it to fail because of mypy errors atm.

[Anton-Aracor]

Is there any way to suppress the "error" or "warning" messages when running mypy --install-types? I want it to install the types in a CI job and don't want it to fail because of mypy errors atm.

I've the same question. At the moment I use this: mypy --install-types --non-interactive But problem it gives an error like this:

error: Can't determine which types to install with no files to check (and no cache from previous mypy run)
Error: Process completed with exit code 2.

If I add the dot in the end, it shows me all the errors in my code from mypy: mypy --install-types --non-interactive

Have no idea how to install reps, and do not run the mypy checks.

[MichaelBonnet]

Is there any way to suppress the "error" or "warning" messages when running mypy --install-types? I want it to install the types in a CI job and don't want it to fail because of mypy errors atm.

I've the same question. At the moment I use this: mypy --install-types --non-interactive But problem it gives an error like this:

error: Can't determine which types to install with no files to check (and no cache from previous mypy run)
Error: Process completed with exit code 2.

If I add the dot in the end, it shows me all the errors in my code from mypy: mypy --install-types --non-interactive

Have no idea how to install reps, and do not run the mypy checks.

I am having this same issue. Hard to know what the proper approach is.

[AntoonGa]

The best I managed to to on my CI is to pre-generate the logs and install the requierements after that.

  # MYPY: checking type def in codebase
  - pip install mypy
  # Run mypy to get stubs requirements. This is because stubs dont ship with packages anymore
  - mypy . || true
  # Extract most stubs from previous mypy log
  - mypy --install-types --non-interactive
   # Run mypy, ignoring missing imports
  - mypy . --ignore-missing-imports

This is slow because mypy needs to run twice, in addition mypy --install-types --non-interactive does not catch all the imports and I have to use the --ignore-missing-imports option.

Is there any better way to implement Mypy in my CI ?

[hauntsaninja]

As documented in https://mypy.readthedocs.io/en/stable/running_mypy.html#library-stubs-not-installed --install-types can be slow because it effectively runs mypy twice. I recommend just collecting the stubs packages it installs into a requirements.txt file that you check into your CI, and then install that with e.g. pip install -r requirements.txt

Also if you're using latest mypy, I recommend --disable-error-code import-untyped as a slightly safer replacement for --ignore-missing-imports

[Simon-Bertrand]

As documented in https://mypy.readthedocs.io/en/stable/running_mypy.html#library-stubs-not-installed --install-types can be slow because it effectively runs mypy twice. I recommend just collecting the stubs packages it installs into a requirements.txt file that you check into your CI, and then install that with e.g. pip install -r requirements.txt

Also if you're using latest mypy, I recommend --disable-error-code import-untyped as a slightly safer replacement for --ignore-missing-imports

Hi, Sorry but I fail to understand something, do we definitively need to add one more file in the project root just to run Mypy in the CI and manage stubs dependencies manually or do we have to run N times Mypy just to get the typed dependencies ?

Maybe I miss something, but it looks very harsh for too few.