Command line `pyright` reports disjoint set of errors from language server pyright

[smackesey]

I have at least one file where the output of pyright /path/to/file.py produces an entirely different set of errors than what the LSP server returns for that file. Both sets of errors are legitimate. Note that:

• the venv python passed to pythonPath to start up the language server is the active venv when running $ pyright /path/to/file
• the language server root directory is the cwd used when running $pyright /path/to/file
• the pyright and pyright-langserver executables used are from the same pyright distribution.

Settings passed to language server (in Lua because this is from neovim):

SETTINGS = {
  python = {
    pythonPath = '.venv/bin/python',
    analysis = {
      autoImportCompletions = true,
      typeCheckingMode = "basic",
      diagnosticMode = "openFilesOnly",
      stubPath = "./typings",
      autoSearchPaths = true,
      -- diagnosticSeverityOverrides = {
      --   
      -- },
      reportUnusedVariable = true,
      useLibraryCodeForTypes = true,
    }
  },
}

And my pyrightconfig.json in the project root:

{
  "exclude": [
    "**/*_tests",
    "**/alembic",
    "**/__pycache__",
    ".venv*"
  ]
}

To Reproduce

This is in a complex codebase, so before attempting to produce a simple reproduction I want to confirm that there's not a simple explanation.

Expected behavior

The diagnostics produced by the language server should straightforwardly match the errors produced by command line pyright.

VS Code extension or command-line

Pyright version 1.1.193, for both the language server (run via Neovim using builtin LSP client) and the command line.

[erictraut]

Yes, if the configuration and settings are the same, the CLI version of pyright should produce the same results as the language server version. Keep in mind that the client settings are not used by the CLI version.

How are you specifying the Python environment when you run the command-line version of pyright? I see that you're specifying it via python.pythonPath in the client. Are you enabling this venv within the shell before you run the CLI version of pyright? The CLI version will use the currently-enabled environment (the one that is invoked by executing python3 from the shell).

I also see that you've enabled a couple of diagnostic checks in the client settings. If you want these to be common to the client and the CLI, you'll need to move these to the pyrightconfig.json file.

[smackesey]

For the command line I am running Python with the virtual environment in .venv activated (so the same venv passed to language server python.pythonPath).

I have partly narrowed the problem down to a subtle difference in type inference that is causing ripple effects. The real code is quite complex but here is a contrived version of what I'm seeing:

### mymod/a.py

from mymod.b import fail

# The inferred return type should be `str`
def f():
    if rand() > 0.5:
        return 'a'
    else:
        fail('blah blah blah')  # fail is a `NoReturn` function

# the language server correctly determines x is `str`
# `$ pyright mymod/a.py` thinks x is `str | None`
# `$ pyright mymod/a.py mymod/b.py` gets it right
x = f()

### mymod/b.py
from typing import NoReturn
def fail(desc) -> NoReturn:
    raise Exception(desc)

So the problem seems to be a difference in pyright's understanding of the fail signature. Both the language server and pyright mymod/a.py mymod/b.py pick up the signature. But if I just run pyright mymod/a.py, it seems pyright can't pick up the signature of fail, therefore doesn't realize that that block always raises an exception, and instead treats it as returning None.

Maybe I misunderstand how command line pyright works, I thought it would read type info frommymod.b even if I don't explicitly pass it on the command line?

[erictraut]

Just to confirm, are you using the same version of pyright in both cases?

You mentioned 1.1.93 in both cases. So I assume you've already thought of that.

[erictraut]

Am I correct in assuming that mymod is a subdirectory in your root project directory, your pyrightconfig.json file is also located in that root project directory, and you are running the CLI version of pyright from within the root directory?

[smackesey]

OK, I've resolved the issue by providing the --lib argument to command-line pyright. I was caught off guard by the fact that pyright by default only looks at installed package source if py.typed is present. I think I understand now: if useLibraryCodeForTypes is true, then pyright will extract types from all available source files regardless of whether py.typed is present. If useLibraryCodeForTypes is false, then it will only look at py.typed packages?

Here's a bit more about the specifics of my situation, because I think a similar problem could bite others. mymod here was a standin for one python package within a large monorepo (Dagster). The actual paths involved were:

• python_modules/dagster/dagster/cli/workspace/cli_target.py (mymod.a)
• python_modules/dagster/dagster/check(mymod.b)

Normally when developing a package, one wouldn't run into this problem for intra-package imports because import rule 3, "Try to resolve code within the workspace", would successfully resolve types defined in the local workspace. However, because of the unusual structure of this monorepo (dagster package not in root directory or src), pyright cannot find the package code for dagster using rule 3 (under default config). So instead it has to find it by rule 4 (installed packages). But despite the fact that dagster has copious inline type info and is installed in the venv, pyright won't use it by default because it doesn't have a py.typed file.

My situation here is a bit niche, but it does seem unintuitive to me that useLibraryCodeForTypes is disabled by default in Pyright but not in Pylance. Is there a reason for this difference? Seems to me that PyLance has it right-- wouldn't most users expect their type checker to leverage any type information available in installed packages, regardless of whether py.typed is present?

[erictraut]

We generally don't recommend using useLibraryCodeForTypes for type checking. It was added specifically for users of pylance who are not interested in static type checking but want completion suggestions and other edit-time conveniences provided by a language server. When useLibraryCodeForTypes is enabled, pyright attempts to infer types from the implementation, but type inference can be inaccurate. Inferred types are good enough for completion suggestions, but they typically lead to many false positives for static type checking. If you are using libraries that have inlined type annotations, then it will tend to work much better, but that case is rare. In most cases, libraries are untyped or fully typed with a "py.typed" designation. Note that other Python type checkers (like mypy) have no equivalence of useLibraryCodeForTypes, which makes sense because they are not designed to be used as a language server.

If you would like pyright to resolve a library in a location other than the root, you can add an extraPaths entry in the pyrightconfig.json file. In your case, you'd want to add python_modules/dagster to the extraPaths array.