search

microsoft / pyright

Static Type Checker for Python
Other
13.52k stars 1.49k forks source link

Cannot resolve google.cloud imports when types-protobuf are installed #2113

Closed nullie closed 3 years ago

nullie commented 3 years ago

Describe the bug When types-protobuf stub is installed, google.cloud imports cannot be resolved. I don't think it's a stub bug, because mypy seems to handle this right.

To Reproduce In a fresh venv:

pip install types-protobuf
pip install google-cloud-firestore
cat > test.py
from google.cloud import firestore
^D
npx pyright --verbose

No configuration file found.
No pyproject.toml file found.
stubPath /home/nullie/work/pyright-bug/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /home/nullie/work/pyright-bug
  /home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stdlib
  /home/nullie/work/pyright-bug
  /home/nullie/work/pyright-bug/typings
  /home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib/python3.8
  /usr/lib/python3.8/lib-dynload
  /home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages
Searching for source files
Auto-excluding /home/nullie/work/pyright-bug/venv
Found 1 source file
Could not import 'google.cloud' in file '/home/nullie/work/pyright-bug/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/home/nullie/work/pyright-bug/typings'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/typings'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/typings'
  Looking in root directory of execution environment '/home/nullie/work/pyright-bug'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug'
  Looking in python search path '/usr/lib/python3.8'
  Attempting to resolve using root path '/usr/lib/python3.8'
  Attempting to resolve using root path '/usr/lib/python3.8'
  Looking in python search path '/usr/lib/python3.8/lib-dynload'
  Attempting to resolve using root path '/usr/lib/python3.8/lib-dynload'
  Attempting to resolve using root path '/usr/lib/python3.8/lib-dynload'
  Looking in python search path '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages'
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages'
  Resolved import with file '/home/nullie/work/pyright-bug/venv/lib/python3.8/site-packages/google-stubs/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/home/nullie/work/pyright-bug/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/home/nullie/work/pyright-bug/test.py
  /home/nullie/work/pyright-bug/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
1 error, 0 warnings, 0 infos
Completed in 0.619sec

Expected behavior No errors

Screenshots or Code from google.cloud import firestore

VS Code extension or command-line Running pyright 1.1.157 with npx

Additional context Add any other context about the problem here.

erictraut commented 3 years ago

I'm receiving a similar error message from mypy:

test13.py:1: error: Skipping analyzing "google.cloud": found module but no type hints or library stubs
test13.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

The problem is that the types-protobuf library is installing a directory called google-stubs. PEP 561 indicates that when a stub package is present, it should be used for type information even if the corresponding implementation library is also present. PEP 561 also provides a way for a stub library to be marked as "partial" using a py.typed file that contains the word partial in it. The types-protobuf library contains no such py.typed file. If I add one, it works as expected. So this appears to be a bug in the types-protobuf stub library. Please report the bug to the library maintainers.

nullie commented 3 years ago

Okay, thanks for the info

nullie commented 3 years ago

The one thing that confuses me is that currently typeshed has no py.typed files

erictraut commented 3 years ago

The typeshed stubs are not "stub-only packages" as defined by PEP 561 — i.e. they're not installed directly in the site-packages directory of your Python environment. Plus, they are not partial. For those reasons, they don't need a py.typed file.

romanofski commented 3 years ago

I wanted to fix https://github.com/python/typeshed/issues/5800 and came across a problem I think is more suited to be commented here rather than on the typeshed issue.

I actually tried out the attempt with the partial py.typed file using the reproducer from here. I don't think this will help us unfortunately if we're dealing with more google packages. It does fix the immediate issue with the two packages (google.cloud.firestore and google.protobuf), however if we add a third google namespace package we're in trouble again:

The partial is in the root of the installed types-protobuf package in my site-packages

goldie :: lib/python3.9/site-packages % tree -L 1 google-stubs                                                                      1 ↵
google-stubs
├── cloud
├── __init__.pyi
├── METADATA.toml
├── protobuf
└── py.typed <-- "partial"

2 directories, 3 files

Using the given reproducer:

$ cat test.py                                                                                                 
from google.cloud import firestore
from google.protobuf import __version__

# I hacked up the __version__ in google-stubs/cloud/__init__.pyi and changed it to str to avoid the checker inferring the type 
# (which is correctly `bytes`) to see whether it's actually picked up.
reveal_type(__version__)

and run it against pyright:

% foo/node_modules/.bin/pyright --version                                                                     
pyright 1.1.160
% foo/node_modules/.bin/pyright test.py                                                                       
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Searching for source files
Found 1 source file
/tmp/test.py
  /tmp/test.py:4:13 - info: Type of "__version__" is "str"
0 errors, 0 warnings, 1 info 
Completed in 0.563sec

No trouble. I install another google.cloud package generated from using the stub_uploader in typeshed-internals (see https://github.com/python/typeshed/issues/5800#issuecomment-894602427):

(python3) goldie :: /tmp % pip install tmp4_qz07i1/                                                                                    
Processing ./tmp4_qz07i1
Requirement already satisfied: types-six in ./python3/lib/python3.9/site-packages (from types-google-cloud-ndb==1.9.0.1) (0.1.7)
Using legacy 'setup.py install' for types-google-cloud-ndb, since package 'wheel' is not installed.
Installing collected packages: types-google-cloud-ndb
[...]

And rerun pyright against the reproducer again I have no trouble. However, if I run it against my reproducer from
https://github.com/python/typeshed/pull/5821#issuecomment-893327709 I get:

 % foo/node_modules/.bin/pyright --verbose foo.py                                                           1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud.ndb' in file '/tmp/foo.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/__init__.pyi'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/cloud/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/foo.py
  /tmp/foo.py:1:6 - error: Import "google.cloud.ndb" could not be resolved (reportMissingImports)
  /tmp/foo.py:8:13 - info: Type of "foo.name" is "Unknown"
  /tmp/foo.py:9:13 - info: Type of "foo.key" is "Unknown"
1 error, 0 warnings, 2 infos 
Completed in 0.559sec

it doesn't pick up the types because the protobuf stubs are installed.

Now you might say, hang on a tick here. The partial py.typed needs to go into the sub-directory of google-stubs/protobuf and google-stubs/cloud/ndb respectively. Then I get:

oldie :: lib/python3.9/site-packages % tree -P py.typed google-stubs                                                                  
google-stubs
├── cloud
│   └── ndb
│       └── py.typed
└── protobuf
    ├── compiler
    ├── internal
    ├── py.typed
    └── util

% foo/node_modules/.bin/pyright --verbose test.py                                                          1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
/tmp/test.py
  /tmp/test.py:1:26 - error: "firestore" is unknown import symbol (reportGeneralTypeIssues)
  /tmp/test.py:4:13 - info: Type of "__version__" is "str"
1 error, 0 warnings, 1 info 
Completed in 0.567sec

The __version__ is correctly typed again, but now I have problems importing google.cloud.firestore.

Shouldn't partial be only used for packages which have unfinished stub definitions?

I removed my virtualenv a few times because I ended up confused with the reproducer and the results. I think, uninstalling the 'typeshed-protobuf' package leaves the google-stubs directory behind, confusing the typecheckers. So perhaps I went all the wrong way into the rabbit hole and this is not an issue. If that's the case, please ignore.

erictraut commented 3 years ago

I'm a bit confused about all of the locations where you have type stub files associated with the google namespace. Based on the logs and the other information you've provided above, I think you have google type stubs in three locations:

  1. In your site-packages directory under the google-stubs subdirectory
  2. In your site-packages directory under the google subdirectory
  3. In the typeshed stubs that are packaged with pyright

That is the search order that pyright uses when attempting to resolve an import, consistent with PEP 561.

Looking at the pyright logs above, it doesn't appear to find any google-stubs subdirectory in site-packages. Are you sure that a google-stubs directory existed at the time you ran pyright? The log includes the line Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages' twice in a row. The first time it was looking for google-stubs, and the second it was looking for google. (I just updated the pyright logging messages to make that more clear.)

It doesn't find a google-stubs subdirectory, but it does find stubs in the google directory based on this log message:

Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/init.pyi'

It's finding an __init__.pyi directly under google, which means that google is not a namespace package. Per PEP 420:

Namespace packages cannot contain an init.py

So I see a couple of issues here. First, I'm not sure why pyright is not finding a google-stubs subdirectory. Second, I don't think there should be a google/__init__.pyi file if google is a namespace package.

romanofski commented 3 years ago

I'm a bit confused about all of the locations where you have type stub files associated with the google namespace. Based on the logs and the other information you've provided above, I think you have google type stubs in three locations:

1. In your site-packages directory under the `google-stubs` subdirectory

2. In your site-packages directory under the `google` subdirectory

3. In the typeshed stubs that are packaged with pyright

That is the search order that pyright uses when attempting to resolve an import, consistent with PEP 561.

Looking at the pyright logs above, it doesn't appear to find any google-stubs subdirectory in site-packages. Are you sure that a google-stubs directory existed at the time you ran pyright?

Yes I'm pretty sure. I went back today, removed the virtualenv and recreated it. Then installed google-cloud-firestore and then types-protobuf. Comparing to my initial starting point from my comment above, I do realize that I did make a mistake. The first code area already includes the types-google-cloud-ndb package installed. However, installing types-protobuf brings the __init__.pyi in the root google-stubs folder:

site-packages % tree -L 1 google-stubs                                                                  
google-stubs
├── __init__.pyi
├── METADATA.toml
└── protobuf

1 directory, 2 files

The uploader would generate a similar package for any other google namespace package in typeshed at the moment. I think that has to be addressed in typeshed (see https://github.com/python/typeshed/issues/5800)

The log includes the line Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages' twice in a row. The first time it was looking for google-stubs, and the second it was looking for google. (I just updated the pyright logging messages to make that more clear.)

It doesn't find a google-stubs subdirectory, but it does find stubs in the google directory based on this log message:

Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google/init.pyi'

It's finding an __init__.pyi directly under google, which means that google is not a namespace package. Per PEP 420:

Namespace packages cannot contain an init.py

So I see a couple of issues here. First, I'm not sure why pyright is not finding a google-stubs subdirectory. Second, I don't think there should be a google/__init__.pyi file if google is a namespace package.

I think there must be a problem with the traversal? Because the google directory does not contain the __init__.pyi, google-stubs does:

site-packages % ls google                                                                             
api  api_core  auth  cloud  gapic  logging  longrunning  oauth2  protobuf  rpc  type
site-packages % ls google-stubs                                                                       
__init__.pyi  METADATA.toml  protobuf  py.typed

I actually did remove the google-stubs/__init__.py file just to see whether that makes a difference. I've entirely removed the py.typed file too and there is no difference between it being absent to it being placed into google-stubs/protobuf/ subdirectory:

/tmp % foo/node_modules/.bin/pyright test.py --verbose                                                          1 ↵
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud' in file '/tmp/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/test.py
  /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
  /tmp/test.py:6:13 - info: Type of "__version__" is "bytes"
1 error, 0 warnings, 1 info 
Completed in 0.56sec
erictraut commented 3 years ago

I published pyright 1.1.161, which improves the logging for import failures. Could you please install the updated version of pyright and paste the logged output? Thanks!

romanofski commented 3 years ago

Ok, here we go. I removed the old virtualenv, and recreated a new python3 venv. Installed google-cloud-firestore as well as types-protobuf:

$ foo/node_modules/.bin/pyright --version                                                                  1 ↵
pyright 1.1.161
$ cat test.py
from google.cloud import firestore
from google.protobuf import __version__

reveal_type(__version__)

$ foo/node_modules/.bin/pyright --verbose test.py                                                             
No configuration file found.
No pyproject.toml file found.
stubPath /tmp/typings is not a valid directory.
Assuming Python platform Linux
Search paths for /tmp
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib
  /tmp
  /tmp/typings
  /tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /usr/lib64/python3.9
  /usr/lib64/python3.9/lib-dynload
  /tmp/python3/lib/python3.9/site-packages
Searching for source files
Found 1 source file
Could not import 'google.cloud' in file '/tmp/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/tmp/typings'
  Attempting to resolve stub package using root path '/tmp/typings'
  Attempting to resolve using root path '/tmp/typings'
  Looking in root directory of execution environment '/tmp'
  Attempting to resolve stub package using root path '/tmp'
  Attempting to resolve using root path '/tmp'
  Looking in python search path '/usr/lib64/python3.9'
  Attempting to resolve stub package using root path '/usr/lib64/python3.9'
  Attempting to resolve using root path '/usr/lib64/python3.9'
  Looking in python search path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve stub package using root path '/usr/lib64/python3.9/lib-dynload'
  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
  Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
  Attempting to resolve stub package using root path '/tmp/python3/lib/python3.9/site-packages'
  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google-stubs/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Resolved import with file '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/__init__.pyi'
  Typeshed path not found
/tmp/test.py
  /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
  /tmp/test.py:4:13 - info: Type of "__version__" is "bytes"
1 error, 0 warnings, 1 info 
Completed in 0.574sec

I thought it might be useful to have a diff too:

-/tmp % foo/node_modules/.bin/pyright test.py --verbose                                                          1 ↵
+$ foo/node_modules/.bin/pyright --verbose test.py
 No configuration file found.
 No pyproject.toml file found.
 stubPath /tmp/typings is not a valid directory.
@@ -18,19 +18,20 @@
   Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stdlib'
   Typeshed path not found
   Looking in stubPath '/tmp/typings'
-  Attempting to resolve using root path '/tmp/typings'
+  Attempting to resolve stub package using root path '/tmp/typings'
   Attempting to resolve using root path '/tmp/typings'
   Looking in root directory of execution environment '/tmp'
-  Attempting to resolve using root path '/tmp'
+  Attempting to resolve stub package using root path '/tmp'
   Attempting to resolve using root path '/tmp'
   Looking in python search path '/usr/lib64/python3.9'
-  Attempting to resolve using root path '/usr/lib64/python3.9'
+  Attempting to resolve stub package using root path '/usr/lib64/python3.9'
   Attempting to resolve using root path '/usr/lib64/python3.9'
   Looking in python search path '/usr/lib64/python3.9/lib-dynload'
-  Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
+  Attempting to resolve stub package using root path '/usr/lib64/python3.9/lib-dynload'
   Attempting to resolve using root path '/usr/lib64/python3.9/lib-dynload'
   Looking in python search path '/tmp/python3/lib/python3.9/site-packages'
-  Attempting to resolve using root path '/tmp/python3/lib/python3.9/site-packages'
+  Attempting to resolve stub package using root path '/tmp/python3/lib/python3.9/site-packages'
+  Resolved import with file '/tmp/python3/lib/python3.9/site-packages/google-stubs/__init__.pyi'
   Looking for typeshed path
   Looking for typeshed stubs path
   Attempting to resolve using root path '/tmp/foo/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
@@ -38,6 +39,6 @@
   Typeshed path not found
 /tmp/test.py
   /tmp/test.py:1:6 - error: Import "google.cloud" could not be resolved (reportMissingImports)
-  /tmp/test.py:6:13 - info: Type of "__version__" is "bytes"
+  /tmp/test.py:4:13 - info: Type of "__version__" is "bytes"
 1 error, 0 warnings, 1 info 
-Completed in 0.56sec
+Completed in 0.574sec
erictraut commented 3 years ago

@romanofski, is this still a problem?

I'm having trouble understanding which stub files are present in your site-packages. The two versions of the logs that you posted seem to indicate that you had different stub files present in each case. That explains part of the differences you see above.

Based on the most recent logs, it looks like there was no cloud subdirectory found within the site-packages/google-stubs directory. As such, the import resolution failed. Your earlier posts in this thread showed a cloud subdirectory.

At this point, I recommend trying to repro this from scratch with a clean venv. If you are still able to repro a behavior you consider to be a bug, then I'd appreciate step-by-step instructions for how to recreate the problem. Thanks!

romanofski commented 3 years ago

Thank you, will do. My apologies for the inconvenience this has caused.

erictraut commented 3 years ago

No worries. I'm going to mark this issue as closed for now. I'm happy to re-open it if you find that problems remain.

nipunn1313 commented 3 years ago

Hiya! I was able to minimally repro this with https://github.com/nipunn1313/pyright_google_repro (run ./run.sh)

Here's the output

+ pyright
Loading configuration file at /Users/nipunn/src/pyright_google_repro/pyrightconfig.json
Assuming Python version 3.9
Assuming Python platform Darwin
No include entries specified; assuming /Users/nipunn/src/pyright_google_repro
Auto-excluding **/node_modules
Auto-excluding **/__pycache__
Auto-excluding .git
stubPath /Users/nipunn/src/pyright_google_repro/typings is not a valid directory.
Search paths for /Users/nipunn/src/pyright_google_repro
  /usr/local/lib/node_modules/pyright/dist/typeshed-fallback/stdlib
  /Users/nipunn/src/pyright_google_repro
  /Users/nipunn/src/pyright_google_repro/typings
  /usr/local/lib/node_modules/pyright/dist/typeshed-fallback/stubs/...
  /Users/nipunn/src/pyright_google_repro/venv/lib/python3.9/site-packages
Searching for source files
Auto-excluding /Users/nipunn/src/pyright_google_repro/venv
Found 1 source file
Could not import 'google.cloud.firestore_v1' in file '/Users/nipunn/src/pyright_google_repro/test.py'
  Looking for typeshed stdlib path
  Attempting to resolve using root path '/usr/local/lib/node_modules/pyright/dist/typeshed-fallback/stdlib'
  Typeshed path not found
  Looking in stubPath '/Users/nipunn/src/pyright_google_repro/typings'
  Attempting to resolve stub package using root path '/Users/nipunn/src/pyright_google_repro/typings'
  Attempting to resolve using root path '/Users/nipunn/src/pyright_google_repro/typings'
  Looking in root directory of execution environment '/Users/nipunn/src/pyright_google_repro'
  Attempting to resolve stub package using root path '/Users/nipunn/src/pyright_google_repro'
  Attempting to resolve using root path '/Users/nipunn/src/pyright_google_repro'
  Looking in python search path '/Users/nipunn/src/pyright_google_repro/venv/lib/python3.9/site-packages'
  Attempting to resolve stub package using root path '/Users/nipunn/src/pyright_google_repro/venv/lib/python3.9/site-packages'
  Resolved import with file '/Users/nipunn/src/pyright_google_repro/venv/lib/python3.9/site-packages/google-stubs/__init__.pyi'
  Looking for typeshed path
  Looking for typeshed stubs path
  Attempting to resolve using root path '/usr/local/lib/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf'
  Typeshed path not found
/Users/nipunn/src/pyright_google_repro/test.py
  /Users/nipunn/src/pyright_google_repro/test.py:4:6 - error: Import "google.cloud.firestore_v1" could not be resolved (reportMissingImports)
1 error, 0 warnings, 0 infos
Completed in 0.519sec

I suspect that the source of issue here is that google and google.cloud are namespace packages where one of the subpackages (protobuf -> google.protobuf) uses a stub-package distribution (types-protobuf), while another (google-cloud-firestore -> google.cloud.firestore_v1) uses a py.typed

The google-stubs package created by the distribution types-protobuf contains a google-stubs/__init__.pyi (discussed further here) - which is at the most a bug, but at least an ambiguity in the PEP561.

I considered "fixing" the google-stubs package itself, however, I've found that even if I eliminate the google-stubs/__init__.pyi, the issue still arises. I (hackily) tried it out via

rm /Users/nipunn/src/pyright_google_repro/venv/lib/python3.9/site-packages/google-stubs/__init__.pyi
pyright

(I can't reopen the question, because I wasn't the original author @romanofski)

nipunn1313 commented 3 years ago

I did find that PEP561 says

This PEP does not support distributing typing information as part of module-only distributions. The code should be refactored into a package-based distribution and indicate that the package supports typing as described above.

I am trying to understand the meaning of module-only here. Particularly - trying to understand if google-cloud-firestore qualifies as a module-only distribution or a package-based distribution.

JelleZijlstra commented 3 years ago

"module-only" means it's distributed as just e.g. google.py for the google distribution. All of the distributions involved here are packages, so I don't think that applies.

erictraut commented 3 years ago

Thanks @nipunn1313 for the clear repro steps.

After investigating this further, I think pyright is doing the right thing here. The problem is that the google-stubs stub library is not marked as partial, so pyright is assuming that it should be used for all symbols in the google package namespace.

PEP 561 is unequivocal in saying:

If a stub package is partial it MUST include partial\n in a top level py.typed file.

If I manually add a py.typed file (with the text "partial\n") to the google-stubs directory, pyright is able to resolve the import from google.cloud.firestore_v1.

If my read is correct, then mypy has a bug in its import resolution logic. In the absence of the py.typed file, it should reject the import from google.cloud.firestore_v1.

Let me know if you agree with that analysis.

nipunn1313 commented 3 years ago

Heya @erictraut - thanks for taking a look. I think PEP561 is ambiguous about how to handle stub packages for namespace packages.

PEP561 also says

For namespace packages (see PEP 420), the py.typed file should be in the submodules of the namespace, to avoid conflicts and for clarity.

which contradicts

If a stub package is partial it MUST include partial\n in a top level py.typed file.

I am trying to make some modifications to PEP 561 to clarify.

I don't think it makes sense for google-stubs to have a py.typed partial file - because there isn't a single package that would be able to provide it - since it's a namespace package - populated by several distributions.

I believe the behavior I'd expect is for distributions like types-protobuf to omit google-stubs/__init__.pyi and for typecheckers to treat namespace packages as (by definition) incomplete, while treating the subpackage (google-stubs/protobuf) as complete based on py.typed in the subpackage

I realize this is breaching the realm of PEP design decisions rather than a pure bug, but I think it's what it'll take to make progress here.

erictraut commented 3 years ago

Thanks for the additional explanation.

If I understand you correctly, you don't expect the sample above to work unless google-stubs/__init__.pyi is manually deleted. With that file present, google-stubs isn't a namespace package.

I've changed pyright's import resolution logic to handle this case accordingly. If I run your batch file and then manually delete google-stubs/__init__.pyi, it now properly resolves both import paths in test.py.

nipunn1313 commented 3 years ago

Yep - that explanation matches my understanding.

I think it should be considered a bug in types-protobuf that it contains a google-stubs/__init__.pyi - however I think PEP 561 is ambiguousunderspecified. Hoping to clarify with a PEP revision.

erictraut commented 3 years ago

Thanks, I appreciate your leadership on this issue.

This change will be in the next release of pyright, probably published around midweek next week.

erictraut commented 3 years ago

This is addressed in pyright 1.1.174, which I just published. It will also be included in the next release of pylance.

nipunn1313 commented 3 years ago

Cool Thanks! Note that the original issue won't be solved until https://github.com/python/typeshed/pull/6106 gets through - but I think pyright's side of the equation has been adjusted appropriately!

joshtemple commented 2 years ago

@nipunn1313, given that python/typeshed#6106 was reverted, what's the best current solution for Pyright with this issue? It seems that mypy --namespace-packages is a fix for mypy, but I haven't been able to find a similar solution for Pyright.

erictraut commented 2 years ago

Are you using the typeshed stubs bundled with pyright, or are you using your own?

When I sync the typeshed stubs that are bundled with pyright, I manually delete the __init__.pyi for the google-cloud-ndb namespace package. I could do the same for the protobuf, since it's the same issue.

In the meantime, you could manually delete the __init__.pyi file.

joshtemple commented 2 years ago

I've installed types-protobuf==3.19.22 (latest version) in my virtualenv.

I've removed ~/.nvm/versions/node/v17.2.0/lib/node_modules/pyright/dist/typeshed-fallback/stubs/protobuf/google/protobuf/__init__.pyi and ~/.pyenv/versions/3.9.7/envs/my-venv/lib/python3.9/site-packages/google-stubs/protobuf/__init__.pyi. Did I miss one?

When running pyright, it still cannot resolve the google.cloud.* imports in my file.

nullie commented 2 years ago

@joshtemple you have to remove ~/.pyenv/versions/3.9.7/envs/my-venv/lib/python3.9/site-packages/google-stubs/__init__.pyi

joshtemple commented 2 years ago

Thank you! That solved it for me.

aabmass commented 2 years ago

Since the fix was reverted and the issue still remains, can this be reopened? It would have saved me some time I spent searching if the issue was still open.

erictraut commented 2 years ago

@aabmass, I'm not sure what you mean by "fix was reverted". Pyright is working as designed, so there's no bug to fix here.

If you are seeing a different issue that you believe is a bug, please open a new issue with the details.