search

pdm-project / pdm

A modern Python package and dependency manager supporting the latest PEP standards
https://pdm-project.org
MIT License
7.81k stars 386 forks source link

`pdm run myscript.py` results in `ModuleNotFoundError: No module named 'pkg_resources'` #985

Closed cbini closed 2 years ago

cbini commented 2 years ago

Steps to reproduce

The google-cloud-storage package appears to depend on pkg_resources, which is a module of setuptools. setuptools is not one of my project dependencies, but shouldn't it be installed as part of the Python environment? Regardless, after ensuring it's installed in the base environment, pdm run ... still fails.

This works: python myscript.py This doesn't: pdm run python myscript.py (see exception below)

I'm using Pyenv to mange my Python versions, and I ensured that my project is configured to use the correct interpreter. Also, pyenv which python and pdm run pyenv which python yield the same path.

Adding setuptools to the project dependencies will resolve the error, but it's a dependency of a dependency and it's installed in the base environment anyway. Something seems buggy.

Actual behavior

Traceback (most recent call last):
  File "/home/knjpythonanywhere/datarobot/powerschool/powerschool_sync/extract.py", line 10, in <module>
    from google.cloud import storage
  File "/home/knjpythonanywhere/datarobot/powerschool/__pypackages__/3.9/lib/google/cloud/storage/__init__.py", line 35, in <module>
    from google.cloud.storage.batch import Batch
  File "/home/knjpythonanywhere/datarobot/powerschool/__pypackages__/3.9/lib/google/cloud/storage/batch.py", line 30, in <module>
    from google.cloud.storage._http import Connection
  File "/home/knjpythonanywhere/datarobot/powerschool/__pypackages__/3.9/lib/google/cloud/storage/_http.py", line 19, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

Expected behavior

pdm run myscript.py should presumably have the same outcome as python myscript.py when I'm running both commands in the same working directory.

Environment Information

PDM version:        1.13.4
Python Interpreter: /home/knjpythonanywhere/.pyenv/versions/3.9.11/bin/python (3.9)
Project Root:       /home/knjpythonanywhere/datarobot/powerschool
Project Packages:   /home/knjpythonanywhere/datarobot/powerschool/__pypackages__/3.9
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.11",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.13.0-1017-aws",
  "platform_system": "Linux",
  "platform_version": "#19~20.04.1-Ubuntu SMP Mon Mar 7 12:53:12 UTC 2022",
  "python_full_version": "3.9.11",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}
frostming commented 2 years ago

Does /home/knjpythonanywhere/datarobot/powerschool/__pypackages__/3.9/lib/pkg_resources exist? What is the output of pdm list --graph? Does setuptools exist in pdm.lock?

cbini commented 2 years ago

Nope, it's not under __pypackages__/, doesn't appear in pdm list --graph, and it's not in pdm.lock. Output below.

Also, to confirm I am running the right Python, I ran pdm run python -c "import sys; print(sys.executable)". That yields: /home/knjpythonanywhere/.pyenv/versions/3.9.11/bin/python

If I run /home/knjpythonanywhere/.pyenv/versions/3.9.11/bin/python -c "import pkg_resources", it runs successfully. However, pdm run python -c "import pkg_resources" results in the original ModuleNotFoundError.

Debug Output: pdm list --graph

google-cloud-storage 2.2.1 [ required: >=2.1.0 ]
├── google-api-core 2.7.1 [ required: !=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 ]
│   ├── google-auth 2.6.0 [ required: <3.0dev,>=1.25.0 ]
│   │   ├── cachetools 5.0.0 [ required: <6.0,>=2.0.0 ]
│   │   ├── pyasn1-modules 0.2.8 [ required: >=0.2.1 ]
│   │   │   └── pyasn1 0.4.8 [ required: <0.5.0,>=0.4.6 ]
│   │   ├── rsa 4.8 [ required: <5,>=3.1.4 ]
│   │   │   └── pyasn1 0.4.8 [ required: >=0.1.3 ]
│   │   └── six 1.16.0 [ required: >=1.9.0 ]
│   ├── googleapis-common-protos 1.55.0 [ required: <2.0dev,>=1.52.0 ]
│   │   └── protobuf 3.19.4 [ required: >=3.12.0 ]
│   ├── protobuf 3.19.4 [ required: >=3.12.0 ]
│   └── requests 2.27.1 [ required: <3.0.0dev,>=2.18.0 ]
│       ├── certifi 2021.10.8 [ required: >=2017.4.17 ]
│       ├── charset-normalizer 2.0.12 [ required: ~=2.0.0 ]
│       ├── idna 3.3 [ required: <4,>=2.5 ]
│       └── urllib3 1.26.9 [ required: <1.27,>=1.21.1 ]
├── google-auth 2.6.0 [ required: <3.0dev,>=1.25.0 ]
│   ├── cachetools 5.0.0 [ required: <6.0,>=2.0.0 ]
│   ├── pyasn1-modules 0.2.8 [ required: >=0.2.1 ]
│   │   └── pyasn1 0.4.8 [ required: <0.5.0,>=0.4.6 ]
│   ├── rsa 4.8 [ required: <5,>=3.1.4 ]
│   │   └── pyasn1 0.4.8 [ required: >=0.1.3 ]
│   └── six 1.16.0 [ required: >=1.9.0 ]
├── google-cloud-core 2.2.3 [ required: <3.0dev,>=1.6.0 ]
│   ├── google-api-core 2.7.1 [ required: !=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 ]
│   │   ├── google-auth 2.6.0 [ required: <3.0dev,>=1.25.0 ]
│   │   │   ├── cachetools 5.0.0 [ required: <6.0,>=2.0.0 ]
│   │   │   ├── pyasn1-modules 0.2.8 [ required: >=0.2.1 ]
│   │   │   │   └── pyasn1 0.4.8 [ required: <0.5.0,>=0.4.6 ]
│   │   │   ├── rsa 4.8 [ required: <5,>=3.1.4 ]
│   │   │   │   └── pyasn1 0.4.8 [ required: >=0.1.3 ]
│   │   │   └── six 1.16.0 [ required: >=1.9.0 ]
│   │   ├── googleapis-common-protos 1.55.0 [ required: <2.0dev,>=1.52.0 ]
│   │   │   └── protobuf 3.19.4 [ required: >=3.12.0 ]
│   │   ├── protobuf 3.19.4 [ required: >=3.12.0 ]
│   │   └── requests 2.27.1 [ required: <3.0.0dev,>=2.18.0 ]
│   │       ├── certifi 2021.10.8 [ required: >=2017.4.17 ]
│   │       ├── charset-normalizer 2.0.12 [ required: ~=2.0.0 ]
│   │       ├── idna 3.3 [ required: <4,>=2.5 ]
│   │       └── urllib3 1.26.9 [ required: <1.27,>=1.21.1 ]
│   └── google-auth 2.6.0 [ required: <3.0dev,>=1.25.0 ]
│       ├── cachetools 5.0.0 [ required: <6.0,>=2.0.0 ]
│       ├── pyasn1-modules 0.2.8 [ required: >=0.2.1 ]
│       │   └── pyasn1 0.4.8 [ required: <0.5.0,>=0.4.6 ]
│       ├── rsa 4.8 [ required: <5,>=3.1.4 ]
│       │   └── pyasn1 0.4.8 [ required: >=0.1.3 ]
│       └── six 1.16.0 [ required: >=1.9.0 ]
├── google-resumable-media 2.3.2 [ required: >=2.3.2 ]
│   └── google-crc32c 1.3.0 [ required: <2.0dev,>=1.0 ]
├── protobuf 3.19.4 [ required: Any ]
└── requests 2.27.1 [ required: <3.0.0dev,>=2.18.0 ]
    ├── certifi 2021.10.8 [ required: >=2017.4.17 ]
    ├── charset-normalizer 2.0.12 [ required: ~=2.0.0 ]
    ├── idna 3.3 [ required: <4,>=2.5 ]
    └── urllib3 1.26.9 [ required: <1.27,>=1.21.1 ]
powerschool 3.1.3 [ required: >=3.1.1 ]
├── fiql-parser 0.15 [ required: >=0.15 ]
├── oauthlib 3.2.0 [ required: >=3.2.0 ]
├── python-dateutil 2.8.2 [ required: >=2.8.2 ]
│   └── six 1.16.0 [ required: >=1.5 ]
├── requests 2.27.1 [ required: >=2.27.1 ]
│   ├── certifi 2021.10.8 [ required: >=2017.4.17 ]
│   ├── charset-normalizer 2.0.12 [ required: ~=2.0.0 ]
│   ├── idna 3.3 [ required: <4,>=2.5 ]
│   └── urllib3 1.26.9 [ required: <1.27,>=1.21.1 ]
└── requests-oauthlib 1.3.1 [ required: >=1.3.1 ]
    ├── oauthlib 3.2.0 [ required: >=3.0.0 ]
    └── requests 2.27.1 [ required: >=2.0.0 ]
        ├── certifi 2021.10.8 [ required: >=2017.4.17 ]
        ├── charset-normalizer 2.0.12 [ required: ~=2.0.0 ]
        ├── idna 3.3 [ required: <4,>=2.5 ]
        └── urllib3 1.26.9 [ required: <1.27,>=1.21.1 ]

ls -la __pypackages__/3.9/lib/

drwxr-xr-x 42 cbini cbini  4096 Mar 22 11:00 .
drwxr-xr-x  5 cbini cbini  4096 Mar 22 11:00 ..
-rw-r--r--  1 cbini cbini  8731 Mar 22 11:00 CHANGELOG.md
-rw-r--r--  1 cbini cbini   577 Mar 22 11:00 LICENSE
-rw-r--r--  1 cbini cbini  1928 Mar 22 11:00 README.md
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 cachetools
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 cachetools-5.0.0.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 certifi
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 certifi-2021.10.8.dist-info
drwxr-xr-x  4 cbini cbini  4096 Mar 22 11:00 charset_normalizer
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 charset_normalizer-2.0.12.dist-info
drwxr-xr-x  5 cbini cbini  4096 Mar 22 11:00 dateutil
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 fiql_parser
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 fiql_parser-0.15.dist-info
drwxr-xr-x 15 cbini cbini  4096 Mar 22 11:00 google
-rw-r--r--  1 cbini cbini   539 Mar 22 11:00 google_api_core-2.7.1-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_api_core-2.7.1.dist-info
-rw-r--r--  1 cbini cbini   539 Mar 22 11:00 google_auth-2.6.0-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_auth-2.6.0.dist-info
-rw-r--r--  1 cbini cbini  1698 Mar 22 11:00 google_cloud_core-2.2.3-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_cloud_core-2.2.3.dist-info
-rw-r--r--  1 cbini cbini  1698 Mar 22 11:00 google_cloud_storage-2.2.1-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_cloud_storage-2.2.1.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_crc32c
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_crc32c-1.3.0.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_crc32c.libs
-rw-r--r--  1 cbini cbini   539 Mar 22 11:00 google_resumable_media-2.3.2-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 google_resumable_media-2.3.2.dist-info
-rw-r--r--  1 cbini cbini  1710 Mar 22 11:00 googleapis_common_protos-1.55.0-py3.10-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 googleapis_common_protos-1.55.0.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 idna
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 idna-3.3.dist-info
drwxr-xr-x  5 cbini cbini  4096 Mar 22 11:00 oauthlib
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 oauthlib-3.2.0.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 powerschool
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 powerschool-3.1.3.dist-info
-rw-r--r--  1 cbini cbini   539 Mar 22 11:00 protobuf-3.19.4-py3.9-nspkg.pth
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 protobuf-3.19.4.dist-info
drwxr-xr-x  5 cbini cbini  4096 Mar 22 11:00 pyasn1
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 pyasn1-0.4.8.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 pyasn1_modules
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 pyasn1_modules-0.2.8.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 python_dateutil-2.8.2.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 requests
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 requests-2.27.1.dist-info
drwxr-xr-x  3 cbini cbini  4096 Mar 22 11:00 requests_oauthlib
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 requests_oauthlib-1.3.1.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 rsa
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 rsa-4.8.dist-info
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 six-1.16.0.dist-info
-rw-r--r--  1 cbini cbini 34549 Mar 22 11:00 six.py
drwxr-xr-x  5 cbini cbini  4096 Mar 22 11:00 urllib3
drwxr-xr-x  2 cbini cbini  4096 Mar 22 11:00 urllib3-1.26.9.dist-info
cbini commented 2 years ago

OK I think I found the root cause.

/home/knjpythonanywhere/.pyenv/versions/3.9.11/bin/python -c "import pkg_resources; print(pkg_resources.__file__)" gives me /home/knjpythonanywhere/.pyenv/versions/3.9.11/lib/python3.9/site-packages/pkg_resources/__init__.py.

I searched site-packages in the docs and found this relevant bit: https://pdm.fming.dev/usage/scripts/#site_packages

So it looks like I need to add site_packages = true to the script table? I guess since setuptools is in a weird gray area where it's not vanilla Python but it's typically included in installations, it caused some confusion. Also, it's not listed as a dependency of google-cloud-storage, so PDM won't pick up that it should be installed.

cbini commented 2 years ago

I can confirm that extract-nwk.site_packages = true will solve it. However, it doesn't seem to work using the global setting key: _.site_packages = true. Should I open a new issue for that?

frostming commented 2 years ago

So google-cloud-storage fails to list setuptools as a dependency, you should add it: pdm add setuptools