Missed packages installed in virtual environment

AndrewUshakov commented 4 years ago

How to reproduce the issue (Windows 10 64 bit, Python 3.8.5 64 bit):

1) Install pipdeptree in site-packages (using admin rights).

2) Create empty directory, switch there.

3) Create virtual environment using venv or virtualenv:

virtualenv --clear --system-site-packages .venv

python -m venv --clear --system-site-packages  .venv

4) Activate it:

.venv/scripts/activate.bat

5) Install any package/module in virtual environment (I initially discovered this issue with netCDF4):

(.venv)>pip install -U netCDF4

6) Run pipdeptree using console entry point:

(.venv)>pipdeptree

Module netCDF4 will not be listed. Initially I thought that it is an error with this specific module, but later discovered, that it occurs with pandas etc.

7) (Workaround) Run pipdeptree as a module

(.venv)>python -m pipdeptree

Module netCDF4 will be listed...

P.S. By the way, pip started as the console entry point (pip) and as a module (python -m pip) in both cases behaves properly.

rickhg12hs commented 4 years ago

Possibly related (Fedora 29 x86_64, Python 3.7.5) ...

When I execute pipdeptree -l in a venv, it misses the venv installed packages and lists all the other packages not in the venv.

python3 -m pipdeptree -l in the venv produces the correct output.

(tryptools) $ python3 -m pipdeptree -l
ipython==7.18.1
  - backcall [required: Any, installed: 0.2.0]
  - decorator [required: Any, installed: 4.4.2]
  - jedi [required: >=0.10, installed: 0.17.2]
    - parso [required: >=0.7.0,<0.8.0, installed: 0.7.1]
  - pexpect [required: >4.3, installed: 4.8.0]
    - ptyprocess [required: >=0.5, installed: 0.6.0]
  - pickleshare [required: Any, installed: 0.7.5]
  - prompt-toolkit [required: >=2.0.0,<3.1.0,!=3.0.1,!=3.0.0, installed: 3.0.7]
    - wcwidth [required: Any, installed: 0.2.5]
  - pygments [required: Any, installed: 2.6.1]
  - setuptools [required: >=18.5, installed: 50.1.0]
  - traitlets [required: >=4.2, installed: 5.0.3]
    - ipython-genutils [required: Any, installed: 0.2.0]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.2]
pipetools==0.3.5
  - setuptools [required: >=0.6b1, installed: 50.1.0]
wheel==0.35.1

naiquevin commented 3 years ago

Sorry for the delay. I missed this issue! Will try to reproduce this and see if it can be fixed in the next release. Thanks!

gaborbernat commented 2 years ago

Seems no update in a while, so closing. If you can still replicate please comment and we can reopen.

AndrewUshakov commented 2 years ago

I am so sorry, but I just reproduced initial issue with Windows 11, Python 3.10.5, pipdeptree 2.2.1, pip 22.1.1, virtualenv 20.16.3 and netCDF4 1.6.0.

gaborbernat commented 2 years ago

Can you try with 2.3.0?

AndrewUshakov commented 2 years ago

Thank you! I do not see this error now.

completementgaga commented 1 year ago

Hello, I am having the same issue here with poetry. In a fresh python 3.10.6 virtual environment obtained running poetry shell: $pip freeze ouputs nothing. This is the expected behavior. On the other hand, $pipdeptree has the same output as run from the system environment.

My system is Ubuntu 22.04.2 with pipdeptree 2.7.0

Best,

bartoszgrabski commented 1 year ago

I've got the same issue on Windows 11, Python 3.9.13, pipdeptree version 2.13, pip version 23.2.1.

It does not list Django as top level package in the output, although it does list it as a dependency of other listed packages.

andy-maier commented 1 year ago

I have the same error. In a virtualenv with Python 3.10, and pipdeptree 2.13.0 installed in the virtualenv:

$ pip list | grep -E "pip|python-jsonschema"
pip                       23.2.1
pipdeptree                2.13.0
python-jsonschema-objects 0.5.0

$ pipdeptree -p python-jsonschema-objects

$ python -m pipdeptree -p python-jsonschema-objects
python-jsonschema-objects==0.5.0
├── inflection [required: >=0.2, installed: 0.5.1]
├── jsonschema [required: >=4.18, installed: 4.19.1]
│   ├── attrs [required: >=22.2.0, installed: 23.1.0]
│   ├── jsonschema-specifications [required: >=2023.03.6, installed: 2023.7.1]
│   │   └── referencing [required: >=0.28.0, installed: 0.30.2]
│   │       ├── attrs [required: >=22.2.0, installed: 23.1.0]
│   │       └── rpds-py [required: >=0.7.0, installed: 0.10.3]
│   ├── referencing [required: >=0.28.4, installed: 0.30.2]
│   │   ├── attrs [required: >=22.2.0, installed: 23.1.0]
│   │   └── rpds-py [required: >=0.7.0, installed: 0.10.3]
│   └── rpds-py [required: >=0.7.1, installed: 0.10.3]
├── Markdown [required: >=2.4, installed: 3.4.4]
└── six [required: >=1.5.2, installed: 1.16.0]

Thanks for the workaround with "python -m pipdeptree", but I still think this should be addressed.

kemzeb commented 6 months ago

I can confirm that this is happening. I have pipdeptree dev dependencies installed, and a virtual environment created by doing the following:

python -m venv --system-site-packages venv

So the virtual environment shouldn't have anything in its site-packages directory.

From executing the following:

$ pipdeptree --local-only
covdefaults==2.3.0
└── coverage [required: >=6.0.2, installed: 7.4.4]
diff_cover==8.0.3
├── chardet [required: >=3.0.0, installed: 5.2.0]
├── Jinja2 [required: >=2.7.1, installed: 3.1.3]
│   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
├── pluggy [required: >=0.13.1,<2, installed: 1.4.0]
└── Pygments [required: >=2.9.0,<3.0.0, installed: 2.17.2]
GitPython==3.1.41
└── gitdb [required: >=4.0.1,<5, installed: 4.0.11]
    └── smmap [required: >=3.0.1,<6, installed: 5.0.1]
pipdeptree==2.16.3.dev27+g757c8ab.d20240329
├── packaging [required: >=23.1, installed: 24.0]
└── pip [required: >=23.1.2, installed: 24.0]
pre-commit==3.7.0
. . .

... we still get all of the global site packages. Running the workaround:

(venv) $ python -m pipdeptree --local-only
pip==23.0.1
setuptools==58.1.0

... gives the expected results.

I did find the reason why this is happening, though. Since I'm running pipdeptree in editable mode, I can change the code to print out the sys.prefix:

(venv) $ pipdeptree --local-only
sys.prefix:  /usr/local
covdefaults==2.3.0
└── coverage [required: >=6.0.2, installed: 7.4.4]
. . .

Even though I'm running in a virtual environment, it uses the Unix default /usr/local. Running the workaround:

(venv) $ python -m pipdeptree --local-only
sys.prefix:  /workspaces/pipdeptree/venv
pip==23.0.1
setuptools==58.1.0

The path points to the virtual env. So my guess right now is that pip is changing sys.prefix... for some reason. We could continue trying to dig a little deeper here, or wait for #175 to be resolved as this would also resolve the problem (since we would be moving away from using pip internal API for package discovery).

Edit: My comment below discusses what I think is happening

kemzeb commented 6 months ago

I do have another guess as to why this is happening. It could be that it depends on where pipdeptree is installed.

If it's installed in the global site packages, where we then execute the console script in the virtual environment, it maybe the case that pipdeptree doesn't recognize that it's running in a virtual environment because we never give it the Python interpreter from the virtual environment. In the following example I installed pipdeptree globally and printed out sys.executable in the virtual env:

$ python -m venv --system-site-packages venv
$ source venv/bin/activate
(venv) $ which pipdeptree
/home/vscode/.local/bin/pipdeptree
(venv) $ pipdeptree --local-only
sys.executable: /usr/local/bin/python # <---- HERE
covdefaults==2.3.0
└── coverage [required: >=6.0.2, installed: 7.4.4]
diff_cover==8.0.3
. . .

Looking at this, pipdeptree is using the system-provided Python interpreter when we expected it to use the one found in the virtual environment (since we executed it in the virtual environment).

When it's installed in the virtual environment itself, I observed that running the console script will give the expected results:

(venv) $ pipdeptree --local-only
pipdeptree==2.16.2
└── pip [required: >=23.1.2, installed: 24.0]
setuptools==58.1.0

We do have a --python option, and when running pipdeptree --python venv/bin/python --local-only, it behaves as expected since internally it's just executing venv/bin/python -m pipdeptree.

kemzeb commented 6 months ago

I've got the same issue on Windows 11, Python 3.9.13, pipdeptree version 2.13, pip version 23.2.1.

It does not list Django as top level package in the output, although it does list it as a dependency of other listed packages.

Hi @bartoszgrabski, it sounds like your problem may have to do with what was discussed in #277.

kdeldycke commented 5 months ago

I can confirm the behavior with pipx and poetry.

Let's install pipdeptree with pipx, system-wide:

$ pipx install pipdeptree
  installed package pipdeptree 2.18.1, installed using Python 3.12.3
  These apps are now globally available
    - pipdeptree
done! ✨ 🌟 ✨

System-wide pipdeptree is reporting its dependencies from the point of view of its venv, as created by pipx:

$ pipdeptree
pipdeptree==2.18.1
├── packaging [required: >=23.1, installed: 24.0]
└── pip [required: >=23.1.2, installed: 24.0]

$ which pipdeptree
/Users/kde/.local/bin/pipdeptree

Now let's I install a Poetry project:

$ git clone https://github.com/kdeldycke/meta-package-manager.git
(...)

$ cd meta-package-manager
$ poetry install
(...)

Here, I expect pipdeptree to report the dependencies of the meta-package-manager project, but it doesn't:

$ pipdeptree
pipdeptree==2.18.1
├── packaging [required: >=23.1, installed: 24.0]
└── pip [required: >=23.1.2, installed: 24.0]

For this to work, I need to pass the venv's Python:

$ pipdeptree --python ./.venv/bin/python
(...)
meta-package-manager==5.15.1
├── boltons [required: >=24.0.0,<25.0.0, installed: 24.0.0]
├── click [required: >=8.1.2,<9.0.0, installed: 8.1.7]
├── click-extra [required: >=4.7.5,<5.0.0, installed: 4.7.5]
│   ├── boltons [required: >=24.0.0,<25.0.0, installed: 24.0.0]
│   ├── click [required: >=8.1.4,<9.0.0, installed: 8.1.7]
│   ├── cloup [required: >=3.0.5,<4.0.0, installed: 3.0.5]
│   │   └── click [required: >=8.0,<9.0, installed: 8.1.7]
│   ├── commentjson [required: >=0.9.0,<0.10.0, installed: 0.9.0]
│   │   └── lark-parser [required: >=0.7.1,<0.8.0, installed: 0.7.8]
│   ├── mergedeep [required: >=1.3.4,<2.0.0, installed: 1.3.4]
│   ├── Pallets-Sphinx-Themes [required: >=2.1.1,<3.0.0, installed: 2.1.1]
│   │   ├── packaging [required: Any, installed: 24.0]
│   │   └── Sphinx [required: >=3, installed: 7.1.2]
│   │       ├── alabaster [required: >=0.7,<0.8, installed: 0.7.13]
│   │       ├── Babel [required: >=2.9, installed: 2.14.0]
│   │       ├── docutils [required: >=0.18.1,<0.21, installed: 0.20.1]
│   │       ├── imagesize [required: >=1.3, installed: 1.4.1]
│   │       ├── Jinja2 [required: >=3.0, installed: 3.1.3]
│   │       │   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
│   │       ├── packaging [required: >=21.0, installed: 24.0]
│   │       ├── Pygments [required: >=2.13, installed: 2.17.2]
│   │       ├── requests [required: >=2.25.0, installed: 2.31.0]
│   │       │   ├── certifi [required: >=2017.4.17, installed: 2024.2.2]
│   │       │   ├── charset-normalizer [required: >=2,<4, installed: 3.3.2]
│   │       │   ├── idna [required: >=2.5,<4, installed: 3.6]
│   │       │   └── urllib3 [required: >=1.21.1,<3, installed: 2.2.1]
│   │       ├── snowballstemmer [required: >=2.0, installed: 2.2.0]
│   │       ├── sphinxcontrib-applehelp [required: Any, installed: 1.0.4]
│   │       ├── sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
│   │       ├── sphinxcontrib-htmlhelp [required: >=2.0.0, installed: 2.0.1]
│   │       ├── sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
│   │       ├── sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
│   │       └── sphinxcontrib-serializinghtml [required: >=1.1.5, installed: 1.1.5]
│   ├── Pygments [required: >=2.14,<3.0, installed: 2.17.2]
│   ├── pygments-ansi-color [required: >=0.3.0,<0.4.0, installed: 0.3.0]
│   │   └── Pygments [required: !=2.7.3, installed: 2.17.2]
│   ├── PyYAML [required: >=6.0.0,<7.0.0, installed: 6.0.1]
│   ├── regex [required: >=2023.3.22,<2024.0.0, installed: 2023.12.25]
│   ├── requests [required: >=2.28.2,<3.0.0, installed: 2.31.0]
│   │   ├── certifi [required: >=2017.4.17, installed: 2024.2.2]
│   │   ├── charset-normalizer [required: >=2,<4, installed: 3.3.2]
│   │   ├── idna [required: >=2.5,<4, installed: 3.6]
│   │   └── urllib3 [required: >=1.21.1,<3, installed: 2.2.1]
│   ├── Sphinx [required: >=6, installed: 7.1.2]
│   │   ├── alabaster [required: >=0.7,<0.8, installed: 0.7.13]
│   │   ├── Babel [required: >=2.9, installed: 2.14.0]
│   │   ├── docutils [required: >=0.18.1,<0.21, installed: 0.20.1]
│   │   ├── imagesize [required: >=1.3, installed: 1.4.1]
│   │   ├── Jinja2 [required: >=3.0, installed: 3.1.3]
│   │   │   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
│   │   ├── packaging [required: >=21.0, installed: 24.0]
│   │   ├── Pygments [required: >=2.13, installed: 2.17.2]
│   │   ├── requests [required: >=2.25.0, installed: 2.31.0]
│   │   │   ├── certifi [required: >=2017.4.17, installed: 2024.2.2]
│   │   │   ├── charset-normalizer [required: >=2,<4, installed: 3.3.2]
│   │   │   ├── idna [required: >=2.5,<4, installed: 3.6]
│   │   │   └── urllib3 [required: >=1.21.1,<3, installed: 2.2.1]
│   │   ├── snowballstemmer [required: >=2.0, installed: 2.2.0]
│   │   ├── sphinxcontrib-applehelp [required: Any, installed: 1.0.4]
│   │   ├── sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
│   │   ├── sphinxcontrib-htmlhelp [required: >=2.0.0, installed: 2.0.1]
│   │   ├── sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
│   │   ├── sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
│   │   └── sphinxcontrib-serializinghtml [required: >=1.1.5, installed: 1.1.5]
│   ├── tabulate [required: >=0.9,<0.10, installed: 0.9.0]
│   ├── wcmatch [required: >=8.5,<9.0, installed: 8.5.1]
│   │   └── bracex [required: >=2.1.1, installed: 2.4]
│   └── xmltodict [required: >=0.13.0,<0.14.0, installed: 0.13.0]
├── packageurl-python [required: >=0.15.0,<0.16.0, installed: 0.15.0]
├── tabulate [required: >=0.9.0,<0.10.0, installed: 0.9.0]
├── tomli_w [required: >=1.0.0,<2.0.0, installed: 1.0.0]
└── xmltodict [required: >=0.13.0,<0.14.0, installed: 0.13.0]
(...)

kdeldycke commented 5 months ago

BTW, a more robust way of running pipdeptree in a Poetry venv:

$ pipdeptree --python "$( poetry env info --executable )"

kemzeb commented 5 months ago

I can confirm the behavior with pipx and poetry.

Let's install pipdeptree with pipx, system-wide:
$ pipx install pipdeptree
  installed package pipdeptree 2.18.1, installed using Python 3.12.3
  These apps are now globally available
    - pipdeptree
done! ✨ 🌟 ✨
System-wide pipdeptree is reporting its dependencies from the point of view of its venv, as created by pipx:
$ pipdeptree
pipdeptree==2.18.1
├── packaging [required: >=23.1, installed: 24.0]
└── pip [required: >=23.1.2, installed: 24.0]

$ which pipdeptree
/Users/kde/.local/bin/pipdeptree

Thanks for your input @kdeldycke! There was a similar issue, https://github.com/tox-dev/pipdeptree/issues/322, that reported a problem when using pipx. The pipdeptree console script would be available globally, but from reading this section in the pipx README, pipdeptree would be sandboxed into it's own virtual environment without system site packages enabled. Here's an example where I installed pipdeptree using pipx:

$ pipx --version
1.5.0
$ which pipdeptree
/usr/local/py-utils/bin/pipdeptree
$ cat /usr/local/py-utils/venvs/pipdeptree/p
pipx_metadata.json  pyvenv.cfg          
$ cat /usr/local/py-utils/venvs/pipdeptree/pyvenv.cfg 
home = /usr/local/py-utils/venvs/pipx/bin
include-system-site-packages = false
version = 3.8.19

include-system-site-packages = false is the culprit as to why you don't see packages from the system environment. I'm not sure if you can configure pipx in a way that will enable it when performing pipx install. I changed the value to true and ran it to see if it would work and it does:

$ /usr/local/py-utils/bin/pipdeptree 
Warning!!! Duplicate package metadata found:
"/home/vscode/.local/lib/python3.8/site-packages"
  packaging                        24.0             (using 24.0, "/usr/local/py-utils/venvs/pipdeptree/lib/python3.8/site-packages")
  pip                              24.0             (using 24.0, "/usr/local/py-utils/shared/lib/python3.8/site-packages")
"/usr/local/lib/python3.8/site-packages"
  pip                              23.0.1           (using 24.0, "/usr/local/py-utils/shared/lib/python3.8/site-packages")
  setuptools                       69.0.3           (using 56.0.0, "/usr/local/py-utils/shared/lib/python3.8/site-packages")
------------------------------------------------------------------------
covdefaults==2.3.0
└── coverage [required: >=6.0.2, installed: 7.4.4]
diff_cover==9.0.0
├── chardet [required: >=3.0.0, installed: 5.2.0]
├── Jinja2 [required: >=2.7.1, installed: 3.1.3]
│   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
├── pluggy [required: >=0.13.1,<2, installed: 1.4.0]
└── Pygments [required: >=2.9.0,<3.0.0, installed: 2.17.2]
. . .

kemzeb commented 5 months ago

Now let's I install a Poetry project:
$ git clone https://github.com/kdeldycke/meta-package-manager.git
(...)

$ cd meta-package-manager
$ poetry install
(...)
Here, I expect pipdeptree to report the dependencies of the meta-package-manager project, but it doesn't:
$ pipdeptree
pipdeptree==2.18.1
├── packaging [required: >=23.1, installed: 24.0]
└── pip [required: >=23.1.2, installed: 24.0]
For this to work, I need to pass the venv's Python:
$ pipdeptree --python ./.venv/bin/python
(...)

Yes, you should use --python as this allows us to query the virtual environment's python for its sys.path (at least this is how it works now in 2.18.0). We would then pass it's sys.path to importlib.metadata.distributions to snatch the necessary metadata.

If users are still having problems after 2.18.0, please let us know and I'll see if I can figure it out. In short, this is how you should run pipdeptree when working with a virtual environment:

If you installed pipdeptree globally, you would need to run something like pipdeptree --python path/to/venv/bin/python so that we can query the custom interpreter for its paths. If system-site-packages is enabled there shouldn't be any problems
If you installed pipdeptree in the virtual environment, you can run pipdeptree without any further issues (but make sure that you don't have pipdeptree also installed in your global environment as this might be chosen over the venv pipdeptree)

kdeldycke commented 4 months ago

With pipdeptree 2.21.0 release (and thanks to #366), we can now rely on --python auto parameter.

So I guess this issue can now be requalified as whether or not we should make --python auto parameter active by default. In which case pipdeptree can be used as-is in a virtual environment and be aware of its context. Reducing the surprises from new users.

zen2 commented 4 months ago

With pipdeptree 2.21.0 release (and thanks to #366), we can now rely on --python auto parameter.

So I guess this issue can now be requalified as whether or not we should make --python auto parameter active by default. In which case pipdeptree can be used as-is in a virtual environment and be aware of its context. Reducing the surprises from new users.

I vote for this default behaviour because I'm here as a new pipdeptree user that was fighting to see a venv dependencies.

tox-dev / pipdeptree

Missed packages installed in virtual environment #130