search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.88k stars 2.28k forks source link

Poetry install removes packages that are necessary #3139

Closed fcruzel closed 2 years ago

fcruzel commented 4 years ago

Issue

poetry install removes packages that are not supposed to be removed. I'm using Poetry in a Dockerfile to install the dependencies of my project. Before that, the parent image has also installed some dependencies that are necessary. 

$ poetry config virtualenvs.create false && poetry install --no-dev --no-interaction -vvv

  • Removing colorama (0.4.3): Skipped for the following reason: Not currently installed
  • Removing dnspython (1.16.0): Skipped for the following reason: Not currently installed
  • Removing colorclass (2.2.0): Skipped for the following reason: Not currently installed
  • Removing future (0.18.2): Skipped for the following reason: Not currently installed
  • Removing cycler (0.10.0): Skipped for the following reason: Not currently installed
  • Removing google-auth (1.22.1): Skipped for the following reason: Not currently installed
  • Removing decorator (4.4.2): Skipped for the following reason: Not currently installed
  • Removing docopt (0.6.2): Skipped for the following reason: Not currently installed
  • Removing fbmessenger (6.0.0): Skipped for the following reason: Not currently installed
  • Removing gast (0.2.2): Skipped for the following reason: Not currently installed
  • Removing gevent (1.5.0): Skipped for the following reason: Not currently installed
  • Removing google-auth-oauthlib (0.4.1): Skipped for the following reason: Not currently installed
  • Removing google-pasta (0.2.0): Skipped for the following reason: Not currently installed
  • Removing greenlet (0.4.17): Skipped for the following reason: Not currently installed
  • Removing grpcio (1.32.0): Skipped for the following reason: Not currently installed
  • Removing h5py (2.10.0): Skipped for the following reason: Not currently installed
  • Removing httplib2 (0.18.1): Skipped for the following reason: Not currently installed
  • Removing ipaddress (1.0.23): Skipped for the following reason: Not currently installed
  • Removing isort (5.5.5): Skipped for the following reason: Not currently installed
  • Removing jmespath (0.10.0): Skipped for the following reason: Not currently installed
  • Removing keras-preprocessing (1.1.0): Skipped for the following reason: Not currently installed
  • Removing jsonpickle (1.4.1): Skipped for the following reason: Not currently installed
  • Removing jsonschema (3.2.0): Skipped for the following reason: Not currently installed
  • Removing kafka-python (1.4.7): Skipped for the following reason: Not currently installed
  • Removing keras-applications (1.0.8): Skipped for the following reason: Not currently installed
  • Removing mattermostwrapper (2.2): Skipped for the following reason: Not currently installed
  • Removing mccabe (0.6.1): Skipped for the following reason: Not currently installed
  • Removing lazy-object-proxy (1.4.3): Skipped for the following reason: Not currently installed
  • Removing networkx (2.4): Skipped for the following reason: Not currently installed
  • Removing matplotlib (3.2.2): Skipped for the following reason: Not currently installed
  • Removing joblib (0.17.0): Skipped for the following reason: Not currently installed
  • Removing kiwisolver (1.2.0): Skipped for the following reason: Not currently installed
  • Removing mypy-extensions (0.4.3): Skipped for the following reason: Not currently installed
  • Removing pika (1.1.0): Skipped for the following reason: Not currently installed
  • Removing prompt-toolkit (2.0.10): Skipped for the following reason: Not currently installed
  • Removing oauthlib (3.1.0): Skipped for the following reason: Not currently installed
  • Removing opt-einsum (3.3.0): Skipped for the following reason: Not currently installed
  • Removing pathspec (0.8.0): Skipped for the following reason: Not currently installed
  • Removing pyasn1-modules (0.2.8): Skipped for the following reason: Not currently installed
  • Removing oauth2client (4.1.3): Skipped for the following reason: Not currently installed
  • Removing pyjwt (1.7.1): Skipped for the following reason: Not currently installed
  • Removing pykwalify (1.7.0): Skipped for the following reason: Not currently installed
  • Removing pyasn1 (0.4.8): Skipped for the following reason: Not currently installed
  • Removing markdown (3.3): Skipped for the following reason: Not currently installed
  • Removing pydot (1.4.1): Skipped for the following reason: Not currently installed
  • Removing protobuf (3.13.0): Skipped for the following reason: Not currently installed
  • Removing psycopg2-binary (2.8.6): Skipped for the following reason: Not currently installed
  • Removing pylint (2.6.0): Skipped for the following reason: Not currently installed
  • Removing pymongo (3.8.0): Skipped for the following reason: Not currently installed
  • Removing pyreadline (2.1): Skipped for the following reason: Not currently installed
  • Removing pyrsistent (0.17.3): Skipped for the following reason: Not currently installed
  • Removing pysocks (1.7.1): Skipped for the following reason: Not currently installed
  • Removing questionary (1.5.2): Skipped for the following reason: Not currently installed
  • Removing python-engineio (3.12.1): Skipped for the following reason: Not currently installed
  • Removing python-socketio (4.5.1): Skipped for the following reason: Not currently installed
  • Removing python-telegram-bot (12.8): Skipped for the following reason: Not currently installed
  • Removing pyyaml (5.3.1): Skipped for the following reason: Not currently installed
  • Removing rocketchat-api (1.3.1): Skipped for the following reason: Not currently installed
  • Removing rasa (1.10.3): Skipped for the following reason: Not currently installed
  • Removing redis (3.5.3): Skipped for the following reason: Not currently installed
  • Removing ruamel.yaml.clib (0.2.2): Skipped for the following reason: Not currently installed
  • Removing requests-oauthlib (1.3.0): Skipped for the following reason: Not currently installed
  • Removing sanic-jwt (1.4.1): Skipped for the following reason: Not currently installed
  • Removing rsa (4.6): Skipped for the following reason: Not currently installed
  • Removing ruamel.yaml (0.16.12): Skipped for the following reason: Not currently installed
  • Removing sklearn-crfsuite (0.3.6): Skipped for the following reason: Not currently installed
  • Removing slackclient (2.9.1): Skipped for the following reason: Not currently installed
  • Removing python-crfsuite (0.9.7): Skipped for the following reason: Not currently installed
  • Removing scikit-learn (0.22.2.post1): Skipped for the following reason: Not currently installed
  • Removing scipy (1.4.1): Skipped for the following reason: Not currently installed
  • Removing regex (2020.9.27): Skipped for the following reason: Not currently installed
  • Removing s3transfer (0.3.3): Skipped for the following reason: Not currently installed
  • Removing sqlalchemy (1.3.19): Skipped for the following reason: Not currently installed
  • Removing tensorflow-hub (0.8.0): Skipped for the following reason: Not currently installed
  • Removing tensorflow-probability (0.9.0): Skipped for the following reason: Not currently installed
  • Removing termcolor (1.1.0): Skipped for the following reason: Not currently installed
  • Removing tensorflow-addons (0.7.1): Skipped for the following reason: Not currently installed
  • Removing toml (0.10.1): Skipped for the following reason: Not currently installed
  • Removing tabulate (0.8.7): Skipped for the following reason: Not currently installed
  • Removing tensorboard (2.1.1): Skipped for the following reason: Not currently installed
  • Removing tensorflow (2.1.1): Skipped for the following reason: Not currently installed
  • Removing terminaltables (3.1.0): Skipped for the following reason: Not currently installed
  • Removing tensorflow-estimator (2.1.0): Skipped for the following reason: Not currently installed
  • Removing tornado (6.0.4): Skipped for the following reason: Not currently installed
  • Removing twilio (6.26.3): Skipped for the following reason: Not currently installed
  • Removing typed-ast (1.4.1): Skipped for the following reason: Not currently installed
  • Removing wrapt (1.12.1): Skipped for the following reason: Not currently installed
  • Removing yarl (1.6.0): Skipped for the following reason: Not currently installed
  • Removing wcwidth (0.2.5): Skipped for the following reason: Not currently installed
  • Removing typing-extensions (3.7.4.3): Skipped for the following reason: Not currently installed
  • Removing tzlocal (2.1): Skipped for the following reason: Not currently installed
  • Removing werkzeug (1.0.1): Skipped for the following reason: Not currently installed
  • Removing webexteamssdk (1.3): Skipped for the following reason: Not currently installed
  • Removing aiofiles (0.5.0)
  • Removing appdirs (1.4.4)
  • Removing cffi (1.14.3)
  • Removing coloredlogs (10.0)
  • Removing cryptography (3.1.1)
  • Removing h11 (0.8.1)
  • Removing h2 (3.2.0)
  • Removing hpack (3.0.0)
  • Removing hstspreload (2020.10.6)
  • Removing httptools (0.1.1)
  • Removing httpx (0.9.3)
  • Removing humanfriendly (8.2)
  • Removing hyperframe (5.2.0)
  • Removing multidict (4.7.6)
  • Removing packaging (20.4)
  • Removing pycparser (2.20)
  • Removing pyparsing (2.4.7)
  • Removing rasa-sdk (1.10.3)
  • Removing requests-toolbelt (0.9.1)
  • Removing rfc3986 (1.4.0)
  • Removing sanic (19.12.2)
  • Removing sanic-cors (0.10.0.post3)
  • Removing sanic-plugins-framework (0.9.4)
  • Removing sniffio (1.1.0)
  • Removing ujson (2.0.3)
  • Removing uvloop (0.14.0)
  • Removing websockets (8.1)
  • Installing zipp (3.3.0): Skipped for the following reason: Already installed
  • Installing cymem (2.0.3)
  • Updating importlib-metadata (1.7.0 -> 2.0.0)
  • Installing murmurhash (1.0.2)
  • Installing numpy (1.19.2)
  • Installing blis (0.4.1)
  • Installing catalogue (1.0.0)
  • Installing certifi (2020.6.20): Skipped for the following reason: Already installed
  • Installing chardet (3.0.4): Skipped for the following reason: Already installed
  • Installing plac (1.1.3)
  • Installing idna (2.10): Skipped for the following reason: Already installed
  • Installing tqdm (4.45.0)
  • Installing wasabi (0.8.0)
  • Installing srsly (1.0.2)
  • Installing preshed (3.0.2)
  • Installing urllib3 (1.25.10): Skipped for the following reason: Already installed
  • Installing requests (2.24.0): Skipped for the following reason: Already installed
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing thinc (7.4.1)
  • Installing python-dateutil (2.8.1)
  • Installing pytz (2019.3)
  • Installing spacy (2.3.2)
  • Installing es-core-news-sm (2.3.1 https://github.com/explosion/spacy-models/releases/download/es_core_news_sm-2.3.1/es_core_news_sm-2.3.1.tar.gz)
  • Installing pandas (1.1.3)
  • Installing unidecode (1.1.1)

As you can see, there are multiple dependencies that are removed. How can I prevent it?

abn commented 4 years ago

If the dependencies are managed by poetry (ie. via lock file) and you specify --no-dev poetry will assume these packages were installed by poetry earlier and requires to be removed. This is expected behaviour for current implementation. In the future we might add an option to skip these operations.

In your scenario, assuming you are pre-loading required dependencies, you might be better of building a wheel of your package and installing it instead.

poetry build -f wheel
pip install dist/*.whl
fcruzel commented 4 years ago

Yes, I did have some preinstalled dependencies that wanted to keep while installing the project dependencies with Poetry. Thank you @abn, I'll try with the wheel.

fcruzel commented 4 years ago

Hi @abn, one question. I was reading the docs. Is this bevahiour already covered by the flag --remove-untracked? Meaning no dependencies should be removed unless the flag is passed to the command (even if --no-dev is passed).

abn commented 4 years ago

Yes and no. The key thing to note here is that the install command is meant to ensure that any new packages required added, existing packages updated and remove packages that were removed. The assumption is that the the environment is "poetry managed". This assumption, needs to be revisited eventually I feel.

Poetry will not remove, iirc, packages that were never managed by poetry. So if your package A depends on B as a dev dependency, and you performed poerty install && poetry run pip install C, B should be removed and C will not be removed. To remove C, you will need to use --remove-untracked.

fcruzel commented 4 years ago

I see. I would be a great idea to document this behavior in the docs. Thanks you @abn

sontek commented 3 years ago

I just hit this and I think it might still be a bug:

Poetry depends on:

If I add a dependency that requires them (black, flake8, pre-commit) and run:

poetry install --no-dev

It removes things it depends on. Which I don't think is the intended result. It really should keep its own dependencies around at a minimum

abn commented 3 years ago

@sontek maybe raise a new big with a bit more information, and maybe an example with logs we could try to reproduce with.

As a note, it is not recommended that use the same environment poetry is installed in as your project's virtual environment.

owais commented 3 years ago

Copying my comment from another issue:

This is quite a big pain point actually. It'd be nice to at least have a flag to turn this behavior off. Here is how I'm trying using it.

I'm working on a library say LibA and I need to manually test it in multiple apps during development. From docs and poetry install --help, it seems poetry install --no-dev is the exactly what I need because of: 

  --no-dev               Do not install the development dependencies.

Intuitively, I felt I'd do the following:

# currently in my poetry managed LibA project
> pwd
~/LibA

# activate my test app's venv
. ~/app/venv/bin/activate

# install LibA as an editable package in app's env
poetry install --no-dev

This installs LibA as an editable install in app's venv. Now I should be able to run the app, make changes to LibA and have them reflect in the app without having to re-install LibA after every change.

However, this doesn't work exactly as expected. When poetry install is executed with the --no-dev option, it looks up for all dev dependencies of LibA and uninstalls them from app's env. This is a huge issue if app has any packages in common with LibA's dev dependencies. What makes it even worse is that poetry removes even indirect dependencies which makes it very likely for the deps to match.

I feel the default behavior should be to not install dev dependencies when --no-dev is passed (as it is documented) and to not remove any packages at all. If that is not desirable for some reason, we should be able to pass in another CLI param that prevents poetry from removing packages. Something like poetry install --no-dev --no-remove.

BTW what use cases demand that dev packages get removed when --no-dev is passed? Poetry already spports a --remove-untracked option. May be if a user wants to remove dev deps, they can run poetry install --no-dev --remove-untracked and this would ensure any dev deps are uninstalled. poetry install --no-dev would only install packages and never remove anything.

If this is not supported and poetry thinks this is not how poetry should be used, is there another ideal/preferred way to make this workflow work?

muppetjones commented 3 years ago

Bumping this. I have a project with a multistage docker that installs package A. In the main package, we specify dependency B. Package B requires A, but it doesn't list it as a dependency (it actually does, but I think they built it incorrectly). Just had a massive headache trying to figure out why package A is suddenly unavailable. There's quite a bit of dependency hell, but the gist of the issue was that poetry was uninstalling A because it was "unnecessary". I'd love to have "cleanup" or "efficiency" checks in poetry but NOT during install.

tl;dr: The "install" command should only "install" by default. It doesn't "upgrade" by default, why should it "remove"?

muppetjones commented 3 years ago

@fcruzel I don't think this should have been closed. The wheel is a treatment, but it's not a solution. This is a legit bug / forced feature.

owais commented 3 years ago

@muppetjones Agree. Try adding any dev dependency to your project that has an indirect dependency on any popular like like requests. dephell is one example. This means poetry will requests from your project's venv causing a lot more pain than poetry install solves in the first place. This forces you to install your dev dependencies into test projects or re-install test project's dependencies after every single poetry install --no-dev.

This means poetry does not provide a valid alternative to plain old setup.py develop command. Imagine setup.py develop either forcing you to install all your dev depencies or forcefully uninstalling any common dependencies from the target env. It wouldn't make any sense.

Ideally, poetry should do one of the following:

  1. Change behavior of install command to not remove packages and only install them.
  2. Remove by default but allow a --no-remove flag to disable behavior.
  3. Offer a new poetry develop command that behaves the same way as python setup.py develop as in does not install dev deps and never removes any deps.
muppetjones commented 3 years ago

Completely agree. All good options. Maybe a fourth option that's the reverse of (2): Only install but add another flag (though it would get confusing with the existing "remove" flag).

On Tue, Mar 30, 2021 at 11:08 AM Owais Lone @.***> wrote:

@muppetjones https://github.com/muppetjones Agree. Try adding any dev dependency to your project that has an indirect dependency on any popular like like requests. dephell is one example. This means poetry will requests from your project's venv causing a lot more pain than poetry install solves in the first place. This forces you to install your dev dependencies into test projects or re-install test project's dependencies after every single poetry install --no-dev.

This means poetry does not provide a valid alternative to plain old setup.py develop command. Imagine setup.py develop either forcing you to install all your dev depencies or forcefully uninstalling any common dependencies from the target env. It wouldn't make any sense.

Ideally, poetry should do one of the following:

  1. Change behavior of install command to not remove packages and only install them.
  2. Remove by default but allow a --no-remove flag to disable behavior.
  3. Offer a new poetry develop command that behaves the same way as python setup.py develop as in does not install dev deps and never removes any deps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/python-poetry/poetry/issues/3139#issuecomment-810388677, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPSOEY4SJMUJZYPBBXRFGDTGHZO5ANCNFSM4SILRKRA .

-- This email and any attachments may contain confidential information. If you believe that you have received it in error, please delete the email and attachments and then notify Tempus by calling 800.739.4137.

muuvmuuv commented 3 years ago

+1 for no-remove. We use Poetry in our Dockerfile's with an image that comes with pre-installed pip's. Now running install removes those pre-installed required dependenices. In our case a new flag or command would even be smarter --ci like npm ci, which does some things by default like no venv creation and the discussed removal.

alfrisch commented 3 years ago

we see this problem appearing with poetry version 1.1.6 when installing poetry in a venv with pip, version 1.1.5 does not show this behavior. poetry install will remove webencodings and such poetry will not work anymore.

+1 for the --no-remove option.

evgmoskalenko commented 3 years ago

Hello, most problem that poetry removes 'appdirs' applications needed to work with poetry itself. If you use dev dependencies which use something like 'appdirs' dependencies (and poetry use this dependencies) and run commands poetry install --no-interaction --no-dev

Look:

[tool.poetry.dependencies]
python = "^3.9"
confluent-kafka = "^1.6.0"
pydantic = "^1.7.3"
six = "^1.15.0"
prometheus-client = "^0.9.0"
sentry-sdk = "^0.20.3"

[tool.poetry.dev-dependencies]
black = "^20.8b1"
Installing dependencies from lock file
Package operations: 5 installs, 0 updates, 5 removals
  • Removing appdirs (1.4.4)
  • Removing packaging (20.9)
  • Removing pexpect (4.8.0)
  • Removing ptyprocess (0.7.0)
  • Removing pyparsing (2.4.7)
  • Installing typing-extensions (3.7.4.3)
  • Installing confluent-kafka (1.6.1)
  • Installing prometheus-client (0.9.0)
  • Installing pydantic (1.8.1)
  • Installing sentry-sdk (0.20.3)
 % poetry show -t
black 20.8b1 The uncompromising code formatter.
├── appdirs *
├── click >=7.1.2
├── mypy-extensions >=0.4.3
├── pathspec >=0.6,<1
├── regex >=2020.1.8
├── toml >=0.10.1
├── typed-ast >=1.4.0
└── typing-extensions >=3.7.4

So, if you want to install dev dependencies in the next step of the pipeline for linting or run unit tests, you will crash with an error (so poetry will not work anymore):

poetry install
Traceback (most recent call last):
  File "/usr/local/bin/poetry", line 5, in <module>
    from poetry.console import main
  File "/usr/local/lib/python3.9/site-packages/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File "/usr/local/lib/python3.9/site-packages/poetry/console/application.py", line 7, in <module>
    from .commands.about import AboutCommand
  File "/usr/local/lib/python3.9/site-packages/poetry/console/commands/__init__.py", line 4, in <module>
    from .check import CheckCommand
  File "/usr/local/lib/python3.9/site-packages/poetry/console/commands/check.py", line 2, in <module>
    from poetry.factory import Factory
  File "/usr/local/lib/python3.9/site-packages/poetry/factory.py", line 18, in <module>
    from .repositories.pypi_repository import PyPiRepository
  File "/usr/local/lib/python3.9/site-packages/poetry/repositories/pypi_repository.py", line 33, in <module>
    from ..inspection.info import PackageInfo
  File "/usr/local/lib/python3.9/site-packages/poetry/inspection/info.py", line 25, in <module>
    from poetry.utils.env import EnvCommandError
  File "/usr/local/lib/python3.9/site-packages/poetry/utils/env.py", line 23, in <module>
    import virtualenv
  File "/usr/local/lib/python3.9/site-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/local/lib/python3.9/site-packages/virtualenv/run/__init__.py", line 7, in <module>
    from ..app_data import make_app_data
  File "/usr/local/lib/python3.9/site-packages/virtualenv/app_data/__init__.py", line 9, in <module>
    from appdirs import user_data_dir
ModuleNotFoundError: No module named 'appdirs'

Probably this can be treated with such a crutch in dockerfile when you put such dependencies in the system :-(

pip install --no-cache-dir --upgrade pip poetry>=1.0.0 && \
poetry config virtualenvs.create false && \
poetry install --no-interaction --no-dev && \
pip install --no-cache-dir appdirs && \
popadi commented 3 years ago

Anyone managed to make it work without hacks in the Dockerfile? Or is there any plan to fix this issue? As other pointed out, if you use dephell or black and you have them in pyproject.toml in the dev-dependencies section stuff blows up.

By the way, another way to make it work is to install poetry like this:

RUN pip install -U pip \
    && curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
ENV PATH="${PATH}:/root/.poetry/bin"

instead of what we most of us probably have:

RUN pip install "poetry==$POETRY_VERSION"

Doing this will allow poetry and its dependencies to be isolated from yours. However, a lot of people don't like or even discourage installing stuff like that (mostly because there's no guarantee that the content of that file won't change).

jbraswell commented 3 years ago

We are also seeing this problem (or a very similar problem) with version 1.1.6. Specifically, in our case, dependencies in poetry.lock that are a sub-dependency of a VCS dependency are removed (or not installed).

Things seem to work fine with 1.1.5.

agronholm commented 3 years ago

+1 for no-remove. We use Poetry in our Dockerfile's with an image that comes with pre-installed pip's. Now running install removes those pre-installed required dependenices. In our case a new flag or command would even be smarter --ci like npm ci, which does some things by default like no venv creation and the discussed removal.

I think this stems from the mindset that Poetry manages its own virtualenv, but when virtualenv creation is disabled, the same behavior remains and this was never fixed when adding the ability to work outside of virtualenvs.

ryangoss commented 3 years ago

I am running poetry inside a conda virtualenv 

conda create -n new_env python=3.8
conda activate new_env
conda install poetry
poetry new project
cd project

I am able to add some packages, but I am getting some failures. Then when I try to install a subset of packages, I get cases where poetry will remove packages which make it impossible to run poetry again without uninstalling and reinstalling:

% poetry add scikit-learn
Using version ^0.24.2 for scikit-learn

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 1 update, 51 removals

  • Removing appnope (0.1.2)
  • Removing argon2-cffi (20.1.0)
  • Removing async-generator (1.10)
  • Removing backcall (0.2.0)
  • Removing bleach (3.3.0)
  • Removing cffi (1.14.5)
  • Removing cycler (0.10.0)
  • Removing decorator (5.0.9)
  • Removing defusedxml (0.7.1)
  • Removing entrypoints (0.3)
  • Removing ipykernel (5.5.5)
  • Removing ipython (7.23.1)
  • Removing ipython-genutils (0.2.0)
  • Removing jedi (0.18.0)
  • Removing jinja2 (3.0.1)
  • Removing jsonschema (3.2.0)
  • Removing jupyter-client (6.2.0)
  • Removing jupyter-core (4.7.1)
  • Removing jupyterlab-pygments (0.1.2)
  • Removing jupyterlab-widgets (1.0.0)
  • Removing kiwisolver (1.3.1)
  • Removing markupsafe (2.0.1)
  • Removing matplotlib-inline (0.1.2)
  • Removing mistune (0.8.4)
  • Removing nbclient (0.5.3)
  • Removing nbconvert (6.0.7)
  • Removing nbformat (5.1.3)
  • Removing nest-asyncio (1.5.1)
  • Removing notebook (6.4.0)
  • Removing pandocfilters (1.4.3)
  • Removing parso (0.8.2)
  • Removing pexpect (4.8.0)
  • Removing pickleshare (0.7.5)
  • Removing pillow (8.2.0)
  • Removing prometheus-client (0.10.1)
  • Removing prompt-toolkit (3.0.18)
  • Removing ptyprocess (0.7.0)
  • Removing pycparser (2.20)
  • Removing pygments (2.9.0)
  • Removing pyrsistent (0.17.3)
  • Removing python-dateutil (2.8.1)
  • Removing pytz (2021.1)
  • Removing pyzmq (22.0.3)
  • Removing send2trash (1.5.0)
  • Removing six (1.16.0)
  • Removing terminado (0.10.0)
  • Removing testpath (0.5.0)
  • Removing tornado (6.1)
  • Removing traitlets (5.0.5)
  • Removing webencodings (0.5.1)
  • Removing widgetsnbextension (3.5.1)
  • Updating numpy (1.20.2 -> 1.20.3)
...

Example: webencodings missing means the next time I try to run poetry ... it will fail because webencodings is missing.

matthewarmand commented 3 years ago

Based on our experience and testing, looks like this might be part of the root issue: https://github.com/python-poetry/poetry/issues/4051

rassie commented 3 years ago

Isn't the issue here that poetry still thinks it's managing a dev-dependency even if --no-dev is given?

akvadrako commented 3 years ago

At least when not using a virtualenv, it seems wrong that poetry would remove distributions that were already installed since they might be used by other services on the system.

For production, binary packages with dependencies on system libraries should be installed by the system package manager, not from pypi. But it is handy to include those deps in a dev or extra section as a convenience to new developers on other platforms.

EDIT: actually, based on my testing, poetry doesn't remove optional dependencies if the extra isn't selected, but they are already installed. So a work-around is to use extras instead of dev.

timothyjlaurent commented 3 years ago

I'm also seeing this with the --no-root option

RUN poetry install --no-root

COPY $PROJECT_DIR/$SRC_DIR ./$SRC_DIR
RUN poetry install -vvv

In the first poetry install --no-root, it 

#16 [11/16] RUN poetry install --no-root
#16 sha256:2d8b4b8d22953e8b9ca32f0a600edc08ce154e4553d62cd0b800ca507b5a0ed3
#16 0.677 Skipping virtualenv creation, as specified in config file.
#16 2.105 Installing dependencies from lock file
#16 4.577 
#16 4.577 Package operations: 69 installs, 6 updates, 9 removals
#16 4.577 
#16 4.577   - Removing backports.entry-points-selectable (1.1.0)
#16 4.907   - Removing cffi (1.14.6)
#16 5.282   - Removing cryptography (3.4.7)
#16 5.706   - Removing distlib (0.3.2)
#16 6.007   - Removing platformdirs (2.2.0)
#16 6.315   - Removing pycparser (2.20)
#16 6.637   - Removing python-dotenv (0.19.0)
#16 7.001   - Removing pyyaml (5.4.1)
#16 7.499   - Removing virtualenv (20.7.2)

and then it error because it can't find virtualenv.

This happens whether I pip install or use pipx or install via install-poetry.py

Jarvis1Tube commented 3 years ago

I found a solution for my self. I used Docker, when the trouble happen too. So in case of Docker I created virutalenv using by poetry instead of standard venv module. Before:

ARG venv="/usr/app/env"

FROM python:3.9-buster AS builder
ARG venv

ENV PIP_DISABLE_PIP_VERSION_CHECK=true
ENV PATH="${venv}/bin/:$PATH"

RUN python3 -m venv "${venv}"

COPY pyproject.toml .
COPY poetry.lock .

RUN pip3 install poetry

RUN poetry config virtualenvs.create false --local
RUN poetry config virtualenvs.in-project false --local

RUN poetry install --no-dev  # here it falls

After:

FROM python:3.9-buster AS builder
WORKDIR /mnt

COPY pyproject.toml .
COPY poetry.lock .

RUN pip3 install poetry

RUN poetry config virtualenvs.create true --local
RUN poetry config virtualenvs.in-project true --local

RUN poetry install --no-dev

It works because poetry installed in OS environment and poetry has no access to extra libraries installed with OS.

tedivm commented 3 years ago

This is really not what I would have expected for this behavior. There should be a way to say "I want to keep what is installed, but also install these packages". Right now making multiplatform container images (that support arm64 and amd64) with poetry is a nightmare that requires building the packages separately, and this issue makes that even worse.

dfontenot commented 3 years ago

Saw there was a workaround for Docker builds with no virtualenv in which you install poetry via curl. At the moment that workaround is also not working (am attempting to install dependencies with: poetry install --no-interaction --no-ansi).

ERROR: Cannot uninstall 'packaging'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I also tried rolling back to poetry 1.1.5 but that still had the issue. I found the issue all the way down to 1.0.0. At that point I just gave in and installed in a virtualenv on my docker image.

tedivm commented 3 years ago

There isn't a workaround as far as I can tell. I've just stopped using poetry because of this, since I need to support multiplatform projects and apparently that's not something this project can currently handle.

adiun commented 2 years ago

Just to add my use case - I have 2 poetry packages that I want installed into a Docker image used at development time:

I'm noticing that Package A installs and then Package B ends up removing Package A's dependencies and Jupyter dependencies. For now I have to make sure the dependencies are exactly the same across both packages and include Jupyter dependencies :( This is a bad workaround and now i'm not sure how to make this work with Poetry...

tschm commented 2 years ago

I have a docker image we use for testing. It has preinstalled numerous packages, such as pylint, black etc. We use gitlab. We essentially install our poetry managed library directly into a container relying on the test image. Poetry comes and deletes first some packages in this container. This is insane. Can you please add a no-remove flag asap.

demizer commented 2 years ago

Another use case for the searchers: poetry is excluding the coverage module required by pytest-cov in docker which results in ModuleNotFoundError: No module named 'coverage' when running pytest from CI.

egormkn commented 2 years ago

I've got ModuleNotFoundError: No module named 'entrypoints' after installing dependencies with POETRY_VIRTUALENVS_CREATE=false environment variable. It seems that during the installation step Poetry updated entrypoints (0.3 -> 0.4), and was not able to find this module anymore for its own purposes.

MingStar commented 2 years ago

We hit the same issue when specifying docker-compose package with a specific platform linux.

With poetry 1.1.13, when running poetry install on MacOS, it will remove all dependencies of docker-compose, which includes requests, despite requests IS a dependency of multiple other installed packages, resulting in runtime error on MacOS.

djmattyg007 commented 2 years ago

Poetry shouldn’t import the virtualenv package at all if it doesn’t need to create a virtualenv. It should be an optional dependency. This would help mitigate the main issue in this thread, though it’s obviously not the full solution.

nylocx commented 2 years ago

I just got bitten by this too while building inside a CI/CD pipeline in a docker environment without the venv option. I currently run a pip install tomlkit virtualenv right after the poetry install --no-dev, but there should be an option to disable this or the removal should be deactivated at all if running without an virtual environment, with a big warning that you should only do that in special cases like docker in ci/cd or something.

finswimmer commented 2 years ago

Hello everyone,

since poetry 1.2.0a2 poetry will no longer remove installed packages unless you use the --sync flag. Please see https://python-poetry.org/blog/announcing-poetry-1.2.0a2/#package-operations-and-synchronization-improvements for more information.

fin swimmer

Cajuteq commented 2 years ago

Hello, I just tested this with version 1.2.0b1 and i got an error 

PoetryException Failed to install root/.cache/pypoetry/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl

that trace back to 

Processing /root/.cache/pypoetry/artifacts/47/3f/ce/b240169f7d8bef1ff24a0269b709721ce86543c2ec25e0b6adb2c2d7ac/packaging-21.3-py3-none-any.whl
Installing collected packages: packaging
Attempting uninstall: packaging
Found existing installation: packaging 20.9
ERROR: Cannot uninstall 'packaging'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

so it seems to me the fix correctly stopped to remove packages Package operations: 37 installs, 6 updates, 0 removals, 4 skipped but still failed to handle cases when updates are done by removing previous version before installing newer.

My use-case is a docker image that pre-install some common dependencies for later use in the ci/cd process of several similar projects. I use the following Dockerfile (alpine based)

RUN curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0b1

ENV PATH /root/.local/bin:$PATH

COPY poetry.lock .
COPY pyproject.toml .

RUN rm -rf /root/.cache/pypoetry/

RUN poetry config virtualenvs.create false \
  && poetry install --no-interaction --no-ansi -vvv

I don't have the problem if i put virtualenvs.create true but building deps in a venv makes it impossible to reuse the dependencies in the later ci/cd builds which slows down the pipeline a lot as installing all deps takes minutes ( #1579 could solve the use case while still using the venv)

tedivm commented 2 years ago

@Cajuteq - you may want to open a new ticket so this gets some attention.

abn commented 2 years ago

@Cajuteq I am not a 100% across your use case. But I can tell you that using a system environment is not really a practice I recommend even in containers.

You could try something like this if it helps.

FROM docker.io/python:3.10

RUN curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0b1

ENV PATH /root/.local/bin:$PATH

WORKDIR /opt/app

COPY poetry.lock .
COPY pyproject.toml .

RUN poetry config virtualenvs.in-project true \
  && poetry install --no-interaction --no-ansi
$ cat pyproject.toml 
[tool.poetry]
<--snipped-->

[tool.poetry.dependencies]
python = "^3.10"
poetry = {git = "https://github.com/python-poetry/poetry.git", rev = "master"}

[tool.poetry.group.dev.dependencies]
#pytest = "^7.1.1"
<--snipped-->
$ docker build -v /tmp/venv:/opt/app/.venv:z -t local/test .
<--snipped-->
STEP 7/7: RUN poetry config virtualenvs.in-project true   && poetry install --no-interaction --no-ansi
Recreating virtualenv foobar in /opt/app/.venv
The virtual environment found in /opt/app/.venv seems to be broken.
Installing dependencies from lock file

Package operations: 38 installs, 0 updates, 0 removals

<--snipped-->
COMMIT local/test
--> c0b1dd2af56
Successfully tagged localhost/local/test:latest
c0b1dd2af566219fc04dc6aafc62929f9c94d8789d47865d661c3b989773019f
$ # add new dev dependency for testing
$ cat pyproject.toml 
<--snipped-->
[tool.poetry.dependencies]
python = "^3.10"
poetry = {git = "https://github.com/python-poetry/poetry.git", rev = "master"}

[tool.poetry.group.dev.dependencies]
pytest = "^7.1.1"
<--snipped-->
$ docker build -v /tmp/venv:/opt/app/.venv:z -t local/test .
<--snipped-->
STEP 7/7: RUN poetry config virtualenvs.in-project true   && poetry install --no-interaction --no-ansi
Installing dependencies from lock file

Package operations: 6 installs, 0 updates, 0 removals

  • Installing attrs (21.4.0)
  • Installing iniconfig (1.1.1)
  • Installing pluggy (1.0.0)
  • Installing py (1.11.0)
  • Installing tomli (2.0.1)
  • Installing pytest (7.1.1)
COMMIT local/test
--> 2edf0b040c3
Successfully tagged localhost/local/test:latest
2edf0b040c3bdf427d461c0f0f640311411952af68f1d4b80b8e09317bbccb7f

Alternatively, you can mount in /root/.cache/pypoetry/virtualenv. This can either be a docker volume or a host file-system directory.

ZiyanLiu16 commented 2 years ago

I've got ModuleNotFoundError: No module named 'entrypoints' after installing dependencies with POETRY_VIRTUALENVS_CREATE=false environment variable. It seems that during the installation step Poetry updated entrypoints (0.3 -> 0.4), and was not able to find this module anymore for its own purposes.

@egormkn I have the exact same issue. Did you manage to resolve this?

runqinggg commented 1 year ago

Hi team, what's the consideration around the following design decision? I have a similar situation where the package A is defined as optional in my pyproject.toml, and our team only install A when the corresponding platform doesn't provide package A (e.g. we use package A's CPU version, but if the platform has a GPU version of package A, we'll not install package A)

Extras are not sensitive to --sync. Any extras not specified will always be removed.

https://python-poetry.org/docs/master/cli/#:~:text=Extras%20are%20not%20sensitive%20to%20%2D%2Dsync.%20Any%20extras%20not%20specified%20will%20always%20be%20removed.

github-actions[bot] commented 9 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.