rectalogic commented 5 years ago

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Question

pipenv has pipenv clean which removes all packages that are no longer in Pipfile.lock from the virtualenv. What is the equivalent with poetry? The docs say there is no need for pipenv sync and pipenv clean due to the remove command https://github.com/sdispater/poetry#remove-command

But how does this work? I would need to run poetry remove XXX against all my virtualenvs - manually keeping track of all packages that have been removed and remembering to remove them myself?

Is there not any way to sync my poetry.lock to my virtualenv to ensure the virtualenv only has the packages specified in poetry.lock installed?

I have a virtualenv on each box I deploy to, and keep pyproject.toml and poetry.lock under source code control. I then want a way to ensure each of those virtualenvs only has the packages in poetry.lock installed.

sdispater commented 5 years ago

I am not sure I understand. Why would your packages no longer be in poetry.lock but still in your virtualenv?

rectalogic commented 5 years ago

Actually I just realized poetry remove won't even work.

Here I remove a package from en1 and then switch to env2 and attempt to remove it from there and it fails

(env1) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry remove boto3
Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 9 removals

Writing lock file

  - Removing boto3 (1.9.47)
  - Removing botocore (1.12.47)
  - Removing docutils (0.14)
  - Removing futures (3.2.0)
  - Removing jmespath (0.9.3)
  - Removing python-dateutil (2.7.5)
  - Removing s3transfer (0.1.13)
  - Removing six (1.11.0)
  - Removing urllib3 (1.22)
(env1) ➜  poetrytest source env2/bin/activate
(env2) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry install
Installing dependencies from lock file

Nothing to install or update

(env2) ➜  poetrytest /Users/aw/Library/Python/2.7/bin/poetry remove boto3

[ValueError]
Package boto3 not found

remove [-D|--dev] [--dry-run] [--] <packages> (<packages>)...

sdispater commented 5 years ago

Oh. I think I understand. You update the lock locally and then deploy. Is that right?

rectalogic commented 5 years ago

I am not sure I understand. Why would your packages no longer be in poetry.lock but still in your virtualenv?

Because I have many virtualenvs - one per EC2 instance. So I copy over my poetry.lock and want to sync each virtualenv using that poetry.lock, adding/upgrading/removing packages as needed to keep the virtualenv in sync (similar to how pipenv sync && pipenv clean would work.

rickhutcheson commented 5 years ago

We have this problem using Poetry as well. One developer poetry removes and deploys the new poetry.lock/pyproject.toml. The other devs pull the changes, but poetry install does not remove the old packages. Are there any plans to support this workflow?

kevinkjt2000 commented 5 years ago

The README mentions clean and sync, but these are not commands on version 0.12.11. I am also curious to know what steps to take to ensure that my local virtualenv matches poetry.lock.

Edit: I have found that I can run poetry shell, then exit, and part of the output will be the path to the virtualenv folder. I use rm -rf on that folder, and then poetry install runs clean.

jobec commented 5 years ago

Same here. would be great if packages not in poetry.lock were removed. Now there is a chance of such packages piling up and unexpectedly influencing your application (some packages look in specific locations for plugins and such).

Recreating a virtual environment each and every time is time consuming and unless it can happen automatically when checking out a branch, developers aren't going to do it.

seansfkelley commented 5 years ago

Another use-case for this is CI build caching. If you remove dependency in a branch and the CI build for that branch pulls from a cache that still has that dependency, I definitely want it removed.

This is how Yarn works which is important because then yarn install means "make my build reproducible" rather than "make sure I have all the dependencies I need", which is not the same thing.

This ticket is in conflict with https://github.com/sdispater/poetry/issues/533 (though AFAICT Poetry indeed does not uninstall dependencies, so not sure what that ticket's about).

jobec commented 5 years ago

It looks like if this line would loop over the installed packages, that it would result in obsolete packages to be cleaned up.

https://github.com/sdispater/poetry/blob/f2e2ed470ca1560111226b5f6f8f9dc4a28d319a/poetry/puzzle/solver.py#L96

I'm trying see what it does if I changed it, but I can't seem to get poetry running from it's own git repo inside PyCharm's debugger 🤔

seansfkelley commented 5 years ago

@kevinkjt2000 AFAICT the only references to clean and sync in the README are talking about Pipenv commands, not Poetry. Also, poetry debug:info (0.12.x) will print out the virtualenv without you having to do weird things to find it.

jobec commented 5 years ago

I did an attempts in pull request #1037 to remove obsolete packages when running poetry install. Still a work in progress, but it looks like it's cleaning up obsolete packages (e.g. installed, but not in the list of resolved packages)

pmav99 commented 5 years ago

@jobec a couple of comments if you don't mind: a) It is not clear to me if removing packages is a good default for poetry install b) I am not a native speaker, but the term "obsolete" seems to be slightly misused in this context. Not sure what to propose though. Maybe "unspecified" packages? c) This is a breaking change. Among other things, it would be making it impossible to implement something like this. IMHV, removing packages should be either a separate subcommand or at the very least a poetry install flag.

jobec commented 5 years ago

First of all, the PR is just a beginning. It's no where near finished nor fine tuned. And I'm open for all feedback.

A) True, although I think people expect it to behave like that. I know I did. "Freezing dependencies" as in "THIS is how it should be". Anyway, I agree that changing current behavior might be a bridge to far.
B) I'm also not a native speaker, but obsolete is the first thing that crossed my mind. I guess there are better synonyms: unknown, unlocked, unused, unspecified, rogue...
C) The comment you point at is indeed a useful use-case. Combined with A) it might be a good reason not to make this behavior the default. Anyway, I think it's best to put it as a flag to poetry install. What about --clean. Matching with git clean

seansfkelley commented 5 years ago

a) I assert that it is a good default, though that depends on if (per my earlier comment) you view "install" to mean "make my build reproducible" (as I do) rather than "make sure I have all the dependencies I need" (which is a good start, but incomplete).

Regarding point (c). From the linked comment:

You might still install a package with confliciting dependencies though. Perhaps it would make sense to have a special mode of poetry add that will resolve dependencies and install a package without adding it to pyproject.toml.

The semantics of this use case seem weird and difficult. If the "temporary" package has overlapping dependencies with non-temporary packages, what happens? Does it change the versions of the common dependencies? If so, does it update the lock file? Does it instead just fail outright?

The OP on that issue says:

Then there are tools that I use personnally to develop, but are not required to run tests. Examples are ipython, icecream, pdb++. Those are more personal preferencies that real requirements.

In node, the accepted solution for this use-case is either (1) install it globally since it's not relevant to the project at hand or (2) install it as part of the project because you use it to develop the project (and hence it is a dev dependency). It seems to me that trying to support a third, middle road is likely to cause more confusion than help.

pmav99 commented 5 years ago

@seansfkelley @jobec I, at the very least, hope that we can all agree that:

"make my build reproducible"
"make sure I have all the dependencies I need"

are both valid and discrete usecases :)

WRT to which one should be the default, I honestly don't have a strong opinion. I tend to prefer the latter option as the default one since it is more common for development + it is the current behaviour. Adding an extra flag and/or using an explicit subcommand on deployment scripts seems more natural.

@seansfkelley I transferred the discussion for installing packages without adding them to pyproject.toml here

pmav99 commented 5 years ago

I guess there are better synonyms: unknown, unlocked, unused, unspecified, rogue...

"orphan" is an option too

drunkwcodes commented 5 years ago

poetry autoremove.

clean has different meaning in other package managers, which clear up cache packages not been installed yet.

seansfkelley commented 5 years ago

poetry autoremove assumes you want a different command for this. What's the use-case for a separate command instead of changing the behavior of install? Would it make sense to use autoremove at any other time except immediately after install?

I tend to prefer the latter option as the default one since it is more common for development + it is the current behaviour.

Possession is 9/10ths of the law. :) You would almost certainly only want the more permissive behavior in development, but even in development, how desirable is it? Enough to be the default?

Adding an extra flag and/or using an explicit subcommand on deployment scripts seems more natural.

In general, yeah, I think requiring CI commands to bear the burden of extra configuration is okay. But at least from my experience using Yarn, npm and Cargo, stricter behavior by default leads to fewer red herrings and unexpected behavior. In fact, I assumed Poetry did do this until I discovered the hard way (leading to my first comment here, https://github.com/sdispater/poetry/issues/648#issuecomment-482763789) that it did not.

jobec commented 5 years ago

I'd say: poetry install --keep-untracked

With a (new) default of removing untracked packages (e.g. not in poetry.lock), as I think the larger percentage of users expect it to work as such.

From the main website:

Develop Poetry comes with all the tools you might need to manage your projects in a deterministic way.

Track Having an insight of your project's dependencies is just one command way.

Poetry uses the exact versions listed in poetry.lock to ensure that the package versions are consistent for everyone working on your project.

Your CI server, production machines, other developers in your team, everything and everyone runs on the same dependencies, which mitigates the potential for bugs affecting only some parts of the deployments.

seansfkelley commented 5 years ago

Nice citation of the website! :) I also think "untracked" is a good word to describe these dependencies.

pmav99 commented 5 years ago

+1 for untracked.

Apart from that, I completely agree that having a way to have a "purged" virtualenv is really important. That being said though, I also think that not purging the virtualenv while developing can also be useful.

The reason I don't really like changing the defaults is that I prefer adding a flag on a deployment script rather than having to add a cli flag when using poetry interactively (i.e. while developing)

drunkwcodes commented 5 years ago

Why? What we are talking about is removing packages, not installing. You will not install any untracked packages anyway.

pmav99 commented 5 years ago

@drunkwcodes this discussion is related to #951

drunkwcodes commented 5 years ago

I won't look at that for a while...

There are many ways to install untracked packages, not just with poetry install --strange-option. Having a poetry autoremove command to remove universally untracked packages is a proper thing to do.

This is useful when you have local editable installations not packaged, and do not want to create new venv.

seansfkelley commented 5 years ago

Having a poetry autoremove command to remove universally untracked packages is a proper thing to do.

Well, this is the point under discussion. I think there is value in increasing the scope of install to handle this case as well (though I disagree with @pmav99 over whether it's a flag or not), specifically, because then install can become a one-stop shop to make your dependencies reflect the lockfile without having to juggle multiple commands. One extra command wouldn't be a huge burden for those of us having this discussion here, but keeping this behavior localized to install instead means that (1) it's more discoverable, (2) it's more correct by default and (3) Poetry doesn't bloat.

That said, in writing this comment I did discover a couple of interesting related commands in other tools:

npm ci is a variant of npm install specifically designed to be stricter for a CI environment, noting that it "...can help catch errors or inconsistencies caused by the incrementally-installed local environments of most npm users."
yarn prune and yarn dedupe are specifically called out as not necessary due to the behavior of yarn install.
yarn check --verify-tree is designed to check what's on disk against what's in the lock file. (Though it should be noted that as a heavy Yarn user for years, I never knew this command existed until now because yarn install is already so dependably predictable.)
cargo fetch is designed exclusively to populate Cargo's dependency cache, which is the closest it has to poetry install. Cargo thinks about this whole problem differently than Poetry or npm/Yarn, preferring to think about run and build rather than a separate dependency-installation step. run and build as a prerequisite ensure that your dependencies are in the right state. fetch is presented as a mere convenience if you're going off-network, as opposed to a core part of the development flow.

My preferred style here is Yarn's, which has to tolerate users installing extra stuff (like npm) but by default tries to assert total control over node_modules. It should be noted that the analogy isn't perfect due to the way node handles conflicting versions (via duplication), which allows it to sidestep some complexity that Poetry has to handle.

On the topic of whether it should be default-on or default-off, I think that depends on one's preferred development flow. As someone who does not install untracked dependencies hardly ever, I clearly prefer @jobec's suggestion of poetry install --keep-untracked. The opposite (--remove-untracked?) would be annoying for me to type every time, which is I suppose the same argument that frequent untracked-dependency users would make about --keep-untracked. Not sure how to resolve that difference of opinion. :)

drunkwcodes commented 5 years ago

It's absolutly doable. What we need is to figure out all kinds of installations which can be with poetry projects at the same time.

And doing it professionally and better than a brutal virtualenv restart is the only concern.

jacebrowning commented 5 years ago

@drunkwcodes In my opinion, this comment does not add anything of value to this discussion. Please refrain from providing your commentary unless you're representing a position that has not already been presented in the various open issues on this project.

jobec commented 5 years ago

Who actually has the final say in this? @sdispater ?

pmav99 commented 5 years ago

@seansfkelley thank you for the thorough comparison)

(2) it's more correct by default

Care to elaborate? It kind of feels like "correctness" in this context is rather subjective than objective (i.e. each tool uses UI/UX different conventions).

seansfkelley commented 5 years ago

Point (2) above was referring only to the comparison of having the behavior in install versus a new command, not about whether install should be remove-by-default or keep-by-default. With that in mind, what I meant by "more correct by default" is that baking this behavior into install means that the default action (poetry install) will be opinionated about what it means to be correct and which mode is preferred, rather than having a pair of related commands and letting you figure it out. Poetry is an opinionated tool, and I think that should extend to this as well.

Note also I said "more correct", not simply "correct", because as we have determined we don't agree about which should be the default but I think we agree that it would be preferable to keep it in install, whatever it is...?

kylebebak commented 4 years ago

Running poetry remove pkg fails, if pkg is in poetry.lock but not in pyproject.toml.

This is really surprising behavior, given that poetry update pkg works just fine.

poetry remove pkg should remove pkg from poetry.lock, even if it's not in pyproject.toml -- after all, this is exactly how poetry update pkg works.

➜  cloud-django git:(328/branch) ✗ poetry remove django-stubs

[ValueError]
Package django-stubs not found

➜  cloud-django git:(328/branch) ✗ poetry update django-stubs
Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 0 installs, 0 updates, 2 removals

  - Removing dataclasses (0.6)
  - Removing django-stubs (0.12.0)
➜  cloud-django git:(328/inspection_event_publishing) ✗

Or with an image:

Screen Shot 2019-10-03 at 6 19 38 PM

trim21 commented 4 years ago

Maybe add a command poetry ci like npm ci🤔. Not changing default behavior and don't need to run command like auto remove/clean after install.

(And much shorter than poetry install

ayroblu commented 4 years ago

Just wondering if this is moving anywhere? It looks like there was a good proposal: poetry install --keep-untracked Not sure if just not enough resources to action this?

To clarify the use case if it's not obvious, if you collaborate with anyone, including CI or have multiple machines, you probably add and remove dependencies organically, it makes no sense to keep removed ones around, especially if they interact with other tools

jobec commented 4 years ago

I had a PR partially implementing this. But is has been closed in the meanwhile.

Also, without a go or direction from a maintainer, there's not much fun in working on it of it won't be merged.

finswimmer commented 4 years ago

Hello @jobec,

I guess I closed your PR as this has the status WIP for a very long time and you didn't react on the stale bot message. So I thought you didn't work anymore on it.

I found this feature very useful and appreciate your contribution. So please go on! :+1:

fin swimmer

jobec commented 4 years ago

Still the question remains:

Maintainers, what direction to go in? Remove obsolete/stale/untracked/... packages by default, or keep them.

finswimmer commented 4 years ago

At the end @sdispater has to decide, because it's a new feature.

I support the version to remove untracked packages by default and have the --keep-untracked parameter for those who really need it (I read some of the arguments why people would do this, but still doesn't understand why ...).

PetterS commented 4 years ago

This is very important and I don't know of any way to do this.

Say several developers work on a project. One performs an update which removes a package from the pyproject and lock files. How will the other developers ever remove this package?

Pipenv, for all of its flaws, at least has pipenv clean.

EDIT: Seems the install command removes packages in some cases: https://github.com/python-poetry/poetry/blob/d27a119d391a83d2f816b68129dc62346f7aef03/poetry/installation/installer.py#L409 (when the dev flag changes).

PetterS commented 4 years ago

@sdispater did you have any input on finswimmers suggestion of --keep-untracked above? I am happy to finish my PR if this is what we want. Having a way of cleaning out unused dependencies is crucial to us.

sdispater commented 4 years ago

Unfortunately, we can't make this the default behavior since it's a breaking change in my opinion.

We should add an option that requires opt-in like --remove-untracked.

PetterS commented 4 years ago

Sure! I'll modify my PR, then?

We could add a configuration file that sets the default behavior, no problem.

PetterS commented 4 years ago

@sdispater @finswimmer Please let me know if there is anything I should do with my PR at this point.

kasteph commented 4 years ago

2172 fixes this issue and should be available in the next release.

earshinov commented 4 years ago

@stephsamson , is there any date for the next release? We are waiting for this feature.

dash-samuel commented 3 years ago

Running poetry remove pkg fails, if pkg is in poetry.lock but not in pyproject.toml.

This is really surprising behavior, given that poetry update pkg works just fine.

poetry remove pkg should remove pkg from poetry.lock, even if it's not in pyproject.toml -- after all, this is exactly how poetry update pkg works.
➜  cloud-django git:(328/branch) ✗ poetry remove django-stubs

[ValueError]
Package django-stubs not found
➜  cloud-django git:(328/branch) ✗ poetry update django-stubs
Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 0 installs, 0 updates, 2 removals

  - Removing dataclasses (0.6)
  - Removing django-stubs (0.12.0)
➜  cloud-django git:(328/inspection_event_publishing) ✗
Or with an image:

I am running into the exact same problem.

Aiky30 commented 1 year ago

Is this closed without a solution? Seems a little overkill to have to destroy my venv to bring my venv back in check with a projects packages. Teams do remove packages ....

trim21 commented 1 year ago

Is this closed without a solution? Seems a little overkill to have to destroy my venv to bring my venv back in check with a projects packages. Teams do remove packages ....

In newer poetry you can run poetry install --sync and it will keep env synced with pyproject.toml

Aiky30 commented 1 year ago

Awesome thanks @trim21 , I didn't see this option in poetry list. Should help anyone falling into this thread in the future.

trim21 commented 1 year ago

Awesome thanks @trim21 , I didn't see this option in poetry list. Should help anyone falling into this thread in the future.

poetry install --help

Description:
  Installs the project dependencies.

Usage:
  install [options]

Options:
      --without=WITHOUT      The dependency groups to ignore. (multiple values allowed)
      --with=WITH            The optional dependency groups to include. (multiple values allowed)
      --only=ONLY            The only dependency groups to include. (multiple values allowed)
      --no-dev               Do not install the development dependencies. (Deprecated)
      --sync                 Synchronize the environment with the locked packages and the specified groups.

github-actions[bot] commented 6 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

How do I uninstall packages that have been removed from pyproject.toml? #648

Question

2172 fixes this issue and should be available in the next release.