pip install -e . equivalent?

[kootenpv]

We cannot run pip install -e . because there is no setup.py. What would be the way to achieve this using poetry?

[sdispater]

No, this is not possible right now. And I don't think I will add an equivalent since it always seemed like a hack to me. But I might be wrong.

Why do you need pip install -e .? So I can see if there is a "poetry way" to do what you want.

[kootenpv]

I often depend on some local packages, for which I want to be able to change some code and not have to reinstall.

[sdispater]

I see.

The thing is poetry is a dependency/project management tool and is not intended for such use cases.

However, if you really want to to this, you can build your project poetry build and take the setup.py file from the generated tarball and then you should be able to do pip install -e.

[kootenpv]

Ah sorry, with local I mostly meant "developing local packages". I do believe poetry is also for helping with that? It sounds like barely any work to automate that part and create e.g. the poetry develop command.

[sdispater]

Well, poetry can help you develop local packages by orchestrating your dependencies, sure. But it is not meant to support the use case of having only local packages that depend on each other since this is not the best practice that poetry tries to endorse.

However, if you are willing to make a Pull Request, I would gladly take a look at it.

[winni2k]

I might be off the mark here, but in my experience, I usually use pip install -e . in order to allow my acceptance tests to see the python project I am testing. Is that what we are talking about here?

[sdispater]

I don't see why you would need pip install -e . for your tests to work. If you are inside a virtualenv your dependencies should be picked up by your tests.

In poetry that would be:

poetry install  # install your dependencies
poetry run pytest tests/

[kootenpv]

I'm not sure, but when you share a module in git, with other dev members, it's nice when it is updated automatically on pull. Or is this automatically managed in pycharm or something? I use emacs without virtualenvs but just simply different python versions.

[winni2k]

In Pycharm you set the environment by choosing a python interpreter in your directory tree. Packages are discovered from that AFAIK.

[winni2k]

I guess this is a bit of a style question, but I like to build a wheel for local packages and use pip to install them into my venv. Is that the workflow that is intended by poetry?

[sdispater]

@winni2k Not at the moment, no. For now, poetry assumes each dependency is located in a remote repository (PyPI or a private one) or is a VCS dependency, and does not support local packages.

That might change in the future, however but I can't give you an ETA.

I am trying first to support the most common use cases, like installing from PyPI before diving into more specific workflow, especially if they are not the recommended way of managing packages.

[kootenpv]

It now also does not allow me to use it in the interactive interpreter :/

I tried using poetry build, then copy out the setup.py to the root level but that doesn't work. I'm very disappointed :(

Until this becomes important, I wish there'd be a nice hack. I'm wondering though how do you develop - how do you avoid having to constantly reinstall?

[sdispater]

@kootenpv I just tested and copying the setup.py file works.

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository.

[woutervh]

I don't know how you develop, but I never need pip install -e .. I either upload my packages to PyPI or a private index, or I install from a git repository

Ugh, that is not developing, that is installing a finished package.

These are roughly equivalent, in your src dir:

pip install -e . python setup.py develop

It's called "installing a package in development mode". They are the starting point of beginning development. If you are not familiar with that, then I honestly do not understand how you do development.

I know two ways of doing development without it:

export PYTHONPATH= or in your code: import sys sys.path.append(....)

Are you recommending this? (I hope not)

[sdispater]

No, I don't modify sys.path(). And please don't be condescending.

So, if I develop a library (or an application for that matter) I put myself in my project directory. At this point I have access to my module: I can import it in my tests to test it.

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

If you need it in another project, and that is why you install it in develop mode, the you have dependencies between project and in this case I recommend to use a private index and release preversions or installing from git repository and updating when needed.

[merwok]

There are many communities of Python users, using different categories of development tools. Depending on practices (e.g. code in src dir vs code at repo root) or tools (e.g. Django runserver vs. Pyramid pserve), you can rely on «current dir is sys.path[0]» or you always install in develop/editable mode.

[merwok]

Another use case is when you’re trying to fix something in a library used by your project. From your project’s virtualenv, you run pip install -e path/to/library so that every change you make in the library repo is directly picked up in your project where you are testing the results.

[sdispater]

@merwok I understand your point, but most of the time you don't need it, in my opinion.

Regarding you last example, this will be supported in the next feature release since it will add the ability to install from a directory.

I know that there is a lot of different workflows out there, but poetry can't support them all. It tries to enforce some of them to help make the dependency management and packaging in Python easier and for that it has to be somewhat opinionated.

I am not saying it will never make it into poetry but since I am the sole developer I have to focus on a workflow that will suit most cases to develop applications and libraries.

[kootenpv]

@sdispater Thanks for your way of developing (private index / from git), I wasn't aware of another way.

I can agree that under some conditions it is more proper and might lead to fewer bugs!

In my case, I do a lot of data science and "live" in Interactive python. It's very convenient when I can just do pip install -e . once and I'm just able to change the code so that in another module it will be picked up and see the results quickly (rather than having to run any install command again). That is the step before writing tests.

I understand you are hesitant from your perspective as it does not seem like a really clean use case, and it looks like the solution would have to be a bit hacky?

Still, I hope you could guide us and be accepting of a solution that might help a few (and not bother others)!

Currently the only way I see is something like:

1. poetry build
2. find the build package (?)
3. extract it
4. move the setup.py a level up
5. find the python version (?) used by poetry and use it like python setup.py develop

Does this sound like the best way?

[ipmb]

I'll chime in and say I also use pip install -e . when doing local development. It is helpful for a couple reasons:

1. It puts my project on the Python path. I don't have to depend on being in a specific directory for things to work which feels hacky to me.
2. It sets up scripts and entry points I wouldn't have otherwise.

[sdispater]

@kootenpv Don't get me wrong, I can understand why you would use pip install -e, especially if you want to debug a dependency.

And, like I said, directory dependencies will be introduced in the next feature release so, with them, you will be able to reproduce this behavior.

[moigagoo]

@sdispater

You can test this yourself: clone the poetry repository place yourself in it install the dependencies with poetry install and execute poetry run pytest tests/ and everything will work. You do not need pip install -e .

Hi! Could you please clarify how you run tests with pytest without installing poetry in the virtualenv? I can't import my package from pytest tests because it's not installed and resides one level higher than the tests.

It seems like poetry has the same source layout, but for some reason the imports work (although the tests fail). Am I missing something?

UPD: Even the pytest docs say that you can only run tests against an installed version of your package under this layout: https://docs.pytest.org/en/latest/goodpractices.html#tests-outside-application-code

[moigagoo]

Sorted this one out. Should've put an empty __init__.py file in tests.

[blueyed]

poetry build and take the setup.py

What do you think about some option to generate the setup.py file directly?

[bersace]

Is it related to #47 ?

If i summarize it, the need is that a poetry managed project should be installable as editable. e.g. i want to debug that library used by my project, i git clone it and poetry install -e the path ?

[bersace]

Or maybe the goal is just to have project console scripts installed in virtualenv ?

[sdispater]

@bersace If you want to test you console scripts, it's possible via the poetry script my-script. For example for poetry that would be poetry script poetry.

Also, I understand the use case of pip install -e path/to/some/dependency/ to allow debugging a dependency since you won't have to reinstall each time you make a modification. That's why it will be possible in the next feature release.

However, I do not think pip install -e . (or python setup.py develop) is that useful and as such introducing an equivalent in poetry is not something I plan on adding at the moment.

Now, that being said, if someone can give me a compelling use case where it's necessary, I will gladly change my view on the subject.

[kootenpv]

I think I get what you mean: you want to enable the option for a package (A) to list a development version of another package (B).

I perhaps see a problem though.

I believe you're saying you could list B as a requirement in A in the new version by listing pip install -e path/to/some/dependency/ in A's poetry file (or some similar syntax, maybe just relative path)?

Isn't the problem then that if both are poetry packages, A will not be able to pip install -e B since poetry does not generate a setup.py for B?

[sdispater]

@kootenpv Exactly.

I think it would take the form:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/" }

And with the equivalent of -e:

[tool.poetry.dependencies]
dependency = { "path" = "relative/path/to/folder/", develop = true }

Regarding the issue when the dependency is also a poetry package: at the moment, there won't be another solution but to generate a temporary setup.py to pip install it. I am not a fan of it but for now it will do, at least until the ability to specify a build backend in pyproject.toml is supported (see https://www.python.org/dev/peps/pep-0517/).

[bersace]

@sdispater I don't understand why i have to put the develop mode in the file. If i want to debug a poetry based dependency, i don't want to modify my pyproject.toml for that, just to poetry install --develop ../my-deps-fork/.

About console scripts, I saw poetry script .... I find odd to not use scripts the same way i will do in production, especiallty when scripts may be calling them each-other, or if i have Makefile or shell script calling the script.

For example, I use https://github.com/amoffat/sh to write functionnal tests of https://github.com/dalibo/ldap2pg . see https://github.com/dalibo/ldap2pg/blob/master/tests/func/test_sync.py#L34-L36 . When developing, it just use virtualenvs scripts. On CI, it tests scripts installed from RPM, without modification nor configuration. It's harder to do this using poetry.

I still think that some editable mode of poetry is required for the comfort of developing project with poetry.

[sdispater]

@bersace The pyproject.toml is what orchestrates your dependencies when using poetry that's why I suggest putting there.

Now, that being said, I can add a --develop option to install (and to add too) to tell poetry: I want this dependency from my dependency set to be installed in development mode. The thing is with that option is if you forget to set it it will install the dependency in non editable mode.

Regarding an editable mode for poetry, I could add it also but this is, at the moment, not high on my priority list. I am the sole developer at the moment so I have to prioritize the feature I want to implement in poetry. But if someone wants to tackle this , they are welcome to and I'd gladly review it.

[bersace]

The pyproject.toml is what orchestrates your dependencies when using poetry that's why I suggest putting there.

I see pyproject.toml as a way to share a reproduceable set of dependencies for both runtime and development. Actually i would create a third env in between : CI. having a reduced set of dependency on CI is nice. e.g. hupper is not needed on CI.

I may be wrong :-)

[merwok]

Indeed, it is very useful to have different sets for runtime, test and dev dependencies. (Also CI bootstrap and server-only deps!)

[digitalresistor]

@bersace that's what extras as for, IMHO. Add an extra for testing that doesn't have hupper, but does have all of the testing extras for example.

[mmerickel]

@sdispater First of all thanks for poetry, I think it's looking fantastic as I dive into it more.

The big use-cases for editable-mode installs the way I see it are:

1) Making entry points and console scripts available in the virtualenv. As you've said yourself, and I agree with, these are properly covered by poetry run and poetry script which should make these available inside that environment. If entry points are not discoverable inside poetry run python then I'd be discouraged but I haven't tested this yet.

2) Using your project out-of-place... as you've said yourself your approach requires you to either munge the PYTHONPATH or to develop within the source folder. Editable mode installs a .pth file in the site-packages, thus placing your project on the path as long as you have access to the python binary.

3) Installing and working on third-party dependencies in your project. This, again, you're handling via [tools.poetry.dependencies] with editable=true which is probably an acceptable way to go although it's unfortunate that you need to temporarily edit a file that's generally version controlled in such a way that you probably don't want to commit. This is probably where poetry install --develop comes into play.

I may be missing something but point 2 seems to be the big case that's unaddressed.

[sdispater]

@mmerickel Thanks for the kind words and the detailed use cases.

Regarding entry points, like you said, poetry script takes care of it. However, there is a limitation: you have to use it from project's directory. So, you won't be able to install your command "globally". This is where a develop command could be useful.

And regarding installing dependencies in editable mode, I think that a --develop option is the way to go. However, there are some details to sort out, like if you do poetry install --develop my-dependency and later simply do poetry install should the editable mode be disabled?

[kootenpv]

@sdispater So I just tested it:

pip install -e .
pip instal my_package

In my case it looks like it was installing it again, but the binding shows it is still using the develop version.

It looks like pip does not care about version when you just do pip install my_package (we knew that, but also not with editable/dev already installed). Whenever you write pip install my_package==1.0.0, it will install it from pip if it doesn't have it locally.

In case you are talking about whether it remove the editable mode status in the file when you poetry install, I guess you wouldn't want that, but maybe a warning that an editable package is being linked?

As for being allowed to publish to pypi - I think that should not be allowed. Or maybe an input You have an editable package X, are you sure want to publish [y/N]?

[radix]

I think #47 (path dependencies) should be able to solve this use case now?

[sdispater]

@radix #47 is a first step towards it but path dependencies cannot be installed in editable mode at the moment. This is something planned for the next feature release.

[mmerickel]

@sdispater For the editable project mode from point 2 above, I suppose it's important to ask whether poetry has goals of providing a PEP 517 compliant build system for projects in the future? Presumably if that were the case then this is solved via pip install -e <path> on the path containing the pyproject.toml which would tell poetry to install itself in editable mode. Similarly a pip install <path> (not editable) is effectively equivalent to poetry build && pip install dist/*.whl in a future when you can upload a project created with poetry to PyPI without needing to generate a fake setup.py. I know this is futuristic but it helps me understand where poetry is going and whether it even makes sense for poetry to support the develop mode - although I suppose it could support poetry develop and in the future embrace the pip integration. This seems like a pretty logical step for poetry as it already has ambitions of making projects redistributable via poetry build and poetry publish.

[sdispater]

@mmerickel Yes, I intend to support a build system. I am not sure it will be integrated directly into poetry since I don't want people to have to depend on the whole poetry's codebase to build a project. So it will likely be another package whose sole purpose will be to build a project from a poetrypyproject.toml file which is more or less the purpose of the masonry package inside poetry so it might be extracted from there.

[moigagoo]

With the introduction of directory dependencies, I thought poetry add --dev --path . mypackage would be possible and would completely solve this issue. This would install the package itself in the virtual environment from the current directory.

However, you can't do that right now (sample taken from my project Cliar):

$ poetry add --dev --path . cliar
[InvalidProjectFile]
[dev-dependencies.cliar] {'path': '.'} is not valid under any of the given schemas

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--allow-prereleases] [--dry-run] [--] <name> (<name>)…

(The line cliar = {path = "."} is added to pyproject.toml though.)

@sdispater Is it an issue to report or a design decision?

P.S. My two cents on the topic, and why I need poetry add --dev --path . cliar to work. Like your own Cleo, Cliar is a package to create CLIs. Because the purpose of the tool is to generate a commandline interface from Python code, I'm testing it this way:

• Create a .py file with the interface definition. This file obviously needs to be able to import cliar in order to create the CLI to test.
• In test files, I'm calling the .py files with subprocess.run like so: run('python clitest.py arg1 arg2', shell=True).
• I'm checking the output with Pytest's capfd fixture.

It would be really nice to put all the tests in tests folder, put the sample .py files in a subfolder, and discover them in the tests using pytest-datadir plugin.

However, I currently cannot do that. Whereas test_*.py files can import cliar, the sample files can't since the cliar package is not installed.

The workaround I use now is place the tests and the sample in the root dir. But I'd rather keep my tests in an isolated directory.

If there is a way to solve my problem without poetry add --dev --path . cliar I'm not seeing, please advise.

Thanks!

[sdispater]

Release 0.10.0 adds a new develop command which does something similar to pip install -e ..

The install command now supports a --develop option which installs path dependencies in development/editable mode. 

[kootenpv]

Thanks a lot for the nice surprise!

A couple of observations:

• the --develop option is missing from the CLI command install (good: badusage is thrown, and poetry develop documentation exists)
• I thought I can add a folder containing a local package and install it using poetry install --develop, but that's not how it works?
• poetry develop allowed me to install the "current" package, and import it, but only if I'm in that folder (e.g. in ~/mypackage. You cannot import mypackage from the example location ~/otherpackage)

Maybe I'm misunderstanding the usage?

[sdispater]

@kootenpv You must pass values to the --develop option. Let's say you have my-pkg1  and my-pkg2 that are path dependencies, you would do:

poetry install --develop my-pkg1 --develop --my-pkg2

poetry develop installs the package in development mode for the project's virtualenv, so that it's available when executing poetry run. If you manage your virtualenvs yourself then it will be installed in the current activated virtualenv.

[moigagoo]

@sdispater Sorry, I still don't quite understand why I can't import my package after running poetry develop. When I run poetry develop, the package gets installed in the current venv, right? So, if I import it from a script in a project's subfolder and run the script with poetry run foo/bar.py, it should work, since poetry run uses the same venv.

It doesn't work however. Is it a bug or expected behavior?

P.S. Congrats on the big release and thanks a ton for your work!

[sdispater]

@moigagoo Yes it should work. So if it's not it's likely a bug.

[moigagoo]

@sdispater made a separate issue https://github.com/sdispater/poetry/issues/162

I think this one can be closed, since the equivalent of pip install -e . does exist now.

[p5a0u9l]

So, this issue is marked "closed" and yet it seems the command mentioned above develop is now deprecated. has pip install -e . survived through some other mechanism? or is there another thread discussing this?

[jacebrowning]

@p5a0u9l The behavior on installing the package in an editable mode (poetry develop) is now part of poetry install.