search

Pylons / pyramid

Pyramid - A Python web framework
https://trypyramid.com/
Other
3.94k stars 880 forks source link

Support poetry, flit, pipenv, or ...? #3270

Open stevepiercy opened 6 years ago

stevepiercy commented 6 years ago

This is a placeholder for supporting whatever becomes the successor to pip and managing virtual environments in Pyramid, including poetry, flit, and pipenv. Please add to it.

Questions

  1. How and where to install SUCCESSOR?
  2. Where does installation of SUCCESSOR fit into our current installation documentation?
  3. Are there any features of SUCCESSOR that we would use beside installation of packages and distribution of an app?
  4. What would happen to the environment variable $VENV in most of our commands?
  5. How do we deal with features in setup.py only, specifically entry points of console_scripts and paster.app_factory?
  6. What will we do with the scaffolds and pcreate?
  7. Should we remove instructions for how to make a distributable Pyramid application, and replace it with references to suggested tools?
  8. Assuming a rewrite of commands for SUCCESSOR, do we retain or remove Windows commands for everything except installation and setup of an environment, with a disclaimer that it is up to the user to know how to translate Unix commands to their platform of choice? Refs: https://github.com/Pylons/pyramid/issues/3260#issuecomment-384217118

Things to update

  1. Documentation wherever pip is used.
  2. Pyramid's packaging related files, including setup.py.
  3. Pyramid cookiecutters.

References

  1. Poetry: https://github.com/sdispater/poetry
  2. Summary article about pipenv: https://realpython.com/pipenv-guide/
mmerickel commented 6 years ago

The main issue I have seen in my experience using pipenv is that it doesn't mix very cleanly with a setup.py. It's possible but you run into questions about where to define the dependencies. I have one project where the Pipfile is literally just a shim pointing to the setup.py where all the deps were declared. This works pretty well actually but it does feel odd.

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.6.5"

[packages]
myapp = {path = ".", editable = true}

[dev-packages]
myapp = {path = ".", extras = ["testing"], editable = true}

The workflow here is to basically edit setup.py and then run pipenv lock which updates Pipfile.lock and you never really change the Pipfile.

The main reasons why we use setup.py in our projects is for console_scripts and paster.app_factory entry points. If a) python had a better way to define entry points, or b) if we were less dependent on them, then we could probably embrace pipenv a little bit more cleanly.

stevepiercy commented 6 years ago

What would be a better way to define entry points? Is the PyPA working on this?

This does not bode well: https://docs.pipenv.org/search/?q=entry+point

mmerickel commented 6 years ago

Entry points is an issue that pypa is starting to talk about on distutils-sig as the next likely thing to get split out from setuptools... but very slowly. The more practical option I think will be to rip setup.py out of the guide entirely and just show people how to make pyramid apps without needing to make a distributable python package. Lots of projects seem to like that approach - see warehouse for an example.

stevepiercy commented 6 years ago

I've updated and added to the questions above.

Regarding making a distributable Python package, do you mean you want to remove the these two pages?

If so, I'd like to at least mention why someone would want to make a dist and provide references to resources for how to do so, instead of simply deleting those pages from the two wiki tutorials.

Or do you mean something else?

piotr-dobrogost commented 6 years ago

Maybe https://github.com/takluyver/entrypoints could be used instead of setuptools/pkg_resources for entry points?

mmerickel commented 6 years ago

@piotr-dobrogost Unfortunately the issue here is how to declare entry points, not how to consume them. That project only looks to consume existing *.egg-info/entry_points.txt files in sys.path.

piotr-dobrogost commented 6 years ago

Ok, how about switching from setuptools to flit which does have support for entry points (https://flit.readthedocs.io/en/latest/pyproject_toml.html#entry-points-sections)?

mmerickel commented 6 years ago

I like flit and would be open to promoting it more. Unfortunately for the purposes of integrating with pipenv it's basically the same as setup.py - it's making your project a redistributable package and thus provides a place to define dependencies which conflicts with the Pipfile. We would need to make a decision where the dependencies are defined. For example I think you'd end up with a Pipfile like I pasted above.

The alternative that integrates with pipenv more cleanly is the flask approach of punting on this and not advocating patterns that use setup.py. You're basically responsible yourself for putting your project on the PYTHONPATH however you want - usually just running it out of the CWD. The real benefits to having an installable package are:

1) Your project, even in editable mode, can be run from anywhere. You can import it and thanks to pth files it's always on the path. 2) Your project has a way to describe dependencies. 3) Your project has a way to provide implementations for entry points.

pipenv's approach (in my experience using it) has been to directly compete with point 2 and doesn't provide a solution for points 1 and 3. It seems mostly useful for pulling together third party projects into a lockfile and less about managing the link between the project-under-development and the virtualenv. Please correct me if anyone disagrees, but this is what I've seen, and yes I have used it for a real project in this way.

mmerickel commented 6 years ago

https://github.com/sdispater/poetry is by far the best solution I have seen to managing a project yet. I think it blows pipenv's approach out of the water from what I have seen so far and solves all the issues I've mentioned above.

merwok commented 6 years ago

Compared to flit for example, it seems like poetry is made by one person who does not follow Python packaging discussions, so I’m not sure about the degree of integration (i.e. implementation of standards) that this tool provides. For example, the author doesn’t see the point of develop installs, and doesn’t accept the uses cases brought up in the ticket.

digitalresistor commented 6 years ago

@merwok there are a variety of projects that don't use develop installs at all, for example warehouse doesn't instead it expects warehouse to be in it's PYTHONPATH automatically. Others like Flask and I think Django too, punt on having installable packages, unless you manually do a bunch of work (its not documented, and not the default recommendation) and thus they don't have a need for editable installs.

While we have promoted setup.py and having installable packages for Pyramid applications it is not a requirement. The author also agrees that there are some use cases for editable installs: https://github.com/sdispater/poetry/issues/34#issuecomment-382784476 and seems amenable to adding them for dependencies.

I personally really like the idea of a single file that contains package pins/Information rather than a "Pipfile" vs "setup.py" (or one of it's replacements) where the data is duplicated, or you end up having on rely on the other.

Using flit doesn't do a bunch of the project management things you might have come to expect from tools like npm/yarn.

mmerickel commented 6 years ago

Right... poetry solves the editable install issue by basically requiring you to use poetry run or poetry scripts for your commands during development... which, coming from nodejs with yarn run and npm run is actually a really nice thing in my opinion.

Finally, the sdist/wheel it outputs if you do choose to poetry build does contain the appropriate metadata for pip to install it correctly in anyone else's environment. With this in mind I've re-thought the benefits of an editable install entirely. For example, it's a pretty common problem for people to forget to do a pip install -e . after changing entry points. This doesn't suffer from that issue because it makes sure they're correct each time you run your commands in development mode.

On top of that, I guess that as @bertjwregeer said, poetry is adding some editable install support in a future version anyway.

josuemontano commented 6 years ago

After reading this thread I decided to try poetry, this is the project I created: https://github.com/josuemontano/API-platform.

The development configuration works as expected, but in production running poetry build doesn't seem to work properly, gunicorn can't find the package created by poetry -I might be missing something. poetry develop works fine in production.

digitalresistor commented 6 years ago

@josuemontano poetry build will create an sdist/wheel from your repository. You'll want to use tooling like pip to install the built packages (see dist/ folder after you run poetry build).

josuemontano commented 6 years ago

Got it. Thanks @bertjwregeer! https://github.com/josuemontano/API-platform/blob/master/scripts/predeploy#L3-L4. Hopefully I can make a cookiecutter out of this soon.

mmerickel commented 5 years ago

FWIW to get rid of pyramid's requirement for an entry point it's possible to replace use = egg:myapp in your ini file with simply use = call:myapp:main. It's something we could consider just to make the "common case" for people a bit simpler when integrating with pipenv where entry points are not considered something an application would need, only libraries. This would remove the need for a setup.py in most cases and all dependencies would be managed through pipenv directly.

cjw296 commented 5 years ago

Was any progress ever made of having config optionally come from something other than an ini file?

tseaver commented 5 years ago

@mmerickel

It's something we could consider just to make the "common case" for people a bit simpler when integrating with pipenv where entry points are not considered something an application would need, only libraries.

I guess the ironies are in the fire, given that entry points were designed for applications, rather than libraries.

mmerickel commented 5 years ago

Was any progress ever made of having config optionally come from something other than an ini file?

@cjw296 yes, see pyramid's usage of plaster in newer releases

I guess the ironies are in the fire, given that entry points were designed for applications, rather than libraries.

The irony is not lost on me... but it is lost on the pipenv developers that have closed issues opened about adding entry point support to it with significant hostility. The main thing to point out is that your link references "installed distributions" and "packages" and pipenv explicitly wants nothing to do with these things. It only wants to manage a virtualenv of stuff, and is not interested in helping you develop a package. That's between you and your build system (setuptools/flit/etc). Since entry points are currently defined as package metadata, pipenv has no intention to provide any specific support for them above what I described in https://github.com/Pylons/pyramid/issues/3270#issuecomment-384487700 (installing your package in editable mode with a setup.py or pyproject.toml next to it).

cjw296 commented 5 years ago

Given the hostile maintainers and lack of functionality, I'd advocate just giving up on pipenv and putting weight behind poetry. I've found it to be a much more pleasant tool to work with, it features a real dependency solver, and the maintainer is polite, pragmatic and practical.

zupo commented 5 years ago

Is anyone using poetry on a decently sized project? I've tried to use it twice in the past year on a 300+ deps project and it exploded. I didn't really have time to investigate and decided to wait a bit until it matures. Has it matured yet?

stevepiercy commented 5 years ago

On top of that, I guess that as @bertjwregeer said, poetry is adding some editable install support in a future version anyway.

FTR, this was done in Poetry 0.10.0 on May 28, 2018.

Has it matured yet?

Hard to say without specifics for your use case, but it has been in active development with recent bug fix and 1.0.0a releases. It looks like they're 2/3 of the way there to 1.0.0.

zupo commented 5 years ago

Hard to say without specifics for your use case, but it has been in active development with recent bug fix and 1.0.0a releases. It looks like they're 2/3 of the way there to 1.0.0.

Of course, that's why I am asking is anyone actually using it in production on a sizeable project and it works great for them. I'm looking for a trigger to try it again :)

digitalresistor commented 5 years ago

@zupo I am happily using it on a project right now for a customer. The main issue I've found is when people upload broken packages to PyPi with bad metadata, which I don't fault poetry for, because pip just blindly downloads and executes code, whereas poetry attempts to avoid that to be able to do proper dependency resolution.

For building an application it is working really well for me right now, and I really enjoy using it. I haven't used it for building a library.

zupo commented 5 years ago

Great to hear, thanks!

cjw296 commented 5 years ago

@zupo - in passing, 300 deps seems quite extreme for one app, how come so many deps?

zupo commented 5 years ago

@cjw296: it's a hosting automation project. Pyramid is used to manage 500+ servers deployed on 20+ hosting providers, dns records on 15 +providers, email services, DNS propagation services, etc. Lots of API clients that we depend on.

merwok commented 4 years ago

Here’s my return on experience FWIW:

flit

Pros:

Cons:

poetry

Pros:

Cons:

pipenv

Tries to integrate existing tools together (pip, virtualenv, pip-tools, safety) and the result sometimes doesn’t feel well integrated. Maintainer hostility was mentioned previously in this ticket (discussions shut down abruptly, cowboy development, etc).

Last I used it, adding one dependency caused updates to existing ones, which is really not wanted and inconvenient. Have switched to poetry for these projects.

See also notes by Nick Coghlan in https://discuss.python.org/t/developing-a-single-tool-for-building-developing-projects/2584/17

Conclusion

The classic distutils-based packaging tools have improved a lot; we can build wheel+sdist with setuptools, develop install with pip, manage pinned requirements with pip-tools, check metadata and upload securely with twine, detect reST issues or use Markdown in metadata long description. That said, there is a lot of old or contradictory doc out there, and it’s five tools with different configs and UX.

flit is easy to use and avoids some classes of problems entrirely. The Pyramid documentation could change setuptools instructions to flit, if it provides all features required by the examples [edit: and if it’s acceptable to have sub-par dependency management (either all open without any conflict/compat check, or all manually pinned and updated)]. I like how it’s one tool for packaging tasks (init, build, upload), and lets you use other tools for other things (pytest, tox, bump2version, etc). That said, I could see people disliking the requirement for VCS setup before you can use flit.

Poetry in my opinion has a couple rough edges but is overall rather great, and could become the community-standard way of managing a web app (except maybe for Django, where typically your app is one or many packages imported from current dir, without install step).

For me, flit is great to package a pure-Python library, and poetry is really nice for an app. A project can also start with flit and move to poetry without a ton of effort (see also https://github.com/takluyver/flit/issues/115#issuecomment-524285026). I’m personally happy using both, but if Pyramid docs want to show only one tool, I think poetry wins. To package Pyramid itself, I think flit would be enough, but poetry may work too if you wanted only one tool overall.

digitalresistor commented 4 years ago

At my new place of employment almost all projects are run directly in a container or lambda functions and there is no pip install project or anything like that. There's a requirements.txt file and a bunch of code.

Recently a co-worker started with the example from trypyramid.com and ran into the following:

  1. Doesn't show how to do a POST request
  2. Doesn't show how to receive JSON post data
  3. Doesn't link to documentation where he could get the above information

There's no real "quick start"... except that we have 2, maybe 3 of them, which caused a bunch of confusion:

  1. We have the the quick tutorial: https://docs.pylonsproject.org/projects/pyramid/en/latest/quick_tutorial/
  2. We have the quick tour of Pyramid: https://docs.pylonsproject.org/projects/pyramid/en/latest/quick_tour.html
  3. And then we have the wiki tutorial (2 of them): https://docs.pylonsproject.org/projects/pyramid/en/latest/tutorials/wiki2/

but the quick tutorial doesn't cover the basics of "how do I deal with various HTTP verbs" nor does it deal with "how do I deal with JSON posted data or sending JSON back out".

I couldn't point my co-worker at one place in the documentation where he could easily see an example and copy and paste (and yes, people learn that way). Instead the project is now a flask project :-/

I went off on a tangent (unrelated to what this ticket is about), and I'll likely create more tickets at a later point in time (although they are likely to get lost anyway), but the point is that focusing on how to create a python package to ship Pyramid is something many people don't care about.

Look at warehouse or hypothesis (thanks for the examples @mmerickel) and notice that they too don't care about packaging their Pyramid applications, however a lot of our narrative documentation currently depends on someone learning all of the packaging stuff just to get going with Pyramid.

I hate to suggest maintaining more documentation, but this is an instance where maybe we should consider simplifying and removing any discussion about packaging your Pyramid application, or at least leaving that as an optional in case you want to take advantage of certain Pyramid features (like easy overrides and all that fun stuff).

stevepiercy commented 4 years ago

but the quick tutorial doesn't cover the basics of "how do I deal with various HTTP verbs" nor does it deal with "how do I deal with JSON posted data or sending JSON back out".

I love me a good hijacking! I would suggest you direct your cow-orkers to have a look at pyramid_openapi3, thanks to Nejc, Domen, and other Niteo folks. There are two demos, and a third full-blown RealWorld.io app. After looking at that, are there any gaps that need to be covered?

digitalresistor commented 4 years ago

Yes, pyramid_openapi3 is a project that allows you to implement what used to be known as swagger endpoints and does not solve the problem of showcasing how a view gets called to handle a POST request.

i.e. there are no examples that I can easily find that have:

@view_config(name='someviewname', route_name='some route', request_method='POST')

Along with an explanation of how that works. The example real world.io app linked also uses open API, setup.py and is a package that can be installed.

stevepiercy commented 4 years ago

The example in https://trypyramid.com links to the following page, where further down are the predicate arguments: https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/viewconfig.html#predicate-arguments

I get your point, though. The content in the narrative docs is presented through technical terms of "predicate" and so on, instead of a "show me a JSON REST API example".

It would definitely make a good cookbook recipe or an additional step in the Quick Tutorial.

mmerickel commented 4 years ago

There's two practical things I could pull out of this discussion:

1) There's alternative project management tools now and setuptools is not the clear winner. 2) It's unclear in the docs and cookiecutter how to use Pyramid without setuptools.

I'd like to find a middle-ground where Pyramid stops making decisions here for the user so that they can adapt the cookiecutter to their own needs. The source of these issues is not specifically dependency management. I think it's actually how the ini file connects to the code through entry points.

So I see two possible changes, in the following order:

1) Modify the cookiecutter to stop using entry points. 2) Modify the cookiecutter to stop using setuptools.

First, modify the ini file to use call: instead of egg: in the use = statements. I've worked with lots of developers and the roundabout way that Pyramid connects the ini file to the project via entry points has never not been a source of confusion when focusing on application development. This would be the largest benefit of the change.

Second, converting a requirements.txt to pipenv, poetry, flit, etc is pretty straightforward and probably a better starting point for projects who want to make a decision there. So we could stop using setup.py / setuptools in the default scaffold / docs. It's a somewhat significant change to Pyramid's narrative docs but a small-ish change to the actual cookiecutter. I'd basically move from setuptools to defining a simple requirements.txt and dev-requirements.txt files with the dependencies.

We'd keep all the virtualenv stuff we do now, but we'd stop defining the project as a publishable package and instead focus just on installing dependencies with pip and requirements.txt.

stevepiercy commented 4 years ago

In our docs and tutorials, what would we suggest to users to manage their *requirements.txt files? I would prefer to recommend a very brief list of tools for managing requirements, and possibly including which tool is useful for a given scenario, and suggest not to manually edit them.

If we want to avoid endorsing a specific tool in the docs, then we probably need to resort to a statement like, "manage your requirements.txt using your favorite tool or manually edit it".

zupo commented 4 years ago

For the purpose of following this tutorial, and for getting started with your first Pyramid project, it is fine to manually edit requirements.txt file. That said, when you are preparing to publish your work we suggest you start using tools such as X, Y and Z for automatic management of requirements.

merwok commented 4 years ago

I have found pip-tools mostly convenient to generate full list of pinned requirements from a file containing loose requirements (i.e. only direct app dependencies, with no version constraints or range constraints, edited by devs). Generating a (or many) requirement file(s) ensures broad compatibility with update checkers (like dependabot, now part of github), Heroku deployments, tools like safety, etc.

stevepiercy commented 4 years ago

@merwok would you please write one or two sentences for pip-tools to use in the docs, similar to what @zupo did? It's a good example of language I will copy-paste into the docs. For the miscellaneous dependency management tools, I think the best place for descriptions is in glossary.rst. Then we can do:

:term:`my-requirements-tool`

and call it soup.

mmerickel commented 4 years ago

The goal of requirements.txt would be to show env/bin/pip install -r requirements.txt in the tutorials which is something everyone understands. There is no pinning, upgrading, etc required in the tutorials. They can then adapt that to use pip-tools, poetry, pipenv, setuptools, etc however they wish.

I do not agree that Pyramid should document how to use it with pip-tools, pipenv, poetry, etc in the core docs.

stevepiercy commented 4 years ago

I agree we should not document how to use those tools in any way. That would be duplicating the tools' documentation.

I think it is fine to provide links to a few of those tools, and a one or two sentence description for when to use each tool ("distributing", "installing dependencies", etc.) in the Glossary. There are precedents for distribute, distutils, pip, and setuptools.

While we are talking about packaging stuff and whether to document it, do we want to delete the distributing pages from the tutorials and anything from Creating a Pyramid Project?

merwok commented 4 years ago

A link to https://packaging.python.org/tutorials/packaging-projects/ should be enough. One could argue that the instructions there are not the most modern (e.g. could be using static metadata in setup.cfg, or use flit instead of setuptools), but that’s actually the point: let packaging people update that packaging tutorial, then people coming from Pyramid docs will benefit from updates.

zupo commented 1 year ago

In case someone wanders here from the Internets, I've created a Poetry-based scaffold of the Pyramid's "SQLAlchemy + URL dispatch" wiki tutorial: https://github.com/zupo/tutorial

yozachar commented 1 year ago

rye seems promising.

wodny commented 1 year ago

rye seems promising.

Although I understand the decision I do not like it:

No System Python: I can't deal with any more linux distribution weird Python installations or whatever mess there is on macOS. I used to build my own Pythons that are the same everywhere, now I use indygreg's Python builds. Rye will automatically download and manage Python builds from there. No compiling, no divergence.

yozachar commented 1 year ago

The author of rye opened & closed https://github.com/mitsuhiko/rye/issues/16 to add support for local Python. But then he add the functionality to use custom toolchains. Ref: https://github.com/mitsuhiko/rye/pull/19.

You can use locally installed Python as mentioned in the documentation:

Additionally it's possible to register an external toolchain with the rye toolchain register command.

rye toolchain register /path/to/python

The name of the toolchain is picked based on the interpreter. For instance linking a regular cpython installation will be called cpython@version, whereas linking pypy would show up as pypy@version. From Rye 0.5.0 onwards -dbg is appended to the name of the toolchain if it's a debug build. To override the name you can pass --name:

rye toolchain register --name=custom /path/to/python