search

David-OConnor / pyflow

An installation and dependency system for Python
MIT License
1.32k stars 44 forks source link

readme

crates.io version Build Status

Pyflow

Simple is better than complex - The Zen of Python

Pyflow streamlines working with Python projects and files. It's an easy-to-use CLI app with a minimalist API. Never worry about having the right version of Python or dependencies.

Example use, including setting up a project and switching Py versions: Demonstration

If your project's already configured, the only command you need is pyflow, or pyflow myscript.py; setting up Python and its dependencies are automatic.

Goals: Make using and publishing Python projects as simple as possible. Actively managing Python environments shouldn't be required to use dependencies safely. We're attempting to fix each stumbling block in the Python workflow, so that it's as elegant as the language itself.

You don't need Python or any other tools installed to use Pyflow.

It runs standalone scripts in their own environments with no config, and project functions directly from the CLI.

It implements PEP 582 -- Python local packages directory and Pep 518 (pyproject.toml).

Installation

Quickstart

Quick-and-dirty start for quick-and-dirty scripts

Why add another Python manager?

Pipenv, Poetry, and Pyenv address parts of Pyflow's raison d'être, but expose stumbling blocks that may frustrate new users, both when installing and using. Some reasons why this is different:

Perhaps the biggest philosophical difference is that Pyflow abstracts over environments, rather than expecting users to manage them.

My OS comes with Python, and Virtual environments are easy. What's the point of this?

Hopefully we're not replacing one problem with another.

Some people like the virtual-environment workflow - it requires only tools included with Python, and uses few console commands to create, and activate and environments. However, it may be tedious depending on workflow: The commands may be long depending on the path of virtual envs and projects, and it requires modifying the state of the terminal for each project, each time you use it, which you may find inconvenient or inelegant.

I think we can do better. This is especially relevant for new Python users who don't understand venvs, or are unaware of the hazards of working with a system Python.

Pipenv improves the workflow by automating environment use, and allowing reproducible dependency graphs. Poetry improves upon Pipenv's API, speed, and dependency resolution, as well as improving the packaging and distributing process by using a consolidating project config. Both are sensitive to the environment they run in, and won't work correctly if it's not as expected.

Conda addresses these problems elegantly, but maintains a separate repository of binaries from PyPi. If all packages you need are available on Conda, it may be the best solution. If not, it requires falling back to Pip, which means using two separate package managers.

When building and deploying packages, a set of overlapping files are traditionally used: setup.py, setup.cfg, requirements.txt and MANIFEST.in. We use pyproject.toml as the single-source of project info required to build and publish.

A thoroughly biased feature table

These tools have different scopes and purposes:

Name Pip + venv Pipenv Poetry pyenv pythonloc Conda this
Manages dependencies ✓ ✓ ✓ ✓ ✓
Resolves/locks deps ✓ ✓ ✓ ✓
Manages Python installations ✓ ✓ ✓
Py-environment-agnostic ✓ ✓ ✓
Included with Python ✓
Stores deps with project ✓ ✓ ✓
Requires changing session state ✓ ✓
Clean build/publish flow ✓ ✓
Supports old Python versions with virtualenv ✓ ✓ ✓ ✓ ✓
Isolated envs for scripts ✓
Runs project fns from CLI ✓ ✓ ✓

Use

Example contents:

[tool.pyflow]
py_version = "3.7"
name = "runcible"
version = "0.3.1"
authors = ["John Hackworth <jhackworth@vic.org>"]

[tool.pyflow.dependencies]
numpy = "^1.16.4"
diffeqpy = "1.1.0"

The [tool.pyflow] section is used for metadata. The only required item in it is py_version, unless building and distributing a package. The [tool.pyflow.dependencies] section contains all dependencies, and is an analog to requirements.txt. You can specify developer dependencies in the [tool.pyflow.dev-dependencies] section. These won't be packed or published, but will be installed locally. You can install these from the cli using the --dev flag. Eg: pyflow install black --dev

You can specify extra dependencies, which will only be installed when passing explicit flags to pyflow install, or when included in another project with the appropriate flag enabled. Ie packages requiring this one can enable with pip install -e etc.

[tool.pyflow.extras]
test = ["pytest", "nose"]
secure = ["crypto"]

If you'd like to an install a dependency with extras, use syntax like this:

[tool.pyflow.dependencies]
ipython = { version = "^7.7.0", extras = ["qtconsole"] }

To install from a local path instead of pypi, use syntax like this:

[tool.pyflow.dependencies]
# packagename = { path = "path-to-package"}
numpy = { path = "../numpy" }

To install from a git repo, use syntax like this:

[tool.pyflow.dependencies]
saturn = { git = "https://github.com/david-oconnor/saturn.git" }  # The trailing `.git` here is optional.

gitdependencies are currently experimental. If you run into problems with them, please submit an issue.

To install a package that includes a . in its name, enclose the name in quotes.

For details on how to specify dependencies in this Cargo.toml-inspired semver format, reference this guide.

We also attempt to parse metadata and dependencies from tool.poetry sections of pyproject.toml, so there's no need to modify the format if you're using that.

You can specify direct entry points to parts of your program using something like this in pyproject.toml:

[tool.pyflow.scripts]
name = "module:function"

Where you replace name, function, and module with the name to call your script with, the function you wish to run, and the module it's in respectively. This is similar to specifying scripts in setup.py for built packages. The key difference is that functions specified here can be run at any time, without having to build the package. Run with pyflow name to do this.

If you run pyflow package on on a package using this, the result will work like normal script entry points for someone using the package, regardless of if they're using this tool.

What you can do

Managing dependencies:

Running REPL and Python files in the environment:

Building and publishing:

Misc:

How installation and locking work

Running pyflow install syncs the project's installed dependencies with those specified in pyproject.toml. It generates pyflow.lock, which on subsequent runs, keeps dependencies each package a fixed version, as long as it continues to meet the constraints specified in pyproject.toml. Adding a package name via the CLI, eg pyflow install matplotlib simply adds that requirement before proceeding. pyflow.lock isn't meant to be edited directly.

Each dependency listed in pyproject.toml is checked for a compatible match in pyflow.lock If a constraint is met by something in the lock file, the version we'll sync will match that listed in the lock file. If not met, a new entry is added to the lock file, containing the highest version allowed by pyproject.toml. Once complete, packages are installed and removed in order to exactly meet those listed in the updated lock file.

This tool downloads and unpacks wheels from pypi, or builds wheels from source if none are available. It verifies the integrity of the downloaded file against that listed on pypi using SHA256, and the exact versions used are stored in a lock file.

When a dependency is removed from pyproject.toml, it, and its subdependencies not also required by other packages are removed from the __pypackages__ folder.

How dependencies are resolved

Compatible versions of dependencies are determined using info from the PyPi Warehouse (available versions, and hash info), and the pydeps database. We use pydeps, which is built specifically for this project, due to inconsistent dependency information stored on pypi. A dependency graph is built using this cached database. We attempt to use the newest compatible version of each package.

If all packages are either only specified once, or specified multiple times with the same newest-compatible version, we're done resolving, and ready to install and sync.

If a package is included more than once with different newest-compatible versions, but one of those newest-compatible is compatible with all requirements, we install that one. If not, we search all versions to find one that's compatible.

If still unable to find a version of a package that satisfies all requirements, we install multiple versions of it as-required, store them in separate directories, and modify their parents' imports as required.

Note that it may be possible to resolve dependencies in cases not listed above, instead of installing multiple versions. Ie we could try different combinations of top-level packages, check for resolutions, then vary children as-required down the hierarchy. We don't do this because it's slow, has no guarantee of success, and involves installing older versions of packages.

Not-yet-implemented

Building and uploading your project to PyPi

In order to build and publish your project, additional info is needed in pyproject.toml, that mimics what would be in setup.py. Example:

[tool.pyflow]
name = "everythingkiller"
py_version = "3.6"
version = "0.3.1"
authors = ["Fraa Erasmas <raz@edhar.math>"]
description = "Small, but packs a punch!"
homepage = "https://everything.math"
repository = "https://github.com/raz/everythingkiller"
license = "MIT"
keywords = ["nanotech", "weapons"]
classifiers = [
    "Topic :: System :: Hardware",
    "Topic :: Scientific/Engineering :: Human Machine Interfaces",
]
python_requires = ">=3.6"
# If not included, will default to `test.pypi.org`
package_url = "https://upload.pypi.org/legacy/"

[tool.pyflow.scripts]
# name = "module:function"
activate = "jeejah:activate"

[tool.pyflow.dependencies]
numpy = "^1.16.4"
manimlib = "0.3.1"
ipython = {version = "^7.7.0", extras=["qtconsole"]}

[tool.pyflow.dev-dependencies]
black = "^18.0"

package_url is used to determine which package repository to upload to. If omitted, Pypi test is used (https://test.pypi.org/legacy/).

Other items you can specify in [tool.pyflow]:

Building this from source

If you’d like to build from source, download and install Rust, clone the repo, and in the repo directory, run cargo build --release.

Ie on linux or Mac:

curl https://sh.rustup.rs -sSf | sh
git clone https://github.com/david-oconnor/pyflow.git
cd pyflow
cargo build --release

Updating

Uninstalling

Contributing

If you notice unexpected behavior or missing features, please post an issue, or submit a PR. If you see unexpected behavior, it's probably a bug! Post an issue listing the dependencies that did not install correctly.

Why not to use this

Dependency cache repo:

Python binary sources:

Repo binaries are downloaded from

Gotchas

References