Scream

Still very much a WIP. Lots of work to be done; please open issues or pull requests.

What you do when you see a bunch of python... packages.

or...

An opinionated CLI tool for Python monorepo MGMT.

It's objective is to ease the creation, testing, and deploying of multiple python packages in a single repository. To ensure non-overlapping names with PYPI, this tool forces you to use namespace packages. Namespaces are defined according to python's pkgutil-style.

• Scream
  • Why use Scream?
    • Problems with most monorepos
    • How scream works
  • Commands
  • Quickstart
  • Using your monorepo packages
  • Configuration
  • Upcoming Features

Why use Scream?

• Scream holds all packages to the same standards

  • Enforces consistent styling.
  • Enforces consistent testing.
  • Enforces consistent documentation.

• Uses tox to setup virtualenvs for isolated testing across python versions.

• Pre-commit hooks to help prevent those gosh darn mistakes.

Problems with most monorepos

1. Testing & CI/CD pipeline can become slow with many packages.

   The other available solutions are:

   • Use a build tool like Pants or Bazel.

   These tools are extremely powerful, but sometimes are overkill, and introduce a fair amount of overhead to manage.

2. Installing intra-monorepo package dependencies is hard with a private repositories.

   The available solutions are:

   • Have all your packages distributed publically.
   • Host a private PYPI index.
   • pip install using dependency links.

How scream works

Scream aims to cause as little overhead as possible for managing your monorepo. No fancy third party configuration or private PYPI repositories.

Scream uses the existing python packaging requirements to resolve intra-monorepo dependencies, and git to detect what's changed since the parent branch.

For example:

cat setup.cfg.

[metadata]
name = company_packageA
version = 0.0.1
description = Your package description!

[options]
packages = find:
install_requires =
    company_packageB

If you make a change to company_packageA then run tests...

> scream test --dry-run

The following packages have changes compared since branch: `master`:
        company_packagea

Packages that require testing:
        company_packagea
        company_packageb

Commands

• scream init - Run this first. Initiates a monorepo in an empty directory.

• scream new <package-name> - Creates new template package.

• scream test [--dry-run][--all][--parallel] - Tests packages and package dependents that have changed.

• scream install <package-name>[--test] - Installs a package. --test installs testing dependencies.

• deploy <package-name> - Runs deploy.py in your package directory.

• scream build - Builds a python wheel and bundles it with all it's dependencies as wheels. (TODO)

• -v will enable verbose logs.

Quickstart

By default packages are tested against python 3.7.x, which means you have it available on your PATH. If you about different versions, please see the configuration options.

> mkdir mymonorepo
> cd mymonorepo

> scream init
Done!
Create a new package with `scream new <namespace>.<package_name>`

> scream new com.packagea
Created project `com.packagea`

> scream new com.packageb
Created project `com.packageb`

> scream test --all --parallel
Running tests... (Using all cores)

> coverage report

Using your monorepo packages

The two common ways you would install packages from this monorepo are:

1. Clone your repository and run scream build or scream install in your CI tool to build packages and ship them to your machines.
2. Standard pip install individual packages to any machine from private or public github repos.

If your repository is public, you can simply install a subpackage anywhere using:

pip install 'git+ssh://git@github.com/ORG/REPO.git@master#subdirectory=examplea'

If your repository is private, you need a few extra steps to make sure packages that depend on other packages in this monorepo can be installed.

1. Specify dependency_links in the setup.cfg for each 'local' dependency:

   dependency_links =
   git+ssh://git@github.com/ORG/REPO.git@master#egg=examplea-0#subdirectory=subpackages/examplea

2. pip install as before, but specifiy the flag --process-dependency-links

   pip install 'git+ssh://git@github.com/ORG/REPO.git@master#subdirectory=examplea' --process-dependency-links

Configuration

1. In your packages setup.cfg the variable python_requires determines which versions of python your package will be tested against.
   • Ex. python_requires = 2.7, 3.6, 3.7

     Note: the python versions you intend to test must be available on your path.

Upcoming Features

• [ ] Auto version handling.
• [ ] Pre commit hook to flake8.
• [ ] Pre commit hook to avoid committing a large file.