This is an example of scaffolding and tooling for a Python based monorepo, with the goal of demonstrating:
This example offers tooling that supports a local development paradigm where changes to shared libraries are immediately available when running & testing individual services. This also supports CI/CD flows where changes to any shared library should cause dependent services to rebuild & integration test those changes immediately. Poetry helps makes this possible. Poetry also allows venvs to be local to the service you're working on, making debugging dependencies a breeze.
The most specific challenge this tooling addresses, is how to properly build all the dependencies into a Docker container. Poetry does not really help here. The Makefiles provide a very lightweight and powerful approach to this. This example provides a shell script that does roughly the same thing - but it rebuilds everything every time. GNU Make gives us some smarts that will help avoid great pains for larger libraries or projects. The shell script offers a tool that is a bit more friendly for CI/CD environments where your build is in a clean environment and will require rebuilding everything anyways.
This example demonstrates calculating the dependencies of a service, building wheels for everything, and using wheels to perform a standard pip installation in a container.
Some advantages to this strategy:
Disadvantages include:
You'll need Poetry, GNU Make and Docker.
cd
your way into a service directorypoetry install
poetry run python -m [service_folder]
make
make clean
make _clean
poetry install
using make reset
cd
into a shared library and make some changes; notice they're available immediately in the service when you poetry run ...
make clean && make
to rebuild the container with updated shared libsCheckout the other helpers, like tools/cloudbuild.yaml
, tools/poetry.dockerfile
and the Makefiles
. Criticism, discussion, issues and PRs are welcome.
Join the discussion about the use of shared libraries in microservices and share your war stories of doing it wrong - or right - or whatever.