I've been experiencing some pain points with our monorepo and its infrastructure tooling. While this is all fresh on my mind, I wanted to create an issue detailing approaches taken and their pros/cons. The hope is to create a space for Earthly users to discuss monorepo pain points, solutions and what potential solutions might exist.
This excellent post from @gf3 outlines what a realistic monorepo set-up may look like. This key point was mentioned:
> Hi @ahsf - see also this simple example: https://github.com/earthly/earthly/tree/main/examples/monorepo
unfortunately this example is a little too simplistic and doesn't accurately represent the structure and needs of most monorepos. most of the monorepos i've worked with have shared these three characteristics:
1. multiple applications or binaries
2. multiple shared libraries used by the above applications
3. shared dependencies across both the applications and libraries
Approaches
There are a few possible approaches one can take with a monorepo. I'll describe and outline the pros/cons of each approach. For the examples, I'll be using an Elixir/Node monorepo with the following structure:
# Earthfile
all:
WAIT
BUILD +test
END
WAIT
IF [ $DEPLOY_ENABLED == "true" ]
BUILD +deploy
END
END
deploy:
# These deploys should only happen if we actually made changes to their respective targets
BUILD ./apps/app1+deploy
BUILD ./apps/app2+deploy
BUILD ./apps/app3+deploy
test:
BUILD ./apps/app1+test
BUILD ./apps/app2+test
BUILD ./apps/app3+test
# apps/app1/Earthfile
test:
BUILD +test-self
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app2/Earthfile
test:
BUILD +test-self
BUILD ../../packages/package1+test
BUILD ../../packages/package2+test
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app3/Earthfile
test:
BUILD +test-self
BUILD ../../packages/package1+test
BUILD ../../packages/package3+test
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# .github/workflows/ci.yml
name: CI
on:
- push
jobs:
all:
name: +all
env:
DEPLOY_ENABLED: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: Run test and deploy
run: |
earthly \
--push -P
+all --DEPLOY_ENABLED=$DEPLOY_ENABLED
Analysis
This set-up allows for Earthly to construct a DAG and theoretically do the minimal work required.
It's also amazing that within each Earthly build target, you can install and run whatever tooling your heart desires (more on this later).
Open questions:
How do we only deploy the affected targets?
Currently, Earthly has no way to only run the deploy for app3 if it detects changes only in app1 or its child packages.
Right now this would all be running on a single node. For our monorepo, local machines grind to a halt when both Node and Elixir builds are running at the same time. This does not scale well as more projects are added.
Perhaps we should annotate each BUILD command with which satelite should be used. Although perhaps it'd be better if there is one giant shared cache with just compute being spread around.
How do we make this more ergonomic?
As build args are added for targets, everything must be passed along. The --pass-args flag proposal could definitely go a long way here.
Is Earthly's caching good enough?
This FAQ of Bazel vs Earthly makes the point that Earthly does not do file level compilation caching. The new CACHE command fixes these issues in my experience. In Elixir, you can cache the _build directory which has compiled Elixir files.
Can we integrate with existing package manger tooling better?
Both app2 and app3 would declare in their package.json that they depend on their javascript packages. Tooling such as NX automatically detect those dependencies.
Perhaps we add a project.json file that Earthly can utilize to detect and declaritively configure child dependencies. NX does an excellent job at this configuration from my experience. How would this work with Earthly's imperative nature?
How about local development?
Earthly is great but it has overhead. Can we have a fast and efficient watch mode? This issue covers this. It's important to note that even tools such as NX and Turborepo do a poor job at supporting this.
How do we handle important CI tasks such as commenting on a pull request with deploy status?
Imagine we have a target such as +pull-request-deploy. We want to update the pull request with the produced deployment URL and Docker image tag.
Since we're using Earthly, we could theoretically just create our own bash script that calls the Github API to post a comment. I do think there are definitely some improved ergonomics that could be found here in making it more declarative. Earthly and Docker excel at "inheritance" but aren't as great with "composition." This is a topic for another issue perhaps.
Approach 2: App/Service Level
Repo Structure
# Earthfile
# various targets
# apps/app1/Earthfile
test:
BUILD +test-self
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app2/Earthfile
test:
BUILD +test-self
BUILD ../../packages/package1+test
BUILD ../../packages/package2+test
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app3/Earthfile
test:
BUILD +test-self
BUILD ../../packages/package1+test
BUILD ../../packages/package3+test
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# .github/workflows/app2-ci.yml
name: CI
on:
- push
jobs:
deploy:
name: +deploy
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: deploy
run: |
earthly \
--push -P
./apps/app1+deploy
test:
name: +test
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: test
run: |
earthly \
--push -P
./apps/app1+test
# .github/workflows/app3-ci.yml
# The same as app2 but for app2 instead
# .github/workflows/app1-ci.yml
# The same as app2 but for app1 instead
Analysis
This changes our CI to split it at the app level. We no longer call a top level Earthfile target and instead do it for each service.
Open questions:
How can we avoid repeatedly testing the dependent packages?
If our app1 and app2 Github Actions workflows are running in parallel, then the dependent packages (pkg1, pkg3) will be repeatedly tested. Satelite caching may help with this but it also may be a race condition. This approach wastes satelite compute minutes (assuming they use the same satelite -- which as discussed above may not be a good idea).
Several of the questions mentioned in approach 1 also apply here. Most notably, how do we only run app deploy targets if they're actually affected by code changes.
Analysis
This changes approach 2 to instead let Github Actions handle the DAG. This allows for multiple services to avoid repeating work if they both use the same package.
Why are we splitting up our DAG when Earthly should be able to handle it all?
I've found working with CI providers to be a rough experience. I think part of that frustration is recreating a DAG that our build tools (such as Earthly) already know about. Instead of letting CI declare the compute split, what if we integrated that better within Earthly?
Approach 3: App/Service/Package Level
This is very similar to approach 2 but instead we don't have apps/services test their child dependencies. They're ran separately.
Repo Structure
# Earthfile
# various targets
# apps/app1/Earthfile
test:
BUILD +test-self
test-self:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app2/Earthfile
test:
FROM earthly/dind:alpine
... # Full stack integration tests
# apps/app3/Earthfile
test:
FROM earthly/dind:alpine
... # Full stack integration tests
# packages/pkg1/Earthfile
test:
FROM +node
RUN yarn run test
# .github/workflows/app2-ci.yml
name: CI
on:
- push
jobs:
deploy:
name: +deploy
# Github Actions don't let you actually easily do this. You have to split it up and it's a horrible DX. See [here](https://stackoverflow.com/questions/58457140/dependencies-between-workflows-on-github-actions/64733705#64733705) for how this works.
depends-on: pkg1-ci
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: deploy
run: |
earthly \
--push -P
./apps/app1+deploy
test:
name: +test
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: test
run: |
earthly \
--push -P
./apps/app1+test
# .github/workflows/pkg1-ci.yml
# The same as app1 but for pkg1 instead
Analysis
This changes approach 2 to instead let Github Actions handle the DAG. This allows for multiple services to avoid repeating work if they both use the same package.
Why are we splitting up our DAG when Earthly should be able to handle it all?
I've found working with CI providers to be a rough experience. I think part of that frustration is recreating a DAG that our build tools (such as Earthly) already know about. Instead of letting CI declare the compute split, what if we integrated that better within Earthly?
Approach 4: Just use a monorepo build tool (NX, Turborepo, Bazel, etc.)
This is very similar to approach #1 but instead leverages a build tool (in this example, NX).
Repo Structure
# Earthfile
deploy:
COPY . .
RUN --push nx --affected --target=deploy
test:
COPY . .
RUN --push nx --affected --target=test
# .github/workflows/ci.yml
name: CI
on:
- push
jobs:
deploy:
name: +deploy
if: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: Run test and deploy
run: |
earthly \
--push -P
+deploy
test:
name: +test
if: ${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
runs-on: ubuntu-latest
steps:
# set-up steps here
- name: Run test and test
run: |
earthly \
--push -P
+test
Analysis
This seems great because existing tools can be leveraged to handle the monorepo side of things.
Open questions:
How do we avoid one giant Docker image?
Our top level targets would require us to install all of the tooling we need for them to run. This means a giant image with both Elixir, Node and any other language we use. It also means one image with all of the binaries used during build/test time.
It doesn't allow us to leverage Earthly to split-up our Docker instructions of what build tools are required for each app/package. Perhaps UDC commands have a place here but it's far from an ideal solution.
How do we spread the compute?
Some of the build tools support distributed task execution but then we lose the benefits of Earthly. We have to use the provider's tools for declaring necessary dependencies.
How can we improve the copy operation?
There are definitely some performance considerations when copying all of the files in the repo. Also, the DX is not great but this issue on nested .earthlyignore files could improve some pain points.
Summary
I think Earthly is in a unique position to take over more of the stack. These are real issues that I've ran into when working on my team's multi-language monorepo. We are a small team and only have two major languages in use.
I've been experiencing some pain points with our monorepo and its infrastructure tooling. While this is all fresh on my mind, I wanted to create an issue detailing approaches taken and their pros/cons. The hope is to create a space for Earthly users to discuss monorepo pain points, solutions and what potential solutions might exist.
This excellent post from @gf3 outlines what a realistic monorepo set-up may look like. This key point was mentioned:
Approaches
There are a few possible approaches one can take with a monorepo. I'll describe and outline the pros/cons of each approach. For the examples, I'll be using an Elixir/Node monorepo with the following structure:
Approach 1: Earthly top-level
Repo Structure
Analysis
This set-up allows for Earthly to construct a DAG and theoretically do the minimal work required.
It's also amazing that within each Earthly build target, you can install and run whatever tooling your heart desires (more on this later).
Open questions:
How do we only deploy the affected targets?
Currently, Earthly has no way to only run the deploy for app3 if it detects changes only in app1 or its child packages.
NX has the
--affected
flag.How do we spread the compute load?
Right now this would all be running on a single node. For our monorepo, local machines grind to a halt when both Node and Elixir builds are running at the same time. This does not scale well as more projects are added.
Perhaps we should annotate each
BUILD
command with which satelite should be used. Although perhaps it'd be better if there is one giant shared cache with just compute being spread around.How do we make this more ergonomic?
As build args are added for targets, everything must be passed along. The
--pass-args
flag proposal could definitely go a long way here.Is Earthly's caching good enough?
This FAQ of Bazel vs Earthly makes the point that Earthly does not do file level compilation caching. The new
CACHE
command fixes these issues in my experience. In Elixir, you can cache the_build
directory which has compiled Elixir files.Can we integrate with existing package manger tooling better?
Both app2 and app3 would declare in their package.json that they depend on their javascript packages. Tooling such as NX automatically detect those dependencies.
Perhaps we add a
project.json
file that Earthly can utilize to detect and declaritively configure child dependencies. NX does an excellent job at this configuration from my experience. How would this work with Earthly's imperative nature?How about local development?
Earthly is great but it has overhead. Can we have a fast and efficient watch mode? This issue covers this. It's important to note that even tools such as NX and Turborepo do a poor job at supporting this.
How do we handle important CI tasks such as commenting on a pull request with deploy status?
Imagine we have a target such as
+pull-request-deploy
. We want to update the pull request with the produced deployment URL and Docker image tag.Since we're using Earthly, we could theoretically just create our own bash script that calls the Github API to post a comment. I do think there are definitely some improved ergonomics that could be found here in making it more declarative. Earthly and Docker excel at "inheritance" but aren't as great with "composition." This is a topic for another issue perhaps.
Approach 2: App/Service Level
Repo Structure
Analysis
This changes our CI to split it at the app level. We no longer call a top level Earthfile target and instead do it for each service.
Open questions:
How can we avoid repeatedly testing the dependent packages?
If our app1 and app2 Github Actions workflows are running in parallel, then the dependent packages (pkg1, pkg3) will be repeatedly tested. Satelite caching may help with this but it also may be a race condition. This approach wastes satelite compute minutes (assuming they use the same satelite -- which as discussed above may not be a good idea).
Several of the questions mentioned in approach 1 also apply here. Most notably, how do we only run app deploy targets if they're actually affected by code changes.
Analysis
This changes approach 2 to instead let Github Actions handle the DAG. This allows for multiple services to avoid repeating work if they both use the same package.
Why are we splitting up our DAG when Earthly should be able to handle it all?
I've found working with CI providers to be a rough experience. I think part of that frustration is recreating a DAG that our build tools (such as Earthly) already know about. Instead of letting CI declare the compute split, what if we integrated that better within Earthly?
Approach 3: App/Service/Package Level
This is very similar to approach 2 but instead we don't have apps/services test their child dependencies. They're ran separately.
Repo Structure
Analysis
This changes approach 2 to instead let Github Actions handle the DAG. This allows for multiple services to avoid repeating work if they both use the same package.
Why are we splitting up our DAG when Earthly should be able to handle it all?
I've found working with CI providers to be a rough experience. I think part of that frustration is recreating a DAG that our build tools (such as Earthly) already know about. Instead of letting CI declare the compute split, what if we integrated that better within Earthly?
Approach 4: Just use a monorepo build tool (NX, Turborepo, Bazel, etc.)
This is very similar to approach #1 but instead leverages a build tool (in this example, NX).
Repo Structure
Analysis
This seems great because existing tools can be leveraged to handle the monorepo side of things.
Open questions:
How do we avoid one giant Docker image?
Our top level targets would require us to install all of the tooling we need for them to run. This means a giant image with both Elixir, Node and any other language we use. It also means one image with all of the binaries used during build/test time.
It doesn't allow us to leverage Earthly to split-up our Docker instructions of what build tools are required for each app/package. Perhaps UDC commands have a place here but it's far from an ideal solution.
How do we spread the compute?
Some of the build tools support distributed task execution but then we lose the benefits of Earthly. We have to use the provider's tools for declaring necessary dependencies.
How can we improve the copy operation?
There are definitely some performance considerations when copying all of the files in the repo. Also, the DX is not great but this issue on nested .earthlyignore files could improve some pain points.
Summary
I think Earthly is in a unique position to take over more of the stack. These are real issues that I've ran into when working on my team's multi-language monorepo. We are a small team and only have two major languages in use.