Monorepo with multiple builds steps/Docker images

[seniorquico]

We have a large, existing monorepo that contains our frontend code (JavaScript transpiled and published via Node.js and npm) and our backend code (C# via Mono 5).

I'm jazzed about the approach presented by the Shippable team for detecting specific changes in a monorepo:

http://blog.shippable.com/build-test-and-deploy-applications-independently-from-a-monorepo

However, I'm not sure how to configure Shippable to build our frontend and backend as separate pipeline steps using different Shippable Docker images. It looks like some others were looking for a way to achieve this, but nobody provided a response (#3024). Is it possible to use multiple shippable.yml files in a monorepo? If not, is our only option to create a monolithic Docker image that contains every build tool needed for all components?

Thanks!

• Kyle

[ttrahan]

Hi @seniorquico: We actually just published a new blog on using Shippable with a mono repo that addresses this scenario. It contains separate Dockerfiles for each component and only builds and pushes the images that have code changes.

It demonstrates the following scenario:

Check it out here: http://blog.shippable.com/ci/cd-of-microservices-using-mono-repos

In regards to your question, it is not possible to use multiple shippable.yml files in a single repo. However, you can define multiple Shippable workflow jobs within a single repo, which is an option that allows for pretty much unlimited configuration options for how to set up your workflow, as seen in the blog example.

[seniorquico]

@ttrahan Ah, I missed that blog post.

OK... going the multiple Shippable workflow jobs route... would the approach be along the lines of ditching the ciRepo resource and runCI job in exchange for a gitRepo resource (say with "buildOnPullRequest: true") and multiple runSh jobs? But... can we use custom Docker images with runSh? Looks like this is discussed in #3717, but that's not available yet(?). Can this be accomplished without #3717?

Thanks!

• Kyle

[ttrahan]

@seniorquico: You could certainly go that route to use runSh jobs instead of runCI jobs for this, but you're correct that this won't work for you if you're using custom images for your CI jobs. This is in development for runSh, but not launched yet.

In the interim, you could try a few different approaches to get you close to what you need:

• Set up a runSh job that will evaluate which components had code changes and will generate an output that stores a variable or set of variables in a Shippable params resource that you'll pass into a runCI job. Use these params values to set the custom image used in the pre_ci_boot section (you can use env variables in that yml block) and which CI job to run. You can even do this with matrix builds and spin up running CI and build/push for each component in separate runs for your job.

For example:

  pre_ci_boot:
    image_name: mydockerrepo/${image_to_use_for_ci}
    image_tag: imagetag
    pull: true
    options:

• Identify a base image common to all components and install additional dependencies, as needed, depending on which component you want to build. You could pass in an Input parameter, generated from an initial runSh job that identifies the component to build, to the runCI job and use that variable to determine which component to run CI on (i.e. which dependencies to install, which tests to run, etc.).

[seniorquico]

@ttrahan: The approach described in the first bullet sounds awesome!

To configure this scenario, would we simply switch off "Process Webhooks" on the project's settings and then configure a job as described? Or would it be more appropriate to leave "Process Webhooks" on and switch off all of "Commits", "Pull Requests", "Tags", and "Releases"? It's unclear if "Process Webhooks" has other consequences... I noticed it causes the project to show as "Paused".

Finally, I'm assuming we'd still need to setup the gitRepo resource. Or is it possible to feed an existing ciRepo resource into a runSh job?

Thanks!

• Kyle

[ttrahan]

@seniorquico: Yes, you can use either method to stop triggering the CI job from the source control webhook that's fired when a commit is made. Other inputs to the runCI job (like the params resource you'll set up) can still trigger the CI job when the webhooks are paused.

And yes, you will need to create a gitRepo resource as an input to the runSh job. Currently, gitRepo and ciRepo can't be shared, though this will consolidate in the future.

[seniorquico]

Thanks, @ttrahan!

[ttrahan]

@seniorquico: I apologize, but just tested and it looks like turning off "Process Webhooks" will pause the runCI job for all inputs, so leave that on and instead turn off the switches for "Commits", "Pull Requests", "Tags", and "Releases" in Project Settings.

[seniorquico]

@ttrahan Got it! Thanks for the help! :smile:

[ttrahan]

@seniorquico: One last clarification. When passing in the variables as inputs to the runCI job, these will be injected as global variables, not matrix variables. So, perhaps a better method, would be to trigger individual CI jobs for each component that has changed by calling the Shippable API from the runSh job. The process would be:

1. runSh job generates list of changed folders (i.e. components)
2. runSh job triggers a new build job via the Shippable API for each component in the list
3. A CI job will run for each component

[sheats]

@ttrahan this is good and almost what I was looking for... What if my microservices are in multiple languages -- seems like the language definition in the root shippable.yml only allows one language?

[sheats]

You can disregard, I found https://github.com/Shippable/support/issues/1222

[seniorquico]

@ttrahan I'm finally building this solution, and I have a follow-up question to your final comment from Oct 18. Is it somehow possible to get back to the original approach you described? Namely:

You can even do this with matrix builds and spin up running CI and build/push for each component in separate runs for your job.

It sounds like input resources to a runCI job and global environment variables injected by a Shippable API call don't work. Is it possible to dynamically configure a matrix build? Or are matrix builds simply limited to the static configuration in shippable.yml?

I'm thinking of the nicety of having the various component builds roll up into a single entry in our build history. If we dispatch multiple build jobs using the Shippable API, won't we have to manually correlate multiple entries in our build history?

Thanks!

• Kyle

[seniorquico]

This looks related to #2714.

[manishas]

@ambarish2012 can you help with this?

[manishas]

@seniorquico sorry for the lack of updates on this issue... was there a resolution? If not, I want to make sure this is on our radar..

[seniorquico]

@manishas We have a workable monorepo build pipeline, but it's less than ideal. I would still consider this an outstanding request and would love to see a feature request come out of this discussion.

Here are the monorepo pipeline options our team discussed (both on and off GitHub threads):

• Use a runSh job to detect changes and trigger a single runCI job with matrix variables using the API. A matrix variable is used to represent the component and select the required Docker image.

  • Our team believes this would be the best solution for our monorepo pipeline. We prefer the runCI UI over the runSh UI for our builds. It keeps the build logs clean (a matrix job for each component, all associated with a single build number).
  • This is not possible with the current API.

• Use a single runCI job with matrix variables. A matrix variable is used to represent the component and select the required Docker image. Each job runs a shell script to detect changes. If the current component was unchanged, the job immediately finishes.

  • This is what we're currently using.
  • This results in every job running, even if the associated component didn't change. This slows down the build pipeline, and often results in many, many unnecessary jobs.

• Use a single runCI job with multiple TASKs and different Docker images.

  • I intended to open a GitHub issue to discuss this as a potential feature request. The thought was along the lines of bringing some of the recent, awesome changes to runSh to runCI. If this were a possibility, I could see this being a better option to the first option described above.
  • runSh: Running multiple tasks
  • runSh: Using a custom docker image

• Use a runSh job to detect changes and trigger multiple runCI jobs with global variables using the API. A global variable is used to represent the component and select the required Docker image.

  • This created separate builds for each job. It became a small burden to correlate the jobs.
  • This also caused intermittent GitHub PR issues where parallel jobs that both failed and succeeded sometimes marked a PR as green for merge.

• Ditch the runCI job and launch one or more runSh jobs.

  • We didn't really explore this option in depth... The runSh job UI in the Shippable dashboard isn't as slick for CI builds. It also wasn't clear if we'd need to manually handle some of the GitHub PR integration work.

• Create a monolithic Docker image with all of the required build tools. Run a single runCI job.

  • Downside: This is high maintenance, slow, and, in our specific situation, not even feasible.

[danielrs]

I think this is an important feature, which will benefit both current users and new users. The only other solution that I know has native support for mono-repos is CodeFresh:

https://codefresh.io/continuous-integration/using-codefresh-with-mono-repos/

This feature will make Shippable a much more attractive solution for new users looking for modern CI/CD tools.