Open springmeyer opened 6 years ago
Research if circle has an API similiar to travis's
trigger
API such that we can maintain the back compatibility and flow of that publishing process
Looks like it does: https://circleci.com/docs/2.0/api-job-trigger/
Now that http://circleci.com 2.x supports OS X it is likely a viable alternative to http://travis-ci.org for our CI/binary building needs. My hope is that circleci will be faster and more flexible (e.g. circle workflows).
Travis has served us well, however:
I therefore think we should try to move mason to circle. The steps needed to get there are, at least:
Fundamental first steps
circle
branch that adds a.circleci/config.yml
.circleci/config.yml
to test the mason core unit tests on both OS X and linux and ensure they work (could steal from https://github.com/mapbox/mapbox-tile-copy/pull/121 and https://github.com/mapbox/mason-js/pull/18/commits/cbf39d844e67b72ccaffb1025803049ceefc5966)trigger
API such that we can continue allowing developers (with commit to this repo) be able to publish packages by triggering a build locally.job
per platform, disabled by default in the mainconfig.yml
, then trigger a build for each based on some per-package config?ubuntu:precise
to ensure forward compatibility of all libs built against libc symbolsGochas include:
.travis.yml
child. The parent sets up the compiler environment and runs the unit tests under normal builds. The per-package child.travis.yml
controls what platforms we build a given package for (using travismatrix
jobs that override the parent). When we "trigger" builds on travis the per-package config is used and inherits the compiler environment from the main.travis.yml
parent in a custom way. Ideally we can do away with the per-package .travis.yml because these have become hard to maintain. I would be nice to come up with a single circleci config that would be used, somehow and a cleaner way to declare what platforms a package should be built for. How this would work and be designed is an open question and depends on what is possible and clean with circleci.source
'ing a bash script. However on circleci each line is an independent shell sosource
'ing does not change the environment for other commands. So we may want/need to change how the compiler toolchain is set up to make this more easy/robust. Circleci has a specialBASH_ENV
that can be worked with to persist things like CXX and other environment variables. But it would likely be wise to make this compiler toolchain setup not circleci-specific such that running the mason unit tests locally is still easy and consistent with how to test on circleci, if possible.Once proven, let's aim higher
Assuming the circleci port proves solid, we could piggyback on this (potentially) large build/publishing change to improve the overall developer experience of creating new packages for mason.
Note: The below ideas could likely be done using travis, but I think moving to circleci is the perfect opportunity to attempt them. In particular circleci workflows might be a great way to ensure the right steps happen during the packaging process, as efficiently as possible.
Critical context on this is that two significant limitations of mason currently are:
1) The PR publishing workflow is unintuative 🙅♂️
2) We don't test packages 😱
Below I'll detail both of these issues to give a starting sense of how they might be addressed using circleci + some elbow grease/creative thinking.
Improving the PR workflow
Currently, when creating a new mason package (either a totally new package or a new version of an existing package), mapbox devs start with a new branch/PR. However, the PR defaults to testing the mason core unit tests on travis. So, the package is not actually built. This is both unintuitive and dangerous: it's possible for a new developer to create a new package, see travis is green for their branch, and assume everything works fine when actually their package is broken in numerous ways. Most often they have forgotten to update the
gitsha
for the source download and they only learn this when they go to publish bytrigger
'ing a build. Oops. Or the package just plain won't compile yet because they do not know to test a build locally or did not have access to all operating systems to test locally first. To scale mason we'd like more developers being able to package themselves in mason, and feeling better supported by the system in doing so.So, ideally when a user creates a PR we could:
./mason build <package> <version>
works on all target operating systemsThis could be a different service eventually via a fancy webhook, but I can imagine this being accomplished in the near term by detecting changed files in the
./scripts
directory withgit diff --name-only master scripts
, parsing out the package names and versions, and forming up a command to run the build for all of them.Then, the PR would fail if the newly added or edited scripts don't build. This would provide the needed feedback to the dev making the package and publishing would happen after all builds work.
Testing packages before and after publishing
Currently we do not test packages. Yes, its true. Mason started with a philosophy of being minimal since we did not have a lot of bandwidth for writing or maintaining a complex and feature rich package manager. Since then mason has grown in usefulness and uptake amongst mapbox projects.
Currently package builds+publish either succeed when a build is trigger'd or they don't. We have no automated way of knowing - before they a package is published - that something might be wrong.
If we started having PRs actually test the package build then we could use circleci workflows to then test that the packages work after they are built. We could even reject new packages unless they had a certain threshold of tests. Mason core/each package would have some starting framework for tests that is extensible so we could add additional tests over time. Examples that could be supported are:
cp -r
bug (which behaves slightly differently on OS X vs Linux). Instead of learning the headers are missing when using the package downstream, ideally we could know they are not in the right location before publishing./cc @mapbox/core-tech @kkaefer @jfirebaugh