Completely revamp build system

[april]

• Kill 3 build systems, 2000 lines of stuff • Add much cleaner local development • Convert to GitHub actions • Update dependencies • Update to Node 14, which should fix various build issues we've had.

[andrewkrug]

As @gene1wood mentioned (this is slightly more nuanced): The original complexity factor of the build systems was driven by the separation of concerns problem.

Some history:

When the MOC existed we were asked to ensure that we could roll back between reproducible builds of the lock. Which spawned the has system of assets that live in the SSO Dashboard CDN.

Auth0 had a number of extensions and tools to help with deploys but none of them really worked very well so we at first rolled our own. This required a highly privileged management API credential. Since the lock could be used by a threat actor to capture user passwords and second factors it created a need for a separate build system. Which is why we chose to use AWS Codebuild. Developers needed real time feedback on their PRs so we employed Travis-CI to give that without giving direct access to Codebuild.

Since anyone who could access Codebuild where the Auth0 tenant credentials were stored could do all kinds of nasty things to the tenant it seemed appropriate.

Developers also needed access to build and deploy from a local machine. Hence the build scripts and such.

Simplicity is great but in this case it does come at a price. All the credentials in one basket. Github's auditing around actions is still untested. Re assembly of a custody chain for 100% of credentials stored there should be addressed prior to the swap along with runbooks and playbooks for a github actions related incident.

Other team's confidence in using Mozilla SSO has to come from a place where they trust all the systems involved in delivery of the custom lock in order to maintain 100% adoption for projects like TaskCluster and Mercurial.

[andrewkrug]

This also removes the docker makefile wrapper to help folks use Docker for local dev. Are you moving away from using Docker?

[april]

Yes, definitely moving away from Docker. It's yet another thing to maintain and update and work with, and it's pretty pointless here given that package.json and package-lock.json are explicitly designed to make it unnecessary.

[april]

I should add that the Auth0 account used here has only the following scopes:

• read:clients • update:clients • read:client_keys • update:client_keys • update:tenant_settings

I don't think there's any obvious way to get all the things you are enumerating — it can't change rules or roles or read users or view log streams — but Auth0 doesn't do a great job of listing what these scopes entail.

[andrewkrug]

Yes, definitely moving away from Docker.

The rest of Mozilla-IAM uses Docker for testing and deployment. I might reconsider. We ran into several situations where builds were non-deterministic.

The scope update:tenant_settings means you can modify the lock itself. Which is would rank as a maximum risk. update:clients also opens the door for allowing additional login methods for auth0 clients.

[april]

Basically everywhere we've encountered Docker in the build systems we have encountered problems. This is simply building a simple webpage with Node.js, basically nobody uses Docker for this purpose. And it adds more systems and more places where there could be security problems.

[andrewkrug]

Basically everywhere we've encountered Docker in the build systems we have encountered problems.

This is likely because the Docker containers were built on Arch not because they were Docker based workflows.

[april]

And I should add that every website and application I have created at Mozilla (which is many of them) has simply used Node and Gulp/Webpack. Having to pull down a Docker image to build something like this is incredibly overkill.

[andrewkrug]

It adds more systems and more places where there could be security problems. Docker affords oodles of advantages when it comes to security and reproducible builds. Building in an OS X environment or in whatever the CI/CD system's flavor of the week is opens up a broad attack surface as well.

[april]

It's using Ubuntu LTS, please tell me what these issues are.

Also I would add that you don't have to maintain this giant mess of a build system anymore, I do. It's been broken for months now because it's so rickity and complicated that everyone who interacts with it is afraid to touch it.

[andrewkrug]

Of course, you are totally free to do whatever you like as the module owner. I was only hoping to share a bit of the history and risks associated with solving the deployment problem. The feedback is all there take it or leave it. Bear in mind that everything gets a little rickity and complicated with age. With that, I now return to building rickity and complicated things.

[april]

This is designed to age gracefully:

• everything is in one place, so there's no need to update things inside AWS and Docker
• it uses well-supported tools to build and deploy (node, npm, gulp, node-sass, aws-cli, a0deploy)
• it uses mainstream, frequently-updated systems to build (ubuntu-latest, and current Node versions)

The only part of the system that should need to be updated is the very occasional bump of Node version and updates to the dependencies within. In my experience, that takes maybe an hour or two of work every six months or so, and which would have been required anyways.

The current build system relies on Node 7, which has been EOL since June 2017. The fact that it has been using a version of Node that has been completely unsupported for security updates for three years now speaks to me that it isn't actually that easy to update and maintain.

[fiji-flo]

I do appreciate reducing complexity in here. I do acknowledge that all of this happened in some historical context. However, I vote for embracing change and not be attached to code (I try to act like that for myself at least).

GitHub actions where not an Option back then, Travis was still Travis and what should we do. In hindsight everyone knows better. Reproducible builds are awesome, but most of the time not a reality and come at a cost. Everything is a compromise.

Let's assess the security implications, but a long as code hosted in this repo is executed on a system that has access to deploy keys, I don't see the big security implications of moving credentials to GitHub.

[gene1wood]

April and I chatted on this topic and the security angle that @andrewkrug brings up is a good point. Our plan is to

• Continue reviewing and likely merge this change
• Enable auto deployment into the Auth0 dev environment by storing the dev credentials in the GitHub Actions Secrets storage
• Not to put Auth0 prod credentials in GitHub Actions Secrets storage for the time being and instead perform production deploys locally, manually for the time being. We currently deploy to prod infrequently and we'll take on the manual overhead for the time being while we explore the security angle further.

@andrewkrug Thank you so much for both sharing the history of how and why the build system got to where it is and pointing out the security concern around storing tenant credentials in GitHub Actions Secret storage.

@april I'll try to review this PR later this week.

[april]

Note that I updated YADR and inadvertently screwed up commit signing. I've since fixed that, but we may need to temporarily turn off signing to merge this. Sorry about that.