Migration to Cicero 1.0 & Outage - 2020-03-103

[dselman]

Discussion 🗣

On 2020-03-03 we had an outage of models.accordproject.org. This is a report of what happened and lessons learned.

Detailed Description

We merged https://github.com/accordproject/models/pull/121 to master, which updates the build script to build and validate the models files using Concerto v1, as well as using Concerto v0.82.x.

Initially there was a short outage because the updated run.js build script now requires Node >= 12.x and while this was used locally, Netlify was using an older Node.js runtime. This was quickly resolved by adding a .nvmrc file to the root of the repository, indicating to Netlify which version of Node.js to use. The engines statement in package.json was also updated.

The more serious issue was that while run.js was updated to either build using Concerto v0.82 or Concerto v1, it cannot build using both, meaning that models that are incompatible with the Concerto version were skipped during the build.

The impact was that once the library had been built and published using Concerto v1, the cicero/runtime.cto file was skipped in the build - which is required to archive existing Cicero templates.

Note that unlike the cicero-template-library repo within this repo we do not check-in and version the results of running the build. They are created by the build, and then the results are served as static files by Netlify.

 Lessons

1. Deploy model changes for v1 to a v1 branch, not to master
2. Investigate whether it is possible to test the results of the running the build, prior to publication by Netlify. For example, how many namespaces do we expect, or how many model files?

Questions

1. Model repository changes can break existing templates - however it is not clear how we can run those tests prior to publication of the models.
2. How do we want to host v0.82 & v1.0 compatible models? If we want to use the same site, then we will need to commit the results of running the build, as we do for templates
3. How do we want to indicate the graph of v1 compatible models? We could create new namespaces, version the files, or create a new domain?

[jeromesimeon]

A preliminary set of answers:

1. this one is hardest, I think proper versioning but also persistent publication like we do for the template library with an index of some kind
2. I think more importantly: prevent overriding of existing models, like we do for the template library
3. Versioning scheme for building for multiple concerto models, along with ability to identify which model is compatible with which version is in 1.0.0-alpha.5 and #128