Deploy Docs for OpenLattice

[tfogo]

Create docs for the OSS version of Lattice

[tfogo]

Just getting started on this.

[tfogo]

So we have 4 components which could have independently updated versions:

• API
• CLI
• SDL
• Guide (i.e. general concepts, tutorials, explanations, etc)

This leads to some unique challenges in structuring the docs. Honestly there are aspects of this which won't become apparent until after release. So we need a best-effort solution to this with the understanding we may need to make improvements in the future.

It sounds like the most important version is the SDL version. The API is for more advanced users. Even if the API changes, the most important thing to people will be how to write the definitions that the API uses. I propose the following layout:

• The guide version is in lock step with the SDL version. The main version selector on the docs will be for the SDL+Guide version.
• The API will be a different web page with its own independent versions. This is the same way Kubernetes does it (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/)
• The CLI versions will be under the main docs site but just have a page for each version.
• Old SDL+Guide versions do not link to newer API/CLI versions. Again, the same as Kubernetes.

Here's a proposed solution to achieve this:

• Note, mkdocs doesn't have a versioning mechanism built in. There are some janky third party plugins that would still require some coding on our end. I propose a more straightforward approach that leverages our CI pipeline.
• An mkdocs static site is built for each branch and pushed to S3. Master is the most current release. Note this means assets won't be cached between branches. But this is what K8s does and I think the performance tradeoff for people looking at old documentation is not a concern.
• A release branch has its branch name added to a JSON file in S3. (This can be done manually or we could automate it based on branch name format.)
• The docs site pulls the S3 file and creates a version drop down. It decides what version it is by inspecting its own URL or mkdocs metadata. This allows old versions of the docs to link to newer versions. Something Kubernetes does not do.
• API docs should be built and deployed separately. They are linked out to from the main docs.
• CLI docs can be embedded in the main docs. Each version is pulled and built for each release.

It would also be nice for PRs to be built. Kubernetes uses Netlify to automate building and deploying in the manner described above. Although it might be easier on Netlify, it may be nice to keep everything in concourse.

I'm working on implementing some of this locally today, then will link up with Ben to see about getting it into concourse.

Undecided questions:

• Under what domain name is this hosted?
• What does it mean when we say "Lattice Version". Will it become confusing if the main versioning of the docs is not the same as "Lattice Versions"?
• Should we do a subdomain for versions or make it part of the path?

[tfogo]

I have implemented a POC of the docs here:

http://v2.lattice.docs.s3-website-us-east-1.amazonaws.com/

The dropdown at the top also brings you to the v1 version of the docs:

http://v1.lattice.docs.s3-website-us-east-1.amazonaws.com/

This dropdown is generated dynamically by mkdocs and is fairly straightforward.

The API and CLI docs are listed by their version. The API docs go to their own site, the CLI docs are included in the main documentation.

[tfogo]

Reminder:

Username: mlab Password: Ws3obK3Mk}

[tfogo]

After the meeting yesterday, it's clear we need to move forward with the OSS Docs. So need to write Latticectl command docs, (probably) OpenAPI spec for Lattice API, and start on the tutorial.

Need to talk to Will to make sure the current versioning scheme looks good.

[tfogo]

We decided on versioning around the Lattice API. Now have to work on automating the generation.

[tfogo]

I was working on other things so no progress with the automation yet.

[tfogo]

List of build actions based on triggers:

• Push to docs repo:

  • Pull corresponding lattice branch
    • build and deploy API docs for that branch
  • Pull all lattice branches that have compatible_cli metadata
    • build and deploy CLI docs
  • Build and deploy docs

• Push to lattice repo

  • Build and deploy API docs
  • Need to build CLI docs and deploy into every docs website that uses that CLI version
    • Check each docs bucket on S3 for compatible_cli metadata.
    • If it matches, deploy into that bucket.

Questions: Are we versioning CLI separately from the API?

If yes, we should separate out latticectl into its own repo so it can be versioned independently using git branches.

[tfogo]

Since we don't have the API docs currently, right now I just want to be able to deploy a branch of the docs and deploy the corresponding latticectl docs from the branch of the same name.

After talking to Ben it sounds like there are issues deploying to different buckets for each version. Need to talk to McDave to look into how to get this working in one bucket.

[tfogo]

After talking to Dave, it sounds like it's possible to deploy different versions of the site into a single bucket. Working with Ben this week to add my build scripts/containers to concourse.

[tfogo]

There were some difficulties getting this working in concourse. Figured it out and now have the master branch and branches of the form "v.X" of the docs building and pushing to S3. Need to spend a few hours cleaning up and then it should be finished.

[tfogo]

Unfortunately I bumped into some issues with having the docs build task trigger when the latticectl docs are updated. There were a couple of tricky things around this. Came up with a solution and now just need to add the CI script to the lattice repo and hook everything up.

[tfogo]

There is an issue uploading the latticectl docs to S3. The Concourse resource that we're using to upload the docs to S3 is trying to upload the build image for the task when it shouldn't. This causes the job to fail, even though the latticectl docs are correctly uplaoded.

I asked Ben for help with this and he suspects there's a bug in the Concourse resource or one of its dependencies. Ben said he'd look at it further. If it is a bug, I have a workaround in mind. If we confirm it's a bug, I think we should just implement the workaround and call it a day for the time being.

[tfogo]

Ben helped me fix the problem I was having. The latticectl docs are now building and uploading correctly - but the versioning isn't quite right yet. Should be a quick fix and we'll be ready to go.

[tfogo]

Bumped into a problem with versioning things on Concourse. This is a known issue with concourse that they are working on. You can read more about the limitations here: https://medium.com/concourse-ci/sneak-peek-spatial-resources-d0eed9bb3fa

Had a talk with Ben and Will this morning. We have decided to use a meta pipeline and a templating language such as jsonnet to generate pipeline files. We will probably want to have a target per version. This side steps the issue.

This is also what I was recommended on the Concourse forums: https://discuss.concourse-ci.org/t/patterns-for-versioning-software-in-concourse/395

So for the time being I'm going to backtrack to the docs releasing a single version. Then when we build this pipeline templating system we'll work in the multiple versions.

[tfogo]

Single version of docs being deployed here: http://lattice-oss-docs-test.s3-website-us-east-1.amazonaws.com/docs-html/v0.1/

Need to clean up some CI stuff then will move on to actually writing docs.

Would love to point this to lattice.org at some point.

Would also love to get the generated Lattice API docs integrated in when it's ready.

[tfogo]

This is complete and cleaned up. Handing off to @BenElgar to work on templating and versioning.

Next steps:

• Working on DNS records for lattice.org
• Need to ask Dave to fix the icons not loading
• Need to add in Abdul's lattice API docs

[tfogo]

Note cloudfront docs are now here: https://d2u0tvivqxdlsb.cloudfront.net/v0.1/

[BenElgar]

There seems to be a DNS record set up already:

[2018-08-24 03:43:23] madmin@walker [0] % m list-dns-records -d lattice.org
    INFO | 2018-08-24 03:46:25,140 | commands | MainProcess | MainThread | list-dns-records 5b7fe201cf27fc0da580f886 (pid 3493)
      API request: GET http://api.dnsmadeeasy.com/V2.0/dns/managed
      API response: status=200, reason=OK
      API request: GET http://api.dnsmadeeasy.com/V2.0/dns/managed/6018527/records
      API response: status=200, reason=OK

DNS records for lattice.org:
NAME                VALUE             TYPE       TTL
=================================================================================================
www                       d2u0tvivqxdlsb.cloudfront.net.                CNAME   60
-------------------------------------------------------------------------------------------------

[tfogo]

Angela is in the process of setting this up. We're creating a lattice prod aws account to start putting the production stuff in.