Automation

[ojeytonwilliams]

This was originally posted by @jmerle in Gitter, I'm re-posting it here so it does not get buried:

Now that I've spent some serious time on thinking about implementation of the QA and deployment automation ideas, I don't think it's actually a good thing to implement now. The current situation of reviewing and deploying a PR is (assuming no review comments for simplicity):

1. Maintainer looks through diff to spot obvious mistakes
2. Maintainer checks out PR locally to scrape the docs and check if it's okay
3. Maintainer deploys the docs to S3 and MaxCDN
4. Maintainer merges the PR
5. Automation workflow deploys changes from PR to Heroku
6. Changes have been deployed

With automation it would be:

1. Maintainer looks through diff to spot obvious mistakes
2. Maintainer marks the issue as containing documentation updates for selected docs
3. Without preview deployments: Maintainer checks out PR locally to scrape the docs and check if it's okay
4. With preview deployments: Maintainer marks the PR as ready to be previewed
5. With preview deployments: Automation workflow runs scrapers for selected docs and deploys preview server
6. With preview deployments: Maintainer inspects the preview build logs and the preview output
7. Maintainer marks the PR as ready for merge
8. Without preview deployments: Automation workflow runs scrapers for selected docs and pushes generated packages to staging storage
9. With preview deployments: Automation workflow pushes previously generated packages to staging storage
10. Automation workflow merges PR
11. With preview deployments: Automation workflow updates staging preview
12. Repeat step 1-11
13. Maintainer merges master into production branch
14. Automation workflow downloads generated packages from staging storage and deploys them to S3 and MaxCDN
15. Automation workflow deploys changes from staging to Heroku
16. Changes have been deployed

There are two issues I see with this now, the first one being the most important:

• The non-automated workflow is pretty much always faster than the automated one.
  • Without preview deployments it definitely is, as running docs:upload is way faster than having the scraping happen again in a CI build and then later uploading this scraped package to S3 and MaxCDN (with potential anomalies because the automated workflow would upload a scraped package which hasn't been checked by a human).
  • With preview deployments you'd have to manually trigger and wait for the re-scrape and preview to be deployed after every commit to see the changes, which would be at most as fast as and probably slower than just checking the changes locally.
• The staging environment delays updates and requires someone to regularly deploy staging to production (this can be automated too, but at that point why even have a staging environment). If a maintainer has reviewed a PR and marked it as "good-to-go" there is not really a reason to not deploy it (except for freeCodeCamp/devdocs#728, but that has better solutions).

Whilst I agree that this automation enables some best practices and it goes against much of what I have said about this so far, I'm reluctant to implement it now as maintainer productivity is more important to me than following these best practices with our current backlog in mind. I would therefore suggest keeping the current situation for now, with only making the switch from Travis to GitHub Actions for now as the latter is way faster in my tests.

[jmerle]

Here's my notes from before I thought "hey, is this even a good idea?" (I didn't code a lot yet, I decided to write it out first to know what to do):

Workflows:

• build
  • When: on every push and pull request
  • Parallel: yes
  • What: runs all tests
• update-check
  • When: once a month
  • Parallel: no
  • What: checks which docs are outdated and creates a new issue containing the gathered information
• mark-doc
  • When: on /markdoc comment by maintainer on a pr
  • Parallel: yes
  • Arguments: docs to update in the format "DOC@version", like "lua@5.4" or "react_native" (the version defaults to the latest version).
  • What: adds doc-DOC@version labels for each argument given to comment, and creates the labels if they do not exist yet. This is preferred over creating the labels manually because by doing it automatically they all have a consistent color and description. If the label already exists it is fine to add it manually via the GitHub interface.
• preview
  • When: on /preview comment by maintainer on a PR and on a push to a PR by a maintainer
  • Parallel: no
  • What: runs the scraper for all labeled docs and launches a preview deployment displaying just those docs. If no docs are labeled, a preview deployment is started with the default docs. Should add a comment to the PR the first time a preview for a PR is deployed.
• merge
  • When: on /merge comment by maintainer on a PR
  • Parallel: no
  • What: runs the scraper for all labeled docs, packages the generated docs and pushes them to the staging-storage branch. It also updates the modified-docs.txt file on the staging-storage branch to include the labeled docs. If everything is succesful, the PR is automatically merged and a comment is placed on the PR letting the user know that the updates are now in staging and will be pushed to production soon.
• production
  • When: on every merge from master to production
  • Parallel: no
  • What: download all added/updated docs from the staging-storage branch and push them to S3 and MaxCDN. Then run a Heroku deployment as usual and push a new version of the Docker image to Docker Hub.

To do:

• Check if there is a limit on the amount of GitHub labels
  • Prune outdated labels every so often?
• Find a good way to do preview deployments
  • Vercel
    • Only sponsors static apps
  • Heroku
    • Costly?
  • Docker
• Come up with a strategy to limit the amount of preview builds
  • Preview builds are kinda expensive, repetitive pushes by a maintainer shouldn't cause too much builds
  • Only launch a preview builds on a /preview comment or when a certain time since the last commit has passed?

Also, this is why I think GitHub Actions is significantly faster than Travis CI (Travis CI runs these builds in ~2 minutes): https://github.com/jmerle/devdocs-ci-testing/actions

And here's the workflow: https://github.com/jmerle/devdocs-ci-testing/blob/master/.github/workflows/build.yml

[simon04]

Your reasoning with respect to the automation makes perfect sense. The current workflow isn't bad at all and gives opportunity for quick solutions (such as re-running the scraper three times in a row to get around spurious HTTP errors). A few scrapers rely on manually downloaded docs (such as python), a few scrapers rely on manually compiled docs (such as ruby) – automating those steps seems almost impossible (given reasonable time constraints).