FEATURE: Auto-merge Crowdin translations

[tiblu]

Possible solutions:

• Manual merge by translation manager
• Automated merge by using tooling - for example https://github.com/features/actions

[tiblu]

@oksks Do you want the merged translations to go to testing environment only or you want full control to release to production also?

[oksks]

Do you want the merged translations to go to testing environment only or you want full control to release to production also?

Ideally, I wouldn't need to push anything anywhere and translations would go live as soon as they are done, let's say, every hour or 30 minutes. But if I have to push manually, then to production of course. Otherwise there's an extra middleman without adding any real value, and translations still won't go live until you guys push them. ))

[tiblu]

@oksks @loorm Moving back to prioritization. It's about 1-2 days of fiddling to get the flows set up and tested and @oksks on board thus I will not squeeze it between other stuff.

[loorm]

Triage #14 time estimate - 2 days.

[tiblu]

Related to https://github.com/citizenos/citizenos-fe/issues/336

[tiblu]

Notes from GitHub Actions docs

• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/configuring-a-workflow#filtering-for-specific-branches-tags-and-paths
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#viewing-logs-to-diagnose-failures
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/managing-a-workflow-run#enabling-debug-logging
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/configuring-a-workflow#adding-a-workflow-status-badge-to-your-repository
• https://help.github.com/en/github/automating-your-workflow-with-github-actions/hosting-your-own-runners
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/using-environment-variables
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-and-using-encrypted-secrets
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/authenticating-with-the-github_token
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/creating-a-javascript-action
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/core-concepts-for-github-actions
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet
• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/events-that-trigger-workflows

Notes from GitHub API docs

• https://developer.github.com/v3/git/refs/

API tests

• Clone a branch
  • curl -i -H 'Authorization: token *****' -d '{"ref": "refs/heads/pr_duplicator_test_clone4", "sha": "63ea7c9db4cc63f166ed8da6c27e10fbf9c65add"}' https://api.github.com/repos/citizenos/playground/git/refs
  • Where ref is the ref to the branch to be created and sha is the sha of the commit from which the branch is created - the last commit sha of the branch to be cloned.
• Get branch information
  • curl -i -H 'Authorization: token *****' https://api.github.com/repos/citizenos/playground/branches/pr_duplicator_test_clone3
    • In the result commit.sha is what we need to use for the cloning.

GitHub Statuses

The status API allows external services to mark commits with an error, failure, pending, or success state, which is then reflected in pull requests involving those commits.

• https://developer.github.com/v3/repos/statuses/

[tiblu]

GitHub Action to auto-merge Crowdin translations

Pre-conditions for auto-mergeing PR:

• PR by "tiblu"
• PR to "master" (testing env)
• PR from "l10n_master" (Crowdin branch)

Steps:

• [x] A person manually merges Crowdins PR from "l10n_master" to "master" - GitHub UI
• [x] After merge to "master" validate all files being valid JSON, HTML or whatever the format is - Travis CI. In Citizen OS FE runs npm test.
• [x] Push to "master", which executes Heroku action to autodeploy to test - Heroku. Heroku has autodeploy configured from the branch, it has "Wait for CI to pass before deploy" so that it waits for Travis CI to buld successfully.
• [x] Notify #CI about deployment to test - GitHub Action - https://github.com/citizenos/citizenos-fe/blob/master/.github/workflows/deployment_status_notifications_to_slack.yaml
• [x] PR to "prod" with the same change from "l10n_master" by making a copy of the branch of the state that was merged to "master". - GitHub Action
• [x] Notify Slack channel #ci about freshly created PR - GitHub Action
• [x] After merge to "prod" validate all files being valid JSON, HTML or whatever the format is - Travis CI. In Citizen OS FE runs npm test.
• [x] Push to "prod", which executes Heroku action to autodeploy to production - Heroku. Heroku has autodeploy configured from the branch, it has "Wait for CI to pass before deploy" so that it waits for Travis CI to buld successfully.
• [x] Notify #ci chat about deployment to production GitHub Action - https://github.com/citizenos/citizenos-fe/blob/master/.github/workflows/deployment_status_notifications_to_slack.yaml

Challenges:

• We don't know when Crowdin finishes the sync. Crowdin creates a PR and then starts adding commits to it, which makes it hard to understand when the sync has finished and it's time to start the merge process.
  • Crowdin working to change that - https://portal.productboard.com/featurerequest/1-submit-feature-requests-ideas-for-crowdin/c/53-enhance-vcs-integrations
  • We will consider Crowdin finished when the PR is merged in GH.

[tiblu]

Work done so far...

Citizen OS FE

• @oksks Has the right to accept PR-s to master
• Added a Mocha test to https://github.com/citizenos/citizenos-fe that validates language files - https://github.com/citizenos/citizenos-fe/blob/master/test/l10n.js
• Added citizenos-fe to Travis CI - https://travis-ci.org/citizenos/citizenos-fe
• Travis CI runs builds for Citizen OS FE for every change in master and prod.
• Travis CI will send notifications about build statuses to #ci channel in Slack.
• Heroku will ONLY deploy if Citizen OS FE build passes.
• Heroku will notify Slacks #ci channel about deployments. Impl - https://github.com/citizenos/citizenos-fe/blob/master/.github/workflows/deployment_status_notifications_to_slack.yaml

[tiblu]

IMPORTANT: Before merging PR, please make sure it is to the desired environment:

• Test env PR-s are "into maser"

  • 

• Production env PR-s are "into prod"

  • 

How to release translations

1. Crowdin will create a new pull request (PR) to relevant repository when there are new translations. For example FE open pull requests are here: https://github.com/citizenos/citizenos-fe/pulls
2. If there is a new PR, Travis will run and test the language file integrity, it will also notify #ci about the status.
3. After verifying that changes are ONLY to language files merge the PR by Crowdin (GH user tiblu) to master branch which is the testing environment.
4. After a delay of 5-10 minutes, Heroku will deploy the application to testing environment and notify #ci chat about the deployment. The notification will contain a link to the application to verify the changes.
5. After receiving deployment notification with a link in #ci channel, verify the changes in the testing environment. Link to the environment is provided in the message.
6. Meanwhile PR Duplicator action has created a duplicate PR "AUTO: PR-Duplicator - New Crowdin translations. " to prod branch, which is to deploy to production.
7. IF changes have been verified in testing environment, you are safe to merge the PR "AUTO: PR-Duplicator - New Crowdin translations. " to prod branch created by the PR Duplicator.
8. After the merge, #ci channel will be notified about build status and deployment to production. The status text will contain a link to the application, but that is not correct as Heroku does not give out the right url for our domain configuration.
9. Verify the changes in the production environment

NOTE: It's easiest to follow the #ci channel and click through the process using the links provided in the messages. NOTE: Initially implemented only on citizenos-fe for testing. Once the flow seems ok, we'll implement it to other projects.

Understanding the #ci feed

[tiblu]

Done. Citizen OS FE as all configured, before we configure Citizen OS API I would like to get some feedback on the flow.

[oksks]

@tiblu I've tried to follow the instructions in order to push translations live - got stuck on the second step already. :(

2) If there is a new PR, Travis will run and test the language file integrity, it will also notify #ci about the status. ON: I opened https://github.com/citizenos/citizenos-fe/pulls, there was one PR from 3 days ago. Even though I've added some translations earlier this morning.

3) After verifying that changes are ONLY to language files merge the PR by Crowdin to master branch which is the testing environment.

• How do I verify that? I currently see "New translations en.json" on every line - if changes would've been to non-language files, what would I see there?
• The PR wasn't by Crowdin, it was by Tiblu.
• I got this message after merging: "Pull request successfully merged and closed. You’re all set—the l10n_master branch can be safely deleted." - am I expected to delete it or to disregard the message?

4) Heroku will deploy the application to testing environment and notify #ci chat about the deployment. 5) After receiving notification in #ci channel, verify the changes in the testing environment. Link to the environment is provided in the message. Not sure which message you mean. There is a lot going on in #ci chat with loads of links :( Which link am I supposed to click? The only link to the test environment was posted way before I merged the PR, so it doesn't seem related to my actions. EDIT: ah ok, #ci chat keeps posting stuff and links, and tiblu has deleted l10n_master branch by itself. Several minutes later a link to the test env was posted, too. I feel really dumb.

6) Meanwhile PR Duplicator action has created a duplicate PR to prod branch, which is to deploy to production. 7) IF changes have been verified in testing environment, you are safe to merge the PR to prod branch created by the PR Duplicator. What? Where? What do I need to do now that I've verified (some) changes in the test env? This one here? "AUTO: PR-Duplicator - "New Crowdin translations". #382".. Why some lines have green checkmarks by them, some yellow circles and some red crosses?? WHAT DOES IT MEEEEEAAAN?? Anyway, I'll assume all is well (as I followed the instructions closely), and will merge that. fingerscrossed

8) After the merge, #ci channel will be notified about build status and deployment to production. 9) Verify the changes in the production environment I survived! Changes are in.

[tiblu]

@oksks

2) If there is a new PR, Travis will run and test the language file integrity, it will also notify #ci about the status. ON: I opened https://github.com/citizenos/citizenos-fe/pulls, there was one PR from 3 days ago. Even though I've added some translations earlier this morning.

Crowdin creates a new PR and starts adding updates to the existing PR until it is merged. The 3 days shows creation date of the PR, not last update date. To see if Crowdin has added your changes, open up the PR and see the list of commits and their times. Also you can open up changed files to see the changes themselves.

3) After verifying that changes are ONLY to language files merge the PR by Crowdin to master branch which is the testing environment. How do I verify that? I currently see "New translations en.json" on every line - if changes would've been to non-language files, what would I see there?

That is correct - make sure only language files are changed. In FE that means *.json in langauges.

The PR wasn't by Crowdin, it was by Tiblu.

Fixed the comment. Yes, the PR-s are by GH user tiblu until Crowdin gets it's own user.

I got this message after merging: "Pull request successfully merged and closed. You’re all set—the l10n_master branch can be safely deleted." - am I expected to delete it or to disregard the message?

Don't delete the branch, just leave it. It enables us double check if we run into any issues. From time-to-time we will delete old branches manually.

4) Heroku will deploy the application to testing environment and notify #ci chat about the deployment. 5) After receiving notification in #ci channel, verify the changes in the testing environment. Link to the environment is provided in the message. Not sure which message you mean. There is a lot going on in #ci chat with loads of links :( Which link am I supposed to click? The only link to the test environment was posted way before I merged the PR, so it doesn't seem related to my actions. EDIT: ah ok, #ci chat keeps posting stuff and links, and tiblu has deleted l10n_master branch by itself. Several minutes later a link to the test env was posted, too. I feel really dumb.

When deploying to Heroku, there is a delay of 5 mins until the first notification is sent. Unfortunately I did not figure out better messaging at the time. It's not about you being dumb, it's about us not having decent messaging.

6) Meanwhile PR Duplicator action has created a duplicate PR to prod branch, which is to deploy to production. 7) IF changes have been verified in testing environment, you are safe to merge the PR to prod branch created by the PR Duplicator. What? Where? What do I need to do now that I've verified (some) changes in the test env? This one here? "AUTO: PR-Duplicator - "New Crowdin translations". #382".. Why some lines have green checkmarks by them, some yellow circles and some red crosses?? WHAT DOES IT MEEEEEAAAN?? Anyway, I'll assume all is well (as I followed the instructions closely), and will merge that. fingerscrossed

The created PR after successful merge to testing is named "AUTO: PR-Duplicator - "New Crowdin translations".". Don't be concerned about the check marks, greens and yellow. IF the merge button is green, you are free to merge it. The checks, that determine the status (green, yellow) of the merge run in background, so if the button is not green, see if any of the checks are still running or if some have failed. If they still run, you just have to wait. If some failed, they will not be fixed without developer interference - call 800-tiblu-ilmar.

[KatiVellak]

Legally reviewed, no impact.