[rush] Automatically format errors for popular CI products

elliot-nelson commented 3 years ago

Summary

It would be nice if the end-of-build errors rush collects were automatically formatted for display in popular CI products. In particular, both Azure DevOps and GitHub Actions have special log prefixes that can be applied to add extra color or context to error messages in the log output.

(Other CI tools out there may have similar functionality and could also be added as time permits.)

Details

Formatting

For Azure, here's a link to the logging command formats. The green ##[section] might be nice for individual heft builds inside the build. By far the feature I'd want most, though, is for errors to show up on Build Summary (without opening the log), which you can accomplish with:

# This message shows up in red in the log, and also appears on the Build Summary page.
echo "##vso[task.logissue type=error]A helpful error message."

# Note that this is not the same - it is colored red, but does NOT appear on Build Summary.
echo "##[error]Another helpful error message."

Similarly, GitHub Actions has docs for their logging commands. To accomplish the same:

# This message appears in red in the log and in shows up on the run summary
echo "::error::A helpful error message"

There are still some formatting challenges here -- I have yet to get either Azure or GitHub to format a multi-line error properly, for example. (Azure will accept a multi-line error string if you replace \r\n with %0D%0A, and that gets printed as a multiline string into the HTML on the Build Summary page, but since there's no translation of \n to line breaks it just shows up as a long unformatted string.)

You'd need to decide, if some task dumps 20 lines of text to STDERR, is the "error" you want to report to build summary the first line of the text? The name of the command that produced it? Report every line as an error? (Probably a very bad idea for generic STDERR output.)

Detection

To allow configuration, it might be nice to have an env var (e.g. RUSH_CI_FORMAT) that controls how and whether this extra CI-specific formatting takes place.

Suggested values:

Value	Behavior
`RUSH_CI_FORMAT=ado`	Add Azure DevOps log formatting
`RUSH_CI_FORMAT=github`	Add GitHub Actions log formatting
`RUSH_CI_FORMAT=none`	Disable any log formatting (use normal terminal coloring etc.)
`RUSH_CI_FORMAT=auto`	Automatically detect and apply appropriate format from above (default)

If in auto mode, you can use existing env vars to determine what environment you are running in. (Typically you'd use the presence of TF_BUILD=True to detect Azure and GITHUB_ACTIONS=true to detect GitHub Actions.)

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
`@microsoft/rush` globally installed version?	5.41.0
`rushVersion` from rush.json?	5.40.7
`useWorkspaces` from rush.json?	true
Operating system?	Mac
Would you consider contributing a PR?	Yes
Node.js version (`node -v`)?	12.17.0

iclanton commented 3 years ago

I like this idea a lot. This is probably something that the individual projects' build tool should support too, so this should probably apply to Heft as well.

Would you be interested in putting together the implementation?

prescience-data commented 3 years ago

This is a great idea! 🕺 Also some templates for these CI tools would be great too (to make it easier for new teams to pick it up and implement auto build and publish in CI using the builtin Rush scripts):

Example:

- name: Check consistent dependencies
  run: node common/scripts/install-run-rush.js check
- name: Check if changes have been provided
  run: node common/scripts/install-run-rush.js change -v
- name: Build everything
  run: |
    node common/scripts/install-run-rush.js install --no-link
    node common/scripts/install-run-rush.js link
    node common/scripts/install-run-rush.js prisma:generate
    node common/scripts/install-run-rush.js sort
    node common/scripts/install-run-rush.js prettier
    node common/scripts/install-run-rush.js lint
    node common/scripts/install-run-rush.js rebuild --verbose
    node common/scripts/install-run-rush.js test --verbose
  env:
    NODE_ENV: test
- name: Publish packages
  run: |
    git config user.email "github-action@users.noreply.github.com"
    git config user.name "GitHub Action"
    node common/scripts/install-run-rush.js version --bump --target-branch main
    node common/scripts/install-run-rush.js publish -a -b main -p --include-all
  env:
    NPM_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

(maybe even a cool "generator" tool on the Rushstack website?)

elliot-nelson commented 3 years ago

Haven't forgotten about this and may come back to it, although for Azure at least I wonder if this is an "Azure problem" and not a "rush problem".

The default "bash exited with error code 1" message in Azure is useless no matter what your pipeline does -- rush build, CDK deploy, terraform apply, "npm run test", etc.

Lately I've been thinking what I really need is a version of the classic Jenkins plugin Text Finder packaged as an azure extension, so that I can look for appropriate errors/warnings in my pipeline regardless of what it is building.

dmichon-msft commented 3 years ago

It occurs to me that it might be worthwhile to develop a custom Azure Pipelines task for running rush commands on a repo, using install-run-rush.js or whatever alternative approach internally. Then that task can better own its failure mode.

microsoft / rushstack