Action Will Not Create Summary Tables if Performance Alerts Occur

[morgangibbs87]

We are trying to have the Summary table always appear - even after an alert-threshold is exceeded.

We use this action to perform benchmarks on several os and device types in a matrix. This splits our benchmarks into separate jobs for each ${{ matrix.os }}-${{ matrix.device }}.

The Summary does not appear for devices where the alert-threshold is exceeded, even when fail-on-alert: false is set and summary-always: true is enabled.

This is true whether comment-on-alert is set or not, and even when we set a very high fail-threshold to ensure it is not using the alert-threshold value by default and make sure that the job does not fail.

After testing several different ways, it looks like the action is not set up to create a summary table at all if a performance alert occurs.

- name: Analyze benchmark results
    id: analyze-bm
    uses: risc0/github-action-benchmark@v1.19.3
    with:
      name: "${{ matrix.os }}-${{ matrix.device }}"
      tool: 'customBiggerIsBetter'
      output-file-path: target/hotbench/fib/benchmark.json
      external-data-json-path: ./cache/external.json
      github-token: ${{ secrets.GITHUB_TOKEN }}
      alert-threshold: '120%'
      fail-threshold: '10000%'
      comment-on-alert: true
      fail-on-alert: false
      summary-always: true