No commit information is found in payload when Storing Benchmark Result

[darrenge]

This could be more of a newbie / user issue than actual issue. When I try to store benchmark result, I get an error: Error: No commit information is found in payload:

What I am doing is below. 1) Run the Benchmark for our project (Garnet) from a GitHub action yml from a powershell script: dotnet run -c $configuration -f $framework --filter $filter --project $BDNbenchmarkPath --exporters json > $resultsFile 2> $BDNbenchmarkErrorFile

Which comes out as: dotnet run -c Release -f net8.0 --filter RespParseStress --project D:\a\garnet\garnet/benchmark/BDN.benchmark --exporters json > D:\a\garnet\garnet/test/BDNPerfTests/results/CI_BDN_Config_RespParseStress_Windows.results 2> D:\a\garnet\garnet/test/BDNPerfTests/errorlog/CI_BDN_Config_RespParseStress_StandardError_Windows.log

2) Github YML to store it using the file from the BenchmarkDotNet.Artifacts/results:

Run github-action-benchmark action

  - name: Store benchmark result
    uses: benchmark-action/github-action-benchmark@v1
    with:
      tool: 'benchmarkdotnet'
      output-file-path: ./test/BDNPerfTests/BenchmarkDotNet.Artifacts/results/BDN.benchmark.Resp.RespParseStress-report-full-compressed.json

      # Where the previous data file is stored
      external-data-json-path: ./cache/benchmark-data.json
      # Workflow will fail when an alert happens
      fail-on-alert: true

It is this step (#2) in the GitHub Action yml file that has the error. The output for this step is: Run benchmark-action/github-action-benchmark@v1 2 with: 3 tool: benchmarkdotnet 4 output-file-path: ./test/BDNPerfTests/BenchmarkDotNet.Artifacts/results/BDN.benchmark.Resp.RespParseStress-report-full-compressed.json 5 external-data-json-path: ./cache/benchmark-data.json 6 fail-on-alert: true 7 name: Benchmark 8 gh-pages-branch: gh-pages 9 benchmark-data-dir-path: dev/bench 10 auto-push: false 11 skip-fetch-gh-pages: false 12 comment-always: false 13 summary-always: false 14 save-data-file: true 15 comment-on-alert: false 16 alert-threshold: 200% 17 env: 18 DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 19 DOTNET_NOLOGO: true 20 DOTNET_ROOT: C:\Program Files\dotnet 21Error: No commit information is found in payload: { 22 "inputs": null, 23 "ref": "refs/heads/main", 24 "repository": { 25 "allow_forking": true, 26 "archive_url": "https://api.github.com/repos/darrenge/garnet/{archive_format}{/ref}", 27 "archived": false, 28 "assignees_url": "https://api.github.com/repos/darrenge/garnet/assignees{/user}", 29 "blobs_url": "https://api.github.com/repos/darrenge/garnet/git/blobs{/sha}", 30 "branches_url": "https://api.github.com/repos/darrenge/garnet/branches{/branch}", 31 "clone_url": "https://github.com/darrenge/garnet.git", 32 "collaborators_url": "https://api.github.com/repos/darrenge/garnet/collaborators{/collaborator}", 33 "comments_url": "https://api.github.com/repos/darrenge/garnet/comments{/number}", 34 "commits_url": "https://api.github.com/repos/darrenge/garnet/commits{/sha}", 35 "compare_url": "https://api.github.com/repos/darrenge/garnet/compare/{base}...{head}", 36 "contents_url": "https://api.github.com/repos/darrenge/garnet/contents/{+path}", 37 "contributors_url": "https://api.github.com/repos/darrenge/garnet/contributors", 38 "created_at": "2024-03-25T16:31:31Z", 39 "default_branch": "main", etc etc

[darrenge]

The project we want to do this for is at https://github.com/microsoft/Garnet and all the benchmark is at https://github.com/microsoft/garnet/tree/main/benchmark/BDN.benchmark. All open source so you can see how we are doing benchmarking etc. Maybe I need to change something there to make this stuff work. Thanks.

[ktrz]

Hi @darrenge

How are you running this action? Is it running on a PR or are you running it manually? If it's being run manually then the action will need to fetch the commit information via the GH API and for that it needs a GH token provided. You can use the automatically generated token

github-token: ${{ secrets.GITHUB_TOKEN }}

You can find example setup here

[darrenge]

I am just running it manually now to try to get it to work. Yep - that is the example I am using. I figured out I needed the TOKEN.:

name: Store benchmark result
        uses: benchmark-action/github-action-benchmark@v1
        with:
          # What benchmark tool created output file
          tool: 'benchmarkdotnet'

          # Where the previous data file is stored
          # #*#*#* DEBUG - can't set if auto-push is set external-data-json-path: ./cache/benchmark-data.json
          # Workflow will fail when an alert happens
          fail-on-alert: true

          github-token: ${{ secrets.GITHUB_TOKEN }}
          auto-push: true
          skip-fetch-gh-pages: true
          summary-always: true

      I created a branch from my fork (https://github.com/darrenge/garnet) called gh-pages and pushed it.  Now I am getting this error:

"C:\Program Files\Git\bin\git.exe" -c user.name=github-action-benchmark -c user.email=github@users.noreply.github.com -c http.https://github.com/.extraheader= switch gh-pages 22fatal: invalid reference: gh-pages 23Error: Command 'git' failed with args '-c user.name=github-action-benchmark -c user.email=github@users.noreply.github.com -c http.https://github.com/.extraheader= switch gh-pages': fatal: invalid reference: gh-pages 24: Error: The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 128

[darrenge]

I would love more insight on what I am doing wrong as I am at this step. Admittedly, I am confused on what my URL would be for this part: https://you.github.io/repo/dev/bench like this.

Steps I am following now: The step which runs github-action-benchmark does followings:

Extract benchmark result from the output in output.txt Switch branch to gh-pages Read existing benchmark results from dev/bench/data.js Update dev/bench/data.js with the extracted benchmark result Generate a commit to store the update in gh-pages branch Push gh-pages branch to remote Compare the results with previous results and make an alert if possible performance regression is detected After the first workflow run, you will get the first result on https://you.github.io/repo/dev/bench like this.

Also - this issue has went off original topic, so if you feel best to close it, that is ok too. I understand.

[darrenge]

Here is the output for my GH Action when I run the Store Benchmark. Note I have a branch named "gh-pages" Run benchmark-action/github-action-benchmark@v1 2 with: 3 tool: benchmarkdotnet 4 output-file-path: ./test/BDNPerfTests/BenchmarkDotNet.Artifacts/results/BDN.benchmark.Resp.RespParseStress-report-full-compressed.json 5 fail-on-alert: true 6 github-token: *** 7 auto-push: true 8 skip-fetch-gh-pages: true 9 summary-always: true 10 name: Benchmark 11 gh-pages-branch: gh-pages 12 benchmark-data-dir-path: dev/bench 13 comment-always: false 14 save-data-file: true 15 comment-on-alert: false 16 alert-threshold: 200% 17 env: 18 DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 19 DOTNET_NOLOGO: true 20 DOTNET_ROOT: C:\Program Files\dotnet 21"C:\Program Files\Git\bin\git.exe" -c user.name=github-action-benchmark -c user.email=github@users.noreply.github.com -c http.https://github.com/.extraheader= switch gh-pages 22fatal: invalid reference: gh-pages 23Error: Command 'git' failed with args '-c user.name=github-action-benchmark -c user.email=github@users.noreply.github.com -c http.https://github.com/.extraheader= switch gh-pages': fatal: invalid reference: gh-pages 24: Error: The process 'C:\Program Files\Git\bin\git.exe' failed with exit code 128

[darrenge]

Closing this out as I was able to go through the How To (https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file#charts-on-github-pages-1) from minimal all way up through Charts on GitHub. This is no longer an issue

[ktrz]

Hey @darrenge Thank you for closing the issue! Is there something in the README that was unclear and could use improvement? I would appreciate any feedback

[darrenge]

I am not sure. When I initially tried to do it, it was failing for me. When I just started over and redid it, I was able to at least get it to run and seem like it works (GH action runs etc). I have not verified the comment in the commit part yet and I have not been able to get to see the charts on GH. Not sure if user error or limitation of cont benchmarking (https://github.com/benchmark-action/github-action-benchmark/issues/270)