Error: No benchmark found for bench suite. Possibly mangled output from Catch2:

[bitbacchus]

I'm about to set up continuous benchmarking for our project. We benchmark C++ code in a combined R/C++ project with catch2 in a docker container.

When I try to deploy the benchmark results, I get an error that I struggle to approach:

Error: No benchmark found for bench suite. Possibly mangled output from Catch2:

The benchmark results look fine, however:

Run leoholz/github-action-benchmark@v1
  with:
    name: Spectre Benchmark
    tool: catch2
    output-file-path: benchmark_result.txt
    github-token: ***
    auto-push: true
    alert-threshold: 150%
    comment-on-alert: true
    fail-on-alert: false
    alert-comment-cc-users: @bitbacchus
    gh-pages-branch: gh-pages
    benchmark-data-dir-path: dev/bench
    skip-fetch-gh-pages: false
    comment-always: false
    save-data-file: true
/usr/bin/docker exec  a9f0ac018dd04072460f5c72e60419a21237b5e40c9c55dd861053ffe2aeba52 sh -c "cat /etc/*release | grep ^ID"
Error: No benchmark found for bench suite. Possibly mangled output from Catch2:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
benchmark is a Catch v2.13.4 host application.
Run with -? for options

-------------------------------------------------------------------------------
MinConf benchmarks
-------------------------------------------------------------------------------
../../tests/benchmark/benchmark-minconf.cpp:6
...............................................................................

benchmark name                       samples       iterations    estimated
                                     mean          low mean      high mean
                                     std dev       low std dev   high std dev
-------------------------------------------------------------------------------
MinConf optimize 3 sites, 3                                                    
species, <<5k iterations                       100             1    146.934 ms 
                                        1.42295 ms    1.42224 ms    1.42373 ms 
                                        3.80964 us    3.33283 us    4.54339 us 

MinConf optimize 100 sites, 139                                                
species, 5 iterations                          100             1     22.8952 s 
                                        227.471 ms    227.076 ms    227.933 ms 
                                        2.18339 ms    1.78353 ms     3.1476 ms 

Calculate commonness for a 100x100                                             
solution                                       100             1     35.275 ms 
                                         353.03 us     352.65 us     353.71 us 
                                        2.52869 us    1.67274 us    3.89837 us 

===============================================================================
test cases: 1 | 1 passed
assertions: - none -

Do you have an idea or suggestions?

[bitbacchus]

For completeness, here is the yml file I use:

name: benchmark

on:
  push:
     branches:
       - dev
       - master
       - benchmark
  #   paths:
  #     - 'src/**'
  #     - 'R**'
  # workflow_dispatch:

jobs:
  benchmark:
    name: Performance regression check
    runs-on: ubuntu-latest
    container: rocker/r-ubuntu:18.04
    if: "!contains(github.event.head_commit.message, 'ci skip')"
    steps:
    - name: Install prerequisites
      run: |
        apt-get update -y
        apt-get install -y qt5-qmake
        R -e 'install.packages(c("Rcpp", "RInside", "remotes"))'
        # the R package of spectre works fine with RcppProgress from CRAN, but here we need some extra functionality
        R -e 'remotes::install_github("bitbacchus/rcpp_progress")' 
    - uses: actions/checkout@v2
    - name: Build and run benchmarks with Catch2
      run: |
        cd spectre_rcpp_dev/benchmark
        /usr/lib/x86_64-linux-gnu/qt5/bin/qmake
        make
        cd ../../
        ./spectre_rcpp_dev/benchmark/benchmark | tee benchmark_result.txt
    # Download previous benchmark result from cache (if exists)
    - name: Download previous benchmark data
      uses: actions/cache@v1
      with:
        path: ./cache
        key: ${{ runner.os }}-benchmark
    # Run `github-action-benchmark` action
    - name: Store benchmark result
      #uses: rhysd/github-action-benchmark@v1
      uses: leoholz/github-action-benchmark@v1 # until PR #38 is merged
      with:
        # What benchmark tool the benchmark_result.txt came from
        name: Spectre Benchmark
        tool: 'catch2'
        output-file-path: benchmark_result.txt
        github-token: ${{ secrets.GITHUB_TOKEN }}
        auto-push: true
        # Show alert with commit comment on detecting possible performance regression
        alert-threshold: "150%"
        comment-on-alert: true
        fail-on-alert: false
        alert-comment-cc-users: "@bitbacchus"

[bitbacchus]

I can reproduce this now. This happens whenever the string for the BENCHMARK name in Catch2 is longer than 35 characters, i.e

TEST_CASE("Fibonacci") {
  // now let's benchmark:
  BENCHMARK("test1234567890test1234567890test123") { return fib(10); };
}

works, but

TEST_CASE("Fibonacci") {
  // now let's benchmark:
  BENCHMARK("test1234567890test1234567890test1234") { return fib(10); };
}

fails.

Luckily, there seems to be a PR, already: #46

[epompeii]

In the meantime, there is a workaround:

As a workaround we can add #define CATCH_CONFIG_CONSOLE_WIDTH 200 (or any width needed) in the benchmark source to overwrite catch2's default output width of 80 characters to prevent long benchmark names wrapping.

https://github.com/benchmark-action/github-action-benchmark/pull/46#issuecomment-1028741207

Or you can try Bencher, the cpp_catch2 adapter can handle multi-line benchmark names.