Enhance run_benchmarks.sh with more options and error handling

[deseti]

The current run_benchmarks.sh script lacks flexibility and robust error handling. Here are some suggestions for improvement : Add Options for Specific Benchmarks: Allow users to specify which benchmarks to run using a --benchmark option (e.g., ./run_benchmarks.sh --benchmark=hash). This would enable running only specific tests instead of all benchmarks.

Allow Specifying Thread Count: Provide a --threads option to let users set the number of threads to use (e.g., ./run_benchmarks.sh --threads=4). This would offer more control over resource utilization.

Implement Output Settings: Enable saving benchmark results to a file (like CSV or JSON) for easier analysis and comparison. Enhance Error Handling: Catch errors from cargo bench and display informative messages to the user. Improve Documentation: Add clear comments explaining the functionality of each part of the script.

implementation example

#!/usr/bin/env bash

# Created by argbash-init v2.10.0
# ARG_OPTIONAL_BOOLEAN([asm])
# ARG_OPTIONAL_BOOLEAN([multi_threads])
# ARG_OPTIONAL_STRING([benchmark])
# ARG_OPTIONAL_INTEGER([threads])
# ARG_HELP([<Jellyfish benchmarks>])
# ARGBASH_GO()
# needed because of Argbash --> m4_ignore([
### START OF CODE GENERATED BY Argbash v2.10.0 one line above ###
# Argbash is a bash code generator used to get arguments parsing right.
# Argbash is FREE SOFTWARE, see https://argbash.dev, https://github.com/matejak/argbash for more info

die()
{
    local _ret="${2:-1}"
    test "${_PRINT_HELP:-no}" = yes && print_help >&2
    echo "$1" >&2
    exit "${_ret}"
}

begins_with_short_option()
{
    local first_option all_short_options='h'
    first_option="${1:0:1}"
    test "$all_short_options" = "${all_short_options/$first_option/}" && return 1 || return 0
}

# THE DEFAULTS INITIALIZATION - OPTIONALS
_arg_asm="off"
_arg_multi_threads="off"
_arg_benchmark=""
_arg_threads=""

print_help()
{
    printf '%s\n' "<Jellyfish benchmarks>"
    printf 'Usage: %s [--(no-)asm] [--(no-)multi_threads] [--benchmark=<benchmark>] [--threads=<threads>] [-h|--help]\n' "$0"
    printf '\t%s\n' "-h, --help: Prints help"
    printf '\t--benchmark=<benchmark>: Specify the benchmark to run (e.g. --benchmark=hash)\n'
    printf '\t--threads=<threads>: Specify the number of threads to use (e.g. --threads=4)\n'
}

parse_commandline()
{
    while test $# -gt 0
    do
        _key="$1"
        case "$_key" in
            --no-asm|--asm)
                _arg_asm="on"
                test "${1:0:5}" = "--no-" && _arg_asm="off"
                ;;
            --no-multi_threads|--multi_threads)
                _arg_multi_threads="on"
                test "${1:0:5}" = "--no-" && _arg_multi_threads="off"
                ;;
            --benchmark=*)
                _arg_benchmark="${1#*=}"
                ;;
            --threads=*)
                _arg_threads="${1#*=}"
                ;;
            -h|--help)
                print_help
                exit 0
                ;;
            -h*)
                print_help
                exit 0
                ;;
            *)
                _PRINT_HELP=yes die "FATAL ERROR: Got an unexpected argument '$1'" 1
                ;;
        esac
        shift
    done
}

parse_commandline "$@"

# OTHER STUFF GENERATED BY Argbash

### END OF CODE GENERATED BY Argbash (sortof) ### ])
# [ <-- needed because of Argbash

cargo clean

if [ "$_arg_multi_threads" = on ]
then
  echo "Multi-threads: ON"
  # Do nothing
else
  echo "Multi-threads: OFF"
  export RAYON_NUM_THREADS=1
fi

if [ "$_arg_threads" != "" ]
then
  echo "Threads: $_arg_threads"
  export RAYON_NUM_THREADS=$_arg_threads
fi

if [ "$_arg_asm" = on ]
then
  echo "Asm feature: ON"
  export RUSTFLAGS="-C target-feature=+bmi2,+adx"
else
  echo "Asm feature: OFF"
  # Do nothing
fi

# Run the benchmark binary
set -e
if [ "$_arg_benchmark" != "" ]
then
  cargo bench --bench $_arg_benchmark
else
  cargo bench
fi

# ^^^  TERMINATE YOUR CODE BEFORE THE BOTTOM ARGBASH MARKER  ^^^

# ] <-- needed because of Argbash

suggestion

onsider using a dedicated benchmarking library like criterion for more accurate and detailed results. Explore adding options to control the number of iterations and warm-up cycles for benchmarks. This enhancement would make the run_benchmarks.sh script more versatile and user-friendly.

source file : https://github.com/EspressoSystems/jellyfish/blob/main/scripts/run_benchmarks.sh