How to run?

[rok-cesnovar]

Hey,

do you have a short example script to show how I can run this for some of the functions? Should have some time to set this up now.

[rok-cesnovar]

So if I understand correctly, its:

# build benchmark and other math libraries
./run_py branch script.sh
chmod +x script.sh
./script.sh

[bbbales2]

Nooooooooooooooooooooo I didn't see my issue! I am a bad Git branch host. I think I need to make my Git pokes more aggressive.

The simplest way to test a benchmark is:

make clean # you basically always have to do make clean since the make here doesn't track changes through to the Math library
make src/dot_self # remember to make clean
src/dot_self # default options are good for development

I'll describe the rest of the scripts and workflow so you see the full picture. Anything here could get changed or chopped up or whatever.

---

For fancier benchmarks, there are two scripts, build.sh and run.py in the repo.

The first argument to build.sh is required. It is the branch you want to build the benchmarks for. It is used like:

git checkout $1

If the second argument is not nothing (if it is anything), then the script builds as many of the benchmarks as possible passing VARMAT=true to the compiler:

if [ -z "$2" ]
then
    make -j8 --ignore-errors all
else
    make -j8 VARMAT=true --ignore-errors all
fi

The compiler builds as much as it can with this flag. Because there is the --ignore-errors, if it fails to build a benchmark the assumption is that VARMATs are not supported and the failure is okay!

(The VARMAT logic wraps into all the C++ benchmarks through the helper macro CAST_VAR here, and an example use here)

---

run.py is a script I wrote to figure out if it is necessary to run benchmarks and generate a script for the things that haven't been run. I suspect this script gets tossed for something else, but for now it takes two arguments.

It takes a few arguments:

parser.add_argument("branch", help = "branch to benchmark")
parser.add_argument("run_script", help = "location to write run script")
parser.add_argument("--overwrite", help = "overwrite output", action = "store_true")
parser.add_argument("--varmat", help = "use var_value<Matrix<T, R, C>> types", action = "store_true")

So if you do:

./run.py develop run.develop.sh

It will check out develop and build it (using build.sh), see if any of these benchmarks need run (by looking at the file system to see if the output files have been produced), and if not write the necessary list of commands to run.develop.sh.

An example script might look like:

src/tcrossprod --benchmark_color=fase --benchmark_format=csv --benchmark_repetitions=30 --benchmark_min_time=0.1 > benchmarks/develop_2cf4310702_matvar/tcrossprod.csv
src/multiply_lower_tri_self_transpose --benchmark_color=fase --benchmark_format=csv --benchmark_repetitions=30 --benchmark_min_time=0.1 > benchmarks/develop_2cf4310702_matvar/multiply_lower_tri_self_transpose.csv

And so I run these with:

cat run.develop.sh | parallel -j2

I don't run too many in parallel since a lot of our functions are memory bound.

---

I have another secret script which I used to update the benchmarks as necessary. It looks like this but I expect it would be replaced by anything automated:

for branch in feature/convert_to_tadj1 feature/a_few_varmats feature/varmat-cholesky                 
do
    branch_base=`basename $branch`
    ./run.py $branch run.$branch_base.matvar.sh
    cat run.$branch_base.matvar.sh | parallel -j2
    ./run.py $branch run.$branch_base.varmat.sh --varmat
    cat run.$branch_base.varmat.sh | parallel -j2
done

And I have an r-script that takes the output and packs it up in a dataframe to be shipped over to the Shiny server:

library(tidyverse)
library(stringr)

df = lapply(list.files("benchmarks"), function(folder) {
  matches = str_match(folder, "(.*)_(\\w+)_([a-z]+)")
  branch = matches[2]
  commit = matches[3]
  varmat = matches[4]

  lapply(list.files(file.path("benchmarks", folder)), function(csv) {
    tryCatch({
      read.csv(file.path("benchmarks", folder, csv)) %>%
        as_tibble() %>%
        rename(time = real_time) %>%
        select(name, time) %>%
        filter(!str_detect(name, "_mean$") &
                 !str_detect(name, "_median$") &
                 !str_detect(name, "_stddev$") &
                 !str_detect(name, "toss_me")) %>%
        separate(name, c("name", "n", "tmp"), "/") %>%
        mutate(n = as.integer(n)) %>%
        select(-tmp) %>%
        mutate(benchmark = folder)
    }, error = function(e) {
      print(paste(folder, csv, e))
      return(NULL)
    })
  }) %>%
    bind_rows()
}) %>%
  bind_rows()

saveRDS(df, file = "df.rds")

[bbbales2]

As a formal apology for my inhospitality, let me pin this issue.

[rok-cesnovar]

Cool cool cool. Thanks!