setup is not ran for each invocation of a test

[yarikoptic]

Having following minimalistic asv.conf

```shell ❯ grep -v -e '^ *//' -e '^ *$' asv.conf.json { "version": 1, "project": "DataLad", "project_url": "http://datalad.org", "repo": ".", "branches": ["master", "maint"], // for git "environment_type": "virtualenv", "show_commit_url": "http://github.com/datalad/datalad/commit/", "pythons": ["3.7"], "benchmark_dir": "benchmarks", "env_dir": ".asv/env", "results_dir": ".asv/results", "html_dir": ".asv/html", } ```

and following minimalistic bogus benchmark file

❯ cat benchmarks/benchmarks.py
import os
import sys
import os.path as op
from glob import glob

class SampleSuperDatasetBenchmarks:
    """
    Setup a sample hierarchy of datasets to be used
    """

    def teardown(self):
        self.log("teardown")

    def log(self, msg, *args):
        """Consistent benchmarks logging"""
        with open("/tmp/bm.log", "a") as f:
            f.write(f"{os.getpid()}: " + str(msg % tuple(args)) + "\n")

    def setup_cache(self):
        self.log("setup_cache")

    def setup(self):
        self.log("setup ")

class supers(SampleSuperDatasetBenchmarks):
    def time_remove(self):
        self.log("I will be removing")
        from time import sleep
        sleep(.01)

I get

❯ rm -f /tmp/bm.log && asv run --python=same --show-stderr ; cat /tmp/bm.log
· No executable found for python 3.7
· Discovering benchmarks
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[ 0.00%] ·· Benchmarking existing-py_home_yoh_proj_datalad_datalad-maint_venvs_dev3.11_bin_python3.11
[50.00%] ··· Setting up benchmarks:21                                                                                                              ok
[50.00%] ··· Running (benchmarks.supers.time_remove--).
[100.00%] ··· benchmarks.supers.time_remove                                                                                                10.2±0.05ms
1831167: setup_cache
1831168: setup 
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: I will be removing
1831168: teardown
1831168: setup 
1831168: I will be removing
1831168: teardown
1831168: setup 
1831168: I will be removing
1831168: teardown
1831168: setup 
1831168: I will be removing
1831168: teardown
1831168: setup 
1831168: I will be removing
1831168: teardown
1831168: setup 
1831168: I will be removing
1831168: teardown
1831169: setup 
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: I will be removing
1831169: teardown
1831169: setup 
1831169: I will be removing
1831169: teardown
1831169: setup 
1831169: I will be removing
1831169: teardown
1831169: setup 
1831169: I will be removing
1831169: teardown
1831169: setup 
1831169: I will be removing
1831169: teardown
1831169: setup 
1831169: I will be removing
1831169: teardown

so as you can see there are two instances where after a single setup we get multiple I will be removing. That breaks our benchmark where setup is creating "file" which "I will be removing". So if setup is not ran after a single run, there would no longer be a file.

Did I misunderstand the point of setup in this case?

[trexfeathers]

Some notes you will hopefully find useful:

• https://asv.readthedocs.io/en/v0.6.1/writing_benchmarks.html#timing
• https://asv.readthedocs.io/en/v0.6.1/benchmarks.html#timing-benchmarks
• https://github.com/SciTools/iris/tree/main/benchmarks#asv-re-run-behaviour

[yarikoptic]

Thanks!

adding

supers.time_remove.warmup_time = 0

for my case seems to provide desired behavior!