BUG: ASV Timeout not working

[thaitranduc2001]

Hello, I am currently working with Gurobi (Search "Gurobi Optimization") in Python, and I am using ASV to benchmark my code. This means, the Gurobi solution process is included within my benchmark.

I set the "timeout" attribute of the class as such, hoping that the benchmark would fail after 30 mins: “class Benchmark: timeout = 30 * 60 # 1800 seconds or 30 minutes def methodExample(): …”

However, the benchmark keeps on running after 30 minutes. In Gurobi, the "Presolve" step is finished, but then it is stuck at the second step "Root Simplex" for way too long (more than 30 minutes). Here is a screenshot of my Root Simplex Log (see image):

The process can be interrupted by CTRL+C every time (I am on MacOS btw), so this is likely an ASV problem, I presume. How to reproduce: simply "asv run -v" to see every detail of the solution process, and the whole GUROBI solution process would appear in the console, and you can see it directly.

[HaoZeke]

Not reproducible, please provide exact steps to reproduce this, using a minimal working example. See https://stackoverflow.com/help/minimal-reproducible-example as an example.

Reopen when this is properly actionable.

[HaoZeke]

Here is an example of a reproducible report:

git clone https://github.com/airspeed-velocity/asv_samples/
cd asv_samples
micromamba create -p $(pwd)/.tmp  -c conda-forge python pip libmambapy conda-build 
micromamba activate -p $(pwd)/.tmp
git checkout memray-plugin
pip install asv
python scripts/gen_asv_conf.py asv.conf.base.json
asv run

This will generate the following:

· Creating environments
· Discovering benchmarks
· Running 2 total benchmarks (1 commits * 2 environments * 1 benchmarks)
[ 0.00%] · For asv_samples commit 08c73aa6 <memray-plugin>:
[ 0.00%] ·· Benchmarking virtualenv-py3.12-asv_=0.6.2-numpy-pip+asv-bench-memray_=0.1.2
[50.00%] ··· ...ks.MemrayBenchmarks.ray_sum                 ok
[50.00%] ··· ======== =======
              param1         
             -------- -------
                10      1.4M 
              20000    1.56M 
             ======== =======

[50.00%] ·· Benchmarking virtualenv-py3.12-asv_=0.6.2-pip+asv-bench-memray_=0.1.2
[100.00%] ··· ...ks.MemrayBenchmarks.ray_sum                 ok
[100.00%] ··· ======== =====
               param1       
              -------- -----
                 10     n/a 
               20000    n/a 
              ======== =====

However, now if you go and modify the benchmarks/benchmarks.py file:

modified   benchmarks/benchmarks.py
@@ -2,6 +2,7 @@ from asv_runner.benchmarks.mark import SkipNotImplemented

 class MemrayBenchmarks:
     params = [10, int(2e4)]
+    timeout = 0.000000000001111

     def ray_sum(self, n):
         try:

Now re-run:

asv run
· Creating environments
· Discovering benchmarks
· Running 2 total benchmarks (1 commits * 2 environments * 1 benchmarks)
[ 0.00%] · For asv_samples commit 08c73aa6 <memray-plugin>:
[ 0.00%] ·· Benchmarking virtualenv-py3.12-asv_=0.6.2-numpy-pip+asv-bench-memray_=0.1.2
[50.00%] ··· ...ks.MemrayBenchmarks.ray_sum             failed
[50.00%] ··· ======== ========
              param1          
             -------- --------
                10     failed 
              20000    failed 
             ======== ========
             For parameters: 10

             asv: benchmark timed out (timeout 1.111e-12s)

             For parameters: 20000

             asv: benchmark timed out (timeout 1.111e-12s)

[50.00%] ·· Benchmarking virtualenv-py3.12-asv_=0.6.2-pip+asv-bench-memray_=0.1.2
[100.00%] ··· ...ks.MemrayBenchmarks.ray_sum             failed
[100.00%] ··· ======== ========
               param1          
              -------- --------
                 10     failed 
               20000    failed 
              ======== ========
              For parameters: 10

              asv: benchmark timed out (timeout 1.111e-12s)

              For parameters: 20000

              asv: benchmark timed out (timeout 1.111e-12s)

So as you can see, the timeout definitely works, and the issue is clearly somewhere else.