Support vmap in pytest benchmarking

microsoft / knossos-ksc

Compiler with automatic differentiation

Other

45 stars 10 forks source link

Support vmap in pytest benchmarking #855

Closed dcrc2 closed 3 years ago

dcrc2 commented 3 years ago

Update pytest benchmarking to use tsmod2ksmod, matching the corresponding change in run-bench.py:

https://github.com/microsoft/knossos-ksc/pull/838/files#diff-19d15b57f2c48a8fac44562c3dfd51f7d9be1f31236e51dd4bd574deabd44cf7

AB#19336

dcrc2 commented 3 years ago

Looks good, has it been run on sqrl benchmark?

Ah, good thing you reminded me about this: compilation succeeds but the results are reported as being incorrect.

dcrc2 commented 3 years ago

compilation succeeds but the results are reported as being incorrect.

So there are two issues in sqrl:

Incorrect derivatives for sin, fixed in #861.
Out-of-memory on the largest example. I think this is a more general SUF issue. I suggest that we just comment out the largest parameter size for the moment if we want to run correctness checks in CI.

cgravill commented 3 years ago

Nice find!

I'm fine with reducing the largest size. I've been wondering about how best to do the correctness checks on PR, but at least for now the overnight is our real benchmark! Perhaps we need to make the benchmark tests individually fail-able. In any case, I'm happy for it to be reduced for now and we'll figure out elsewhere.