TuringLang / TuringBenchmarking.jl

https://turinglang.org/TuringBenchmarking.jl/
MIT License
7 stars 1 forks source link

Allow specification of parameters to test for in kwargs #14

Closed torfjelde closed 1 year ago

torfjelde commented 1 year ago

Some AD backends, e.g. ReverseDiff, have a compiled mode which relies on the particular input value provided in the ADgradient call. Since we're currently just checking gradient evaluations for the same input values as the one we use for compilation of the tape, we don't catch issues related to this.

This PR just makes the input values we run benchmarks on part of the kwargs, with the default values differing from the values present in the varinfo we use for compilation.

Related discussion: https://github.com/TuringLang/Turing.jl/issues/2091#issuecomment-1778824039

Example from that discussion is now correctly identified to have issues:

julia> using Turing, TuringBenchmarking, Random

julia> @model demo() = L ~ LKJCholesky(3, 1.0)

demo (generic function with 2 methods)

julia> model = demo();

julia> make_turing_suite(model; check=true, adbackends=[:reversediff, :reversediff_compiled]);
┌ Warning: There is disagreement in the gradients!
└ @ TuringBenchmarking /drive-2/Projects/public/TuringBenchmarking.jl/src/TuringBenchmarking.jl:264
┌──────────────────────────────────────┬──────────┐
│                               Linked │ Gradient │
│                              backend │ distance │
├──────────────────────────────────────┼──────────┤
│ ReverseDiff[compiled] vs ReverseDiff │     0.31 │
└──────────────────────────────────────┴──────────┘