Add some benchmarking code with Stan and Nimble

sunxd3 commented 1 month ago

The scripts can be used to run examples from Volume 1 of WinBUGS examples.

Stan examples are more optimized (vectorized computation, human effort went into transformed data and generated quantities, etc.). ~~Regardless, on these smaller problems, we should be able to match it's performance when compilation works.~~ With ReverseDiff compiled tape, JuliaBUGS gradient and density computation is several times (5- 10x) slower than Stan. ReverseDiff is quite optimized, but it still has some fundamental overhead with the tape. To get comparable performance as Stan, I think we need to generate Julia code and use Mooncake or Enzyme.

Nimble's interface is a bit more tricky, the logp and gradient computation code is from Chapter 17. (code from Chapter 16 runs slowly). I am a bit confused by how to write nimblefunction. But the speed is good and it seems to work. Nimble's BUGS examples are a bit out-of-date, it looks to me some examples are from WinBUGS 0.5 era. But the models are generally the same.

I think moving forward, use Stan as the target is good, easy to setup and also a good performance aim.

[x] integrate Stan into the current benchmark code

coveralls commented 1 month ago

Pull Request Test Coverage Report for Build 11163678764

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.098%

Totals
Change from base Build 11092099802:	0.0%
Covered Lines:	1470
Relevant Lines:	1769

💛 - Coveralls

github-actions[bot] commented 4 weeks ago

BridgeStan not found at location specified by $BRIDGESTAN environment variable, downloading version 2.5.0 to /home/runner/.bridgestan/bridgestan-2.5.0 Done! Model dogs produces error: ErrorException("log_density() failed with exception: Exception: bernoulli_lpmf: Probability parameter is inf, but must be in the interval [0, 1] (in '/home/runner/work/JuliaBUGS.jl/JuliaBUGS.jl/benchmark/stan-example-models/bugs_examples/vol1/dogs/dogs.stan', line 37, column 6 to line 38, column 62)\n")	Model	Parameter Count	Data Count	Stan Density Time (µs)	Stan Density Gradient Time (µs)	JuliaBUGS Density Time with Graph Walk (µs)
rats	65	150	5.4782	8.47267	8049.13	96.57
pumps	12	10	0.999103	1.33438	188.192	7.15075
dogs	2	720	NA	NA	3486.91	205.945
seeds	26	21	2.60127	3.19378	736.786	21.55
surgical_realistic	14	12	1.29105	1.70612	262.3	9.182
magnesium	108	96	10.5045	11.9975	10767.9	86.302
salm	22	18	2.36942	3.07567	612.419	13.6555
equiv	15	20	2.45364	3.42512	469.137	17.232
dyes	9	30	1.00975	1.35573	283.53	13.6055
stacks	6	21	1.15616	1.72735	620.448	15.639
epil	303	236	32.41	38.051	73412.8	273.05
blockers	47	44	3.24722	3.75271	2272.28	33.142
oxford	244	240	16.32	19.126	54837.5	215.372
lsat	1006	5000	174.005	214.971	3.42789e6	1820.51
bones	33	422	72.957	90.319	15077.0	203.435
mice	20	65	7.44067	9.541	1135.7	46.247
kidney	64	58	10.8255	16.471	4077.78	92.4275
leuk	18	714	22.352	26.81	10311.3	298.848
leukfr	40	714	25.327	31.509	20590.7	255.788

sunxd3 commented 4 weeks ago

@yebai a new CI for benchmark is set up and it reports something like https://github.com/TuringLang/JuliaBUGS.jl/pull/214#issuecomment-2392137988.

I didn't add nimble to the benchmark to be run on PR, but the code is in the repo (we just don't run it), because running Nimble requires more dependency through R etc, and the comparison to Stan should be sufficient information (nimble is 2-3x slower than Stan).

I am merging this now, can start another PR for further improvement.

TuringLang / JuliaBUGS.jl