[WIP] add Lurk benchmarks

[austinabell]

Quickly hacked together, just opening for visibility if someone else wants to contribute to this or has something to build off of. The Lurk APIs aren't very compatible with this framework, and likely a lot of this can be streamlined.

Disclaimer that I have no affiliation with the Lurk team, just doing this for my own learning/curiosity.

I'll let the benchmarks run overnight and share to give a rough idea of timings given this setup. Proving was between 10-30s and verify between 100-800ms on my machine for the small fib benchmarks.

First run on local machine

# Benchmarks ## Table of Contents - [Benchmark Results](#benchmark-results) - [fibonacci: compile](#fibonacci:-compile) - [fibonacci: prove](#fibonacci:-prove) - [fibonacci: verify](#fibonacci:-verify) - [fibonacci:](#fibonacci:) - [fibonacci large: compile](#fibonacci-large:-compile) - [fibonacci large: prove](#fibonacci-large:-prove) - [fibonacci large: verify](#fibonacci-large:-verify) - [fibonacci large:](#fibonacci-large:) ## Benchmark Results ### fibonacci: compile | | `Triton: fibonacci-50` | `Triton: fibonacci-93` | `Miden: iter-93` | `Miden: fixed-92` | `Miden: fixed-50` | `Risc0: iter-93` | `Risc0: iter-50` | `Risc0: fixed-50` | `Risc0: fixed-92` | `Lurk: fibonacci-50 Reduction-100` | `Lurk: fibonacci-93 Reduction-100` | `Lurk: fibonacci-50 Reduction-10` | `Lurk: fibonacci-93 Reduction-10` | |:-------|:--------------------------------|:--------------------------------|:----------------------------------|:----------------------------------|:----------------------------------|:-------------------------------------|:-------------------------------------|:-------------------------------------|:-------------------------------------|:--------------------------------------------|:--------------------------------------------|:-------------------------------------------|:------------------------------------------- | | | `43.13 ns` (✅ **1.00x**) | `44.90 ns` (✅ **1.04x slower**) | `37.94 us` (❌ *879.57x slower*) | `33.13 us` (❌ *768.15x slower*) | `27.52 us` (❌ *638.14x slower*) | `30.82 ms` (❌ *714546.45x slower*) | `30.64 ms` (❌ *710340.07x slower*) | `30.47 ms` (❌ *706555.10x slower*) | `30.11 ms` (❌ *698098.67x slower*) | `23.30 ms` (❌ *540276.41x slower*) | `16.64 ms` (❌ *385757.44x slower*) | `17.22 ms` (❌ *399259.64x slower*) | `16.57 ms` (❌ *384145.13x slower*) | ### fibonacci: prove | | `Triton: fibonacci-50` | `Triton: fibonacci-93` | `Miden: iter-93` | `Miden: fixed-92` | `Miden: fixed-50` | `Risc0: iter-93` | `Risc0: iter-50` | `Risc0: fixed-50` | `Risc0: fixed-92` | `Lurk: fibonacci-50 Reduction-100` | `Lurk: fibonacci-93 Reduction-100` | `Lurk: fibonacci-50 Reduction-10` | `Lurk: fibonacci-93 Reduction-10` | |:-------|:--------------------------------|:---------------------------------|:---------------------------------|:--------------------------------|:--------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:--------------------------------------------|:--------------------------------------------|:-------------------------------------------|:------------------------------------------- | | | `68.40 ms` (✅ **1.00x**) | `128.82 ms` (❌ *1.88x slower*) | `135.59 ms` (❌ *1.98x slower*) | `68.13 ms` (✅ **1.00x faster**) | `71.71 ms` (✅ **1.05x slower**) | `3.79 s` (❌ *55.40x slower*) | `3.74 s` (❌ *54.64x slower*) | `3.75 s` (❌ *54.79x slower*) | `3.75 s` (❌ *54.87x slower*) | `9.90 s` (❌ *144.76x slower*) | `16.82 s` (❌ *245.92x slower*) | `14.07 s` (❌ *205.68x slower*) | `25.93 s` (❌ *379.14x slower*) | ### fibonacci: verify | | `Triton: fibonacci-50` | `Triton: fibonacci-93` | `Miden: iter-93` | `Miden: fixed-92` | `Miden: fixed-50` | `Risc0: iter-93` | `Risc0: iter-50` | `Risc0: fixed-50` | `Risc0: fixed-92` | `Lurk: fibonacci-50 Reduction-100` | `Lurk: fibonacci-93 Reduction-100` | `Lurk: fibonacci-50 Reduction-10` | `Lurk: fibonacci-93 Reduction-10` | |:-------|:--------------------------------|:--------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:--------------------------------------------|:--------------------------------------------|:-------------------------------------------|:------------------------------------------- | | | `11.44 ms` (✅ **1.00x**) | `12.89 ms` (❌ *1.13x slower*) | `2.13 ms` (🚀 **5.37x faster**) | `2.09 ms` (🚀 **5.47x faster**) | `2.09 ms` (🚀 **5.47x faster**) | `2.92 ms` (🚀 **3.91x faster**) | `2.90 ms` (🚀 **3.95x faster**) | `2.92 ms` (🚀 **3.92x faster**) | `3.00 ms` (🚀 **3.81x faster**) | `699.58 ms` (❌ *61.18x slower*) | `711.20 ms` (❌ *62.19x slower*) | `109.23 ms` (❌ *9.55x slower*) | `111.75 ms` (❌ *9.77x slower*) | ### fibonacci: | | `Triton: fibonacci-50` | `Triton: fibonacci-93` | `Miden: iter-93` | `Miden: fixed-92` | `Miden: fixed-50` | `Risc0: iter-93` | `Risc0: iter-50` | `Risc0: fixed-50` | `Risc0: fixed-92` | `Lurk: fibonacci-50 Reduction-100` | `Lurk: fibonacci-93 Reduction-100` | `Lurk: fibonacci-50 Reduction-10` | `Lurk: fibonacci-93 Reduction-10` | |:-------|:--------------------------------|:---------------------------------|:---------------------------------|:--------------------------------|:--------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:-------------------------------|:--------------------------------------------|:--------------------------------------------|:-------------------------------------------|:------------------------------------------- | | | `85.14 ms` (✅ **1.00x**) | `150.05 ms` (❌ *1.76x slower*) | `137.30 ms` (❌ *1.61x slower*) | `70.54 ms` (✅ **1.21x faster**) | `74.03 ms` (✅ **1.15x faster**) | `3.87 s` (❌ *45.42x slower*) | `3.86 s` (❌ *45.38x slower*) | `3.90 s` (❌ *45.75x slower*) | `3.86 s` (❌ *45.38x slower*) | `10.58 s` (❌ *124.29x slower*) | `17.52 s` (❌ *205.79x slower*) | `14.21 s` (❌ *166.93x slower*) | `25.87 s` (❌ *303.82x slower*) | ### fibonacci large: compile | | `Triton: fibonacci-1000` | `Miden: iter-1000` | `Risc0: iter-1000` | `Lurk: fibonacci-1000 Reduction-100` | `Lurk: fibonacci-1000 Reduction-10` | |:-------|:----------------------------------|:----------------------------------|:-------------------------------------|:----------------------------------------------|:--------------------------------------------- | | | `49.94 ns` (✅ **1.00x**) | `38.06 us` (❌ *762.25x slower*) | `30.77 ms` (❌ *616218.46x slower*) | `16.75 ms` (❌ *335484.88x slower*) | `16.57 ms` (❌ *331822.51x slower*) | ### fibonacci large: prove | | `Triton: fibonacci-1000` | `Miden: iter-1000` | `Risc0: iter-1000` | `Lurk: fibonacci-1000 Reduction-100` | `Lurk: fibonacci-1000 Reduction-10` | |:-------|:----------------------------------|:------------------------------|:------------------------------|:----------------------------------------------|:--------------------------------------------- | | | `2.06 s` (✅ **1.00x**) | `1.12 s` (🚀 **1.84x faster**) | `3.89 s` (❌ *1.89x slower*) | `189.23 s` (❌ *91.97x slower*) | `276.43 s` (❌ *134.36x slower*) | ### fibonacci large: verify | | `Triton: fibonacci-1000` | `Miden: iter-1000` | `Risc0: iter-1000` | `Lurk: fibonacci-1000 Reduction-100` | `Lurk: fibonacci-1000 Reduction-10` | |:-------|:----------------------------------|:-------------------------------|:-------------------------------|:----------------------------------------------|:--------------------------------------------- | | | `20.66 ms` (✅ **1.00x**) | `2.38 ms` (🚀 **8.69x faster**) | `2.95 ms` (🚀 **7.01x faster**) | `699.12 ms` (❌ *33.84x slower*) | `114.04 ms` (❌ *5.52x slower*) | ### fibonacci large: | | `Triton: fibonacci-1000` | `Miden: iter-1000` | `Risc0: iter-1000` | `Lurk: fibonacci-1000 Reduction-100` | `Lurk: fibonacci-1000 Reduction-10` | |:-------|:----------------------------------|:------------------------------|:------------------------------|:----------------------------------------------|:--------------------------------------------- | | | `2.11 s` (✅ **1.00x**) | `1.11 s` (🚀 **1.90x faster**) | `4.18 s` (❌ *1.98x slower*) | `195.70 s` (❌ *92.86x slower*) | `291.43 s` (❌ *138.28x slower*) | --- Made with [criterion-table](https://github.com/nu11ptr/criterion-table) ```

[mariari]

Thanks for the submission, I've wanted Lurk in the benchmark for a while now.

I've always thought the system was more interactive, and a self contained system so I've never attempted to include it.

How was the rust API over it? I assume It's harder than running the system by hand? Then again I'm not sure if the lurk team has to integrate it with their other rust work.

[austinabell]

Yeah, it definitely doesn't fit cleanly, and it would make sense if it isn't included in general; it was just to explore to see if it could. I added the initial run benchmark in the PR description, but something is likely off with the setup as it is orders of magnitude off.

My initial guess is that the params, reductions, and/or the IO side effects might be slowing this down. Hard to get a good idea intuitively, but I might look more into this later.

@huitseeker any thoughts from you or someone on the Lurk team to point this in the right direction or indicate if it doesn't fit within this benchmark?

[mariari]

It seems this fails to build on my end

error: no matching package found
searched package name: `abomonation_derive`
perhaps you meant:      abomonation_derive_ng
location searched: https://github.com/lurk-lab/abomonation_derive.git
required by package `lurk v0.2.0 (https://github.com/lurk-lab/lurk-rs?rev=d65a71ced96b190c31dd1d19ad1f58fed3b9f4aa#d65a71ce)`
    ... which satisfies git dependency `lurk` of package `lurk v0.1.0 (/home/taichi/Documents/Work/Repo/ZKP/zkp-compiler-shootout/shootout/lurk)`

[austinabell]

Closing as I don't have any plan to resolve breaking updates, and I realistically shouldn't be adding the benchmarks since I'm not affiliated with the team. Was just opening the PR as a starting point if someone from the team wanted to take over (they had mentioned wanting to benchmark against other systems somewhere).

[mariari]

Closing as I don't have any plan to resolve breaking updates, and I realistically shouldn't be adding the benchmarks since I'm not affiliated with the team. Was just opening the PR as a starting point if someone from the team wanted to take over (they had mentioned wanting to benchmark against other systems somewhere).

You don't have to be affiliated with the team to open benchmarks, I'm not affiliated with triton nor miden, I'm happy with any contribution to the project