search

shutter-network / rolling-shutter

Rolling Shutter is an MEV protection system to be plugged into rollups.
https://twitter.com/project_shutter/
26 stars 7 forks source link

Benchmark message handling #455

Open jannikluhn opened 2 weeks ago

jannikluhn commented 2 weeks ago

To run them try the following commands:

go test ./keyper/epochkghandler/... -bench ^BenchmarkValidateKeysIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkHandleKeysIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkHandleFirstKeySharesIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkValidateFirstKeySharesIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkHandleFirstKeySharesIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkValidateSecondKeySharesIntegration$ -run ^$
go test ./keyper/epochkghandler/... -bench ^BenchmarkHandleSecondKeySharesIntegration$ -run ^$
konradkonrad commented 1 week ago

I had to start a local postgres instance in order for this to run:

docker run --rm -it -e POSTGRES_USER=pguser -e POSTGRES_PASSWORD=password -e POSTGRES_DB=testdb --net=host cimg/postgres:13.9

Then I had to set an environment variable:

export ROLLING_SHUTTER_TESTDB_URL=postgres://pguser:password@localhost:5432/testdb
konradkonrad commented 1 week ago

Are these results comparable to what you are seeing?

goos: linux
goarch: amd64
pkg: github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler
cpu: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
BenchmarkValidateKeysIntegration-8                      1000000000               0.6120 ns/op
BenchmarkHandleKeysIntegration-8                        1000000000               0.09876 ns/op
BenchmarkValidateFirstKeySharesIntegration-8            1000000000               0.2187 ns/op
BenchmarkHandleFirstKeySharesIntegration-8              1000000000               0.1358 ns/op
BenchmarkValidateSecondKeySharesIntegration-8           1000000000               0.2173 ns/op
BenchmarkHandleSecondKeySharesIntegration-8             1000000000               0.6793 ns/op
PASS
ok      github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler        125.678s
jannikluhn commented 1 week ago

Yes, for a few hunderd identity preimages or less. If I increase the number to 500 - 1000 it goes way up. Note that you also have to increase MaxNumKeysPerMessage if you try that.

konradkonrad commented 1 week ago

If I increase the number to 500 - 1000 it goes way up.

that doesn't sound linear.

konradkonrad commented 1 week ago

okay, there is definitely something odd going on. I configured MaxNumKeysPerMessage = 512 and numIdentityPreimages = 500. Then I did variations on -benchtime=Nx (see https://pkg.go.dev/cmd/go#hdr-Testing_flags). The ns/op goes down dramatically with higher N. Also the overall run time was very similar in all tests I ran (about 42s). 

go test -bench=. ./keyper/epochkghandler -cpuprofile keyhandler512.profile.out -run=^# -benchtime=10000x
goos: linux
goarch: amd64
pkg: github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler
cpu: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
BenchmarkValidateKeysIntegration-8                         10000            297247 ns/op
BenchmarkHandleKeysIntegration-8                           10000             42700 ns/op
BenchmarkValidateFirstKeySharesIntegration-8               10000            108306 ns/op
BenchmarkHandleFirstKeySharesIntegration-8                 10000             31613 ns/op
BenchmarkValidateSecondKeySharesIntegration-8              10000            108203 ns/op
BenchmarkHandleSecondKeySharesIntegration-8                10000            322352 ns/op
PASS
ok      github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler        42.261s
go test -bench=. ./keyper/epochkghandler -cpuprofile keyhandler512.profile.out -run=^# -benchtime=10x   
goos: linux
goarch: amd64
pkg: github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler
cpu: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
BenchmarkValidateKeysIntegration-8                            10         305131048 ns/op
BenchmarkHandleKeysIntegration-8                              10          29710135 ns/op
BenchmarkValidateFirstKeySharesIntegration-8                  10         111074644 ns/op
BenchmarkHandleFirstKeySharesIntegration-8                    10          38698010 ns/op
BenchmarkValidateSecondKeySharesIntegration-8                 10         110245458 ns/op
BenchmarkHandleSecondKeySharesIntegration-8                   10         334928515 ns/op
PASS
ok      github.com/shutter-network/rolling-shutter/rolling-shutter/keyper/epochkghandler        43.328s
konradkonrad commented 1 week ago

okay, there is definitely something odd going on.

it turns out, the benchmark wasn't looping. that is fixed now.