search

ssbc / netsim-cookbook

collecting various netsim test scripts in one place
5 stars 0 forks source link

Initial multi-implementation netsim test results #2

Open mycognosist opened 3 years ago

mycognosist commented 3 years ago

Sharing the report here for discussion before I make a PR with test scripts etc.

Netsim Replication Benchmarks

Full Sync Two SSB Nodes

All tests were run with the same fixtures consisting of 100,000 messages from 20 unique authors. Two identities were chosen: one is labelled as server and the other as peer. The skipoffset peer netsim command was used to ensure the peer starts with 0 messages. The peer is connected to the server and we time how long it takes for peer to replicate the full feed of server (4219 messages). A mutual follow is established before connecting the peer to the server (hence the 4220 message count in the test output). Replication rate is calculated as the total peer messages divided by the run time.

See the bottom of the page for the full test script.

Summary

server peer messages time messages / second
go go 4220 0.509 8291
js js 4220 2.65 1592
go js 4220 2.831 1491
go js (db2) 4220 3.948 1069
js (db2) js (db2) 4220 7.239 583
js go 76 160.596 0.5
js (db2) go 1 160.597 0

JS - JS (db)

# Total time: 16.891932964s
# Active time: 4.391932964s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            16.291s       3.791s         4220
# peer              14.203s       3.703s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                 2.65s

Replication rate (msgs / second): 1592

JS - JS (db2)

# Total time: 21.009802043s
# Active time: 8.509802043s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            20.668s       8.168s         4220
# peer              18.328s       7.828s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                7.239s

Replication rate (msgs / second): 583

Go - Go

# Total time: 14.898761001s
# Active time: 2.398761001s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server             13.78s        1.28s         4220
# peer              12.799s       2.299s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                 509ms

Replication rate (msgs / second): 8291

Go - JS (db)

server : go ... peer : js

# Total time: 16.877485387s
# Active time: 4.377485387s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            16.332s       3.832s         4220
# peer              14.767s       4.267s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                2.831s

Replication rate (msgs / second): 1491

server : js ... peer : go

# Total time: 2m54.868505121s
# Active time: 2m32.368505121s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server          2m53.837s    2m31.337s         4220
# peer            2m52.303s    2m31.803s           76
# 
# Started timers & final elapsed time
# Label                Time
# run             2m40.596s

Replication rate (msgs / second): 0.5

Go - JS (db2)

server : go ... peer : js

# Total time: 17.870999912s
# Active time: 5.370999912s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            17.577s       5.077s         4219
# peer              15.767s       5.267s         4220
# 
# Started timers & final elapsed time
# Label                Time
# run                3.948s

Replication rate (msgs / second): 1069

server : js ... peer : go

# Total time: 2m54.645813867s
# Active time: 2m32.145813867s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server          2m53.614s    2m31.114s         4220
# peer             2m51.95s     2m31.45s            1
# 
# Started timers & final elapsed time
# Label                Time
# run             2m40.597s

Replication rate (msgs / second): 0

Test Script

The only changes to this script between runs are the start server go-sbot and start peer go-sbot commands; go-sbot is substituted for ssb-server or ssb-server-db1 when required.

comment [ configuring peer ]
enter peer
hops peer 3
load peer @sWJqvI4hdv96aRMk1pQOJo1frXRFr7oBB5/r/jBMcxw=.ed25519
skipoffset peer

comment [ configuring server ]
enter server
hops server 3
load server @NZpjhmgTaHOHFZI6G6dIuWwa3UqwSpfRpMbY5ue6vAI=.ed25519

start server go-sbot
start peer go-sbot

follow server peer
wait 4000
follow peer server
wait 4000

timerstart run

connect peer server
waituntil peer server@latest

timerstop run

disconnect peer server
stop server
stop peer
mycognosist commented 3 years ago

Six Second Replication

All tests were run with the same fixtures consisting of 100,000 messages from 20 unique authors. Two identities were chosen: one is labelled as server and the other as peer. The alloffsets server netsim command was used to ensure the server starts with all 100000 messages. The peer was connected to the server and we measured how many messages were replicated from server to peer in 6 seconds. A mutual follow was established before connecting the peer to the server (hence the 10249 message count for peer in the test output).

Each test was run three times and averages were calculated for run time and message counts. Replication rate is calculated as (the total peer messages minus 10249) divided by the run time.

Summary

All values are calculated as an average across three test runs for each implementation.

server peer messages time messages / second relative performance
go go 59754 6.513 9175 2.1x
js (db2) js (db2) 42161 6.521 6465 1.5x
js js 28394 6.545 4338 1.0x

Go - Go

Sample output from a single test run:

Note: go-sbot was run without the -enable-ebt flag in this test.

# Total time: 32.75269056s
# Active time: 2.25269056s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            34.151s       3.651s       100002
# peer              18.749s        249ms        69727
# 
# Started timers & final elapsed time
# Label                Time
# run                6.512s

JS - JS (db2)

Sample output from a single test run:

# Total time: 46.21878452s
# Active time: 25.71878452s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server            58.721s      38.221s       100001
# peer              19.376s        876ms        55613
# 
# Started timers & final elapsed time
# Label                Time
# run                6.529s

JS - JS

Sample output from a single test run:

# Total time: 57.427148477s
# Active time: 26.927148477s
# Puppet count: 2
# Puppet         Total time  Active time   # messages
# server          1m11.849s      41.349s       100002
# peer               20.62s        2.12s        38779
# 
# Started timers & final elapsed time
# Label                Time
# run                6.514s

Test Script

The only changes to this script between runs are the start server go-sbot and start peer go-sbot commands; go-sbot is substituted for ssb-server or ssb-server-db1 when required.

comment [ six second sync two js db2 ssb servers 100k msgs ]

comment [ configuring peer ]
enter peer
hops peer 3
load peer @sWJqvI4hdv96aRMk1pQOJo1frXRFr7oBB5/r/jBMcxw=.ed25519

comment [ configuring server ]
enter server
hops server 3
load server @NZpjhmgTaHOHFZI6G6dIuWwa3UqwSpfRpMbY5ue6vAI=.ed25519
alloffsets server

start server ssb-server
start peer ssb-server

follow server peer
follow peer server
wait 10000

timerstart run

connect peer server
wait 6000

timerstop run