search

timescale / tsbs

Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
MIT License
1.26k stars 297 forks source link

Optimize row validation in QuestDB loader #186

Open puzpuzpuz opened 2 years ago

puzpuzpuz commented 2 years ago

Environment: Ubuntu 20.04, Go 1.16.7, RAM disk + socat /dev/null,ignoreeof tcp-listen:9010,fork,reuseaddr TCP server to make sure that the benchmark itself is under the load test.

Before:

$ GOMAXPROCS=4 ./tsbs_load_questdb --ilp-bind-to 127.0.0.1:9010 --file /mnt/ramdisk/data --workers 4
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1644310831,7179259.87,7.180000E+07,7179259.87,717925.99,7.180000E+06,717925.99
1644310841,7149649.00,1.433000E+08,7164454.84,714964.90,1.433000E+07,716445.48
1644310851,7260084.91,2.159000E+08,7196329.66,726008.49,2.159000E+07,719632.97
1644310861,7179762.17,2.877000E+08,7192187.83,717976.22,2.877000E+07,719218.78

Summary:
loaded 345600000 metrics in 48.071sec with 4 workers (mean rate 7189420.06 metrics/sec)
loaded 34560000 rows in 48.071sec with 4 workers (mean rate 718942.01 rows/sec)

After:

$ GOMAXPROCS=4 ./tsbs_load_questdb --file /mnt/ramdisk/data --ilp-bind-to 127.0.0.1:9010 --workers 4
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1644319487,20755596.15,2.076000E+08,20755596.15,2075559.62,2.076000E+07,2075559.62

Summary:
loaded 345600000 metrics in 16.528sec with 4 workers (mean rate 20910444.40 metrics/sec)
loaded 34560000 rows in 16.528sec with 4 workers (mean rate 2091044.44 rows/sec)
jerrinot commented 1 year ago

Hello, is there any chance to have this merged?