search

timescale / tsbs

Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data
MIT License
1.26k stars 299 forks source link

Add the --analyze option for tsbs_load_timescaledb #148

Closed mdcallag closed 3 years ago

mdcallag commented 3 years ago

The default for the option is false and vacuum analyze is done for each table when true. Even for insert-only workloads Postgres vacuum is useful as it sets bits in the visibility map so that secondary index scans can really be index-only.

Note that DbCreatorCloser.Close used to be called after the load and was changed as part of the refactoring to get called when Loader.preRun finishes (which is after create table, but before the load is started). I assume that change was a mistake and restore the old behavior here.

Example output is below. The output from Close() is interleaved with the stats reporter output because the stats reporter goroutine isn't stopped before process exist and a pending PR fixes that.

time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s

Summary: loaded 17452800 metrics in 9.803sec with 4 workers (mean rate 1780368.33 metrics/sec) loaded 1555200 rows in 9.803sec with 4 workers (mean rate 158646.68 rows/sec) Close: vacuum analyze tags Close: vacuum analyze cpu Close: vacuum analyze diskio 1610580022,1745255.84,1.745280E+07,1745255.84,155517.85,1.555200E+06,155517.85 Close: vacuum analyze nginx Close: vacuum analyze redis Close: vacuum analyze disk Close: vacuum analyze kernel Close: vacuum analyze mem Close: vacuum analyze net Close: vacuum analyze postgresl

mdcallag commented 3 years ago

Hello reviewer, I am a golang noob. The code runs with and without setting the option. However, I don't know which tests I should run other than doing "go test" in a few directories.

codecov-io commented 3 years ago

Codecov Report

Merging #148 (4712dfc) into master (3cc83b2) will decrease coverage by 0.13%. The diff coverage is 0.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #148      +/-   ##
==========================================
- Coverage   65.40%   65.27%   -0.14%     
==========================================
  Files         145      145              
  Lines        5952     5964      +12     
==========================================
  Hits         3893     3893              
- Misses       1946     1958      +12     
  Partials      113      113
Impacted Files Coverage Δ
load/loader-no-flow-control.go 0.00% <0.00%> (ø)
load/loader.go 50.79% <0.00%> (-0.41%) :arrow_down:
pkg/targets/timescaledb/benchmark.go 0.00% <0.00%> (ø)
pkg/targets/timescaledb/creator.go 41.42% <0.00%> (-2.85%) :arrow_down:
pkg/targets/timescaledb/implemented_target.go 0.00% <0.00%> (ø)
pkg/targets/timescaledb/program_options.go 60.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3cc83b2...4712dfc. Read the comment docs.

jonatas commented 3 years ago

@mdcallag thanks for the contribution. I'll close this PR as you already shared that you're no longer interested in contributing here.

This option looks very interesting but I'm not able to finalize it soon, so let's close it for now and we can reuse it in the future if we go in this direction.