search

TPC-Council / HammerDB

HammerDB Database Load Testing and Benchmarking Tool
http://www.hammerdb.com
GNU General Public License v3.0
542 stars 115 forks source link

Add parent jobid to group specific jobs and chart generation for multi-VUs runs #661

Closed alfer-edb closed 3 weeks ago

alfer-edb commented 5 months ago

Is your feature request related to a problem? Please describe. it would be nice to have a feature to generate a chart showing VUs on the x-axis and NOPM on the y-axis in a multi-VUs run.

Describe the solution you'd like Group VUs metrics (timings and results) under the same job id for a multi-VUs run. Generate chart for the whole run (NOPM y-axis, VUs x-axis)

Describe alternatives you've considered Run the command jobs results into hammerdbcli, get all the output and parse with an external script to collect all the datapoints of interest filtering by start/end date and excluding build metrics. For timing metrics, retrieve job id from hammerdb.log and store them into some sort of py dictionary. Eventually a script to build a chart with the above metrics.

Additional context

Something like the following could be a TCL run script to be run:

puts "SEQUENCE STARTED"
< set jobid >
foreach z { {{ tprocc_vusers }} } {
puts "$z VU TEST"
vuset vu $z
vucreate
vurun
vudestroy
        }
puts "TEST SEQUENCE COMPLETE"
puts "Jobid is $jobid"
exit
sm-shaw commented 1 month ago

This feature has been developed and can be browsed/tested here: https://github.com/sm-shaw/HammerDB/tree/661 it supports both GUI (autopilot feature) and CLI "jobs profileid"

After further testing/debugging a PR is planned in the next few days.

scr1 scr2

sm-shaw commented 3 weeks ago

PR #707 adds this feature.

Example script used for test shows use of new jobs profileid 1 command to group subsequent jobs under a profileid.

#!/bin/tclsh
# maintainer: Pooja Jain

set tmpdir $::env(TMP)
puts "SETTING CONFIGURATION"
jobs profileid 1
dbset db maria
dbset bm TPC-C
giset timeprofile xt_gather_timeout 1200
giset commandline keepalive_margin 1200

diset connection maria_host 127.0.0.1
diset connection maria_port 3306
diset connection maria_socket /tmp/mariadb.sock
#
diset tpcc maria_user root
diset tpcc maria_pass maria
diset tpcc maria_dbase tpcc
diset tpcc maria_driver timed
diset tpcc maria_rampup 2
diset tpcc maria_duration 5
diset tpcc maria_no_stored_procs false
diset tpcc maria_allwarehouse false
diset tpcc maria_timeprofile true
diset tpcc maria_purge true
#start CPU
metstart
puts "TEST STARTED"
foreach z { 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88
92 96 100 104 108 112 116 120 } {
loadscript
vuset vu $z
vuset logtotemp 1
vucreate
tcstart
tcstatus
vurun
tcstop
vudestroy
}
metstop
puts "TEST COMPLETE"

When complete web service shows output such as follows with new performance profile section.

app1

Clicking on the profile id shows the performance profile for all of the jobs

app2

Charts are interactive so can show just NOPM or TPM

app3

Clicking on the maxjob takes directly to the job details

app4

Example shows new metrics

app5

CLI jobs profile command can manually retrieve a performance profile which is a subset of jobs result. 

hammerdb>job profile 1
{
  "66686B0D61A903E233236323": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:19:41",
    "activevu": "1",
    "nopm": "16254",
    "tpm": "37709"
  },
  "66686CB661A903E243430373": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:26:46",
    "activevu": "2",
    "nopm": "35434",
    "tpm": "82497"
  },
  "66686E6161A903E263536313": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:33:53",
    "activevu": "4",
    "nopm": "71549",
    "tpm": "166405"
  },
  "6668700E61A903E283834363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:41:02",
    "activevu": "8",
    "nopm": "135774",
    "tpm": "316012"
  },
  "666871C161A903E203230333": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:48:17",
    "activevu": "12",
    "nopm": "196216",
    "tpm": "455867"
  },
  "6668737A61A903E283131383": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 08:55:38",
    "activevu": "16",
    "nopm": "258999",
    "tpm": "601697"
  },
  "6668753861A903E283437363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:03:04",
    "activevu": "20",
    "nopm": "310608",
    "tpm": "721658"
  },
  "666876FB61A903E293532383": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:10:35",
    "activevu": "24",
    "nopm": "358359",
    "tpm": "832467"
  },
  "666878C361A903E203836353": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:18:11",
    "activevu": "28",
    "nopm": "399416",
    "tpm": "928993"
  },
  "66687A8E61A903E213737353": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:25:50",
    "activevu": "32",
    "nopm": "438920",
    "tpm": "1018825"
  },
  "66687C5E61A903E263536313": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:33:34",
    "activevu": "36",
    "nopm": "471456",
    "tpm": "1095396"
  },
  "66687E2961A903E263635343": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:41:13",
    "activevu": "40",
    "nopm": "522600",
    "tpm": "1213621"
  },
  "66687FF761AA03E293130303": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:48:55",
    "activevu": "44",
    "nopm": "548843",
    "tpm": "1275641"
  },
  "666881CE61AA03E283137393": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 09:56:46",
    "activevu": "48",
    "nopm": "569173",
    "tpm": "1322266"
  },
  "666883A861AA03E293734373": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:04:40",
    "activevu": "52",
    "nopm": "591268",
    "tpm": "1373413"
  },
  "6668858561AA03E223038303": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:12:37",
    "activevu": "56",
    "nopm": "609316",
    "tpm": "1415349"
  },
  "6668877161AA03E243737363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:20:49",
    "activevu": "60",
    "nopm": "628853",
    "tpm": "1462056"
  },
  "6668896861AA03E253531393": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:29:12",
    "activevu": "64",
    "nopm": "642036",
    "tpm": "1491259"
  },
  "66688B6861AA03E203133313": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:37:44",
    "activevu": "68",
    "nopm": "656450",
    "tpm": "1524772"
  },
  "66688D7161AA03E293331363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:46:25",
    "activevu": "72",
    "nopm": "661515",
    "tpm": "1536318"
  },
  "66688F8361AA03E203331333": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 10:55:15",
    "activevu": "76",
    "nopm": "670828",
    "tpm": "1557467"
  },
  "6668919E61AA03E263931383": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:04:14",
    "activevu": "80",
    "nopm": "672855",
    "tpm": "1563938"
  },
  "666893BF61AA03E213631303": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:13:19",
    "activevu": "84",
    "nopm": "680334",
    "tpm": "1580944"
  },
  "666895E761AA03E253330363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:22:31",
    "activevu": "88",
    "nopm": "685096",
    "tpm": "1590859"
  },
  "6668981461AA03E213832303": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:31:48",
    "activevu": "92",
    "nopm": "680334",
    "tpm": "1580664"
  },
  "66689A4661AA03E213938343": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:41:10",
    "activevu": "96",
    "nopm": "679016",
    "tpm": "1577343"
  },
  "66689C7E61AA03E213730313": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 11:50:38",
    "activevu": "100",
    "nopm": "678694",
    "tpm": "1577827"
  },
  "66689EB561AA03E233739373": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 12:00:05",
    "activevu": "104",
    "nopm": "674706",
    "tpm": "1566707"
  },
  "6668A0F461AA03E243230353": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 12:09:40",
    "activevu": "108",
    "nopm": "670643",
    "tpm": "1557979"
  },
  "6668A33361AA03E293836323": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 12:19:15",
    "activevu": "112",
    "nopm": "665293",
    "tpm": "1545308"
  },
  "6668A57A61AA03E203433363": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 12:28:58",
    "activevu": "116",
    "nopm": "661495",
    "tpm": "1536360"
  },
  "6668A7CF61AA03E253833343": {
    "db": "MariaDB",
    "bm": "TPC-C",
    "tstamp": "2024-06-11 12:38:55",
    "activevu": "120",
    "nopm": "655745",
    "tpm": "1523566"
  }
}
sm-shaw commented 3 weeks ago

For completeness on functionality and to keep the description together in this issue, running the following command:

jobs profileid 0 (which is the default) means jobs are run as standalone jobs as per previous functionality and no performance profiles are found in the SQLite database the HammerDB uses for storage.

pp9

Also the functionality is available from the GUI in both the jobs dialog

pp3

and also the autopilot dialog where it is expected to be run from

pp4

After running an autopilot sequence e.g.

pp6

The jobs are grouped by profile id

pp7

and the performance profile graph shown by clicking on the profileid link

pp8