search

cloud-bulldozer / benchmark-wrapper

Python Library to run benchmarks
https://benchmark-wrapper.readthedocs.io
Apache License 2.0
19 stars 56 forks source link

Wrapper code for nighthawk workload #446

Closed vishnuchalla closed 1 year ago

vishnuchalla commented 2 years ago

Description

POC for adding nighthawk as a new workload. Workload Repo:https://github.com/envoyproxy/nighthawk

Relevant Sheet: https://docs.google.com/spreadsheets/d/19__erchn4qRKS4khnOM4LZoxT8VTbSwoq6YOLzLrYhY/edit#gid=476339394

Workload Sample Execution and Output

root@ip-10-0-137-160:/# nighthawk_client --concurrency 8 --duration 60 --connections 80 --max-requests-per-connection 50 --rps 1000000 http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com --output-format fortio
[19:31:42.213024][44][I] c-ares library initialized.
[19:31:42.213128][44][I] Starting 8 threads / event loops. Time limit: 60 seconds.
[19:31:42.213138][44][I] Global targets: 640 connections and 8000000 calls per second.
[19:31:42.213144][44][I]    (Per-worker targets: 80 connections and 1000000 calls per second)
[19:32:43.147964][63][I] Stopping after 59999 ms. Initiated: 16673 / Completed: 16672. (Completion rate was 277.87077915419815 per second.)
[19:32:43.148243][50][I] Stopping after 60000 ms. Initiated: 16669 / Completed: 16668. (Completion rate was 277.7997036803161 per second.)
[19:32:43.148703][54][I] Stopping after 59998 ms. Initiated: 16830 / Completed: 16829. (Completion rate was 280.4895602015698 per second.)
[19:32:43.149068][52][I] Stopping after 60001 ms. Initiated: 16603 / Completed: 16602. (Completion rate was 276.6947658573459 per second.)
[19:32:43.147964][55][I] Stopping after 59999 ms. Initiated: 16882 / Completed: 16881. (Completion rate was 281.3504829849958 per second.)
[19:32:43.149405][53][I] Stopping after 59984 ms. Initiated: 16764 / Completed: 16764. (Completion rate was 279.47415846794945 per second.)
[19:32:43.149868][57][I] Stopping after 60001 ms. Initiated: 16625 / Completed: 16625. (Completion rate was 277.07690976697523 per second.)
[19:32:43.150250][51][I] Stopping after 60002 ms. Initiated: 16562 / Completed: 16562. (Completion rate was 276.0227064591346 per second.)
{
 "Labels": "",
 "RequestedQPS": 8000000,
 "ActualQPS": 2216.3282709885111,
 "ActualDuration": 59998332260,
 "NumThreads": 640,
 "DurationHistogram": {
  "Count": "132971",
  "Data": [
   {
    "Start": 0.000961791,
    "End": 0.000961791,
    "Percent": 0,
    "Count": "1"
   },
   {
    "Start": 0.000961791,
    "End": 0.001523263,
    "Percent": 10,
    "Count": "13301"
   },
   {
    "Start": 0.001523263,
    "End": 0.001724223,
    "Percent": 20,
    "Count": "13293"
   },
   {
    "Start": 0.001724223,
    "End": 0.001933951,
    "Percent": 30,
    "Count": "13299"
   },
   {
    "Start": 0.001933951,
    "End": 0.002157311,
    "Percent": 40,
    "Count": "13299"
   },
   {
    "Start": 0.002157311,
    "End": 0.002409855,
    "Percent": 50,
    "Count": "13302"
   },
   {
    "Start": 0.002409855,
    "End": 0.002513279,
    "Percent": 55.000000000000007,
    "Count": "6644"
   },
   {
    "Start": 0.002513279,
    "End": 0.002646655,
    "Percent": 60,
    "Count": "6649"
   },
   {
    "Start": 0.002646655,
    "End": 0.002811519,
    "Percent": 65,
    "Count": "6646"
   },
   {
    "Start": 0.002811519,
    "End": 0.002990463,
    "Percent": 70,
    "Count": "6650"
   },
   {
    "Start": 0.002990463,
    "End": 0.003289983,
    "Percent": 75,
    "Count": "6645"
   },
   {
    "Start": 0.003289983,
    "End": 0.003502207,
    "Percent": 77.5,
    "Count": "3324"
   },
   {
    "Start": 0.003502207,
    "End": 0.003809279,
    "Percent": 80,
    "Count": "3325"
   },
   {
    "Start": 0.003809279,
    "End": 0.004266495,
    "Percent": 82.5,
    "Count": "3325"
   },
   {
    "Start": 0.004266495,
    "End": 0.004988415,
    "Percent": 85,
    "Count": "3323"
   },
   {
    "Start": 0.004988415,
    "End": 0.006108415,
    "Percent": 87.5,
    "Count": "3324"
   },
   {
    "Start": 0.006108415,
    "End": 0.006681087,
    "Percent": 88.75,
    "Count": "1662"
   },
   {
    "Start": 0.006681087,
    "End": 0.007223551,
    "Percent": 90,
    "Count": "1663"
   },
   {
    "Start": 0.007223551,
    "End": 0.007787007,
    "Percent": 91.25,
    "Count": "1662"
   },
   {
    "Start": 0.007787007,
    "End": 0.008299775,
    "Percent": 92.5,
    "Count": "1663"
   },
   {
    "Start": 0.008299775,
    "End": 0.008876543,
    "Percent": 93.75,
    "Count": "1661"
   },
   {
    "Start": 0.008876543,
    "End": 0.009168895,
    "Percent": 94.375,
    "Count": "831"
   },
   {
    "Start": 0.009168895,
    "End": 0.009506815,
    "Percent": 95,
    "Count": "832"
   },
   {
    "Start": 0.009506815,
    "End": 0.009908223,
    "Percent": 95.625,
    "Count": "830"
   },
   {
    "Start": 0.009908223,
    "End": 0.010358271,
    "Percent": 96.25,
    "Count": "833"
   },
   {
    "Start": 0.010358271,
    "End": 0.010967039,
    "Percent": 96.875,
    "Count": "829"
   },
   {
    "Start": 0.010967039,
    "End": 0.011298303,
    "Percent": 97.1875,
    "Count": "416"
   },
   {
    "Start": 0.011298303,
    "End": 0.011692543,
    "Percent": 97.5,
    "Count": "416"
   },
   {
    "Start": 0.011692543,
    "End": 0.012035071,
    "Percent": 97.8125,
    "Count": "415"
   },
   {
    "Start": 0.012035071,
    "End": 0.012531199,
    "Percent": 98.125,
    "Count": "416"
   },
   {
    "Start": 0.012531199,
    "End": 0.013052415,
    "Percent": 98.4375,
    "Count": "415"
   },
   {
    "Start": 0.013052415,
    "End": 0.013375487,
    "Percent": 98.59375,
    "Count": "208"
   },
   {
    "Start": 0.013375487,
    "End": 0.013698047,
    "Percent": 98.75,
    "Count": "208"
   },
   {
    "Start": 0.013698047,
    "End": 0.014095871,
    "Percent": 98.90625,
    "Count": "207"
   },
   {
    "Start": 0.014095871,
    "End": 0.014558207,
    "Percent": 99.0625,
    "Count": "208"
   },
   {
    "Start": 0.014558207,
    "End": 0.015131135,
    "Percent": 99.21875,
    "Count": "208"
   },
   {
    "Start": 0.015131135,
    "End": 0.015491071,
    "Percent": 99.296875,
    "Count": "104"
   },
   {
    "Start": 0.015491071,
    "End": 0.015969791,
    "Percent": 99.375,
    "Count": "103"
   },
   {
    "Start": 0.015969791,
    "End": 0.016360447,
    "Percent": 99.453125,
    "Count": "104"
   },
   {
    "Start": 0.016360447,
    "End": 0.016818175,
    "Percent": 99.53125,
    "Count": "104"
   },
   {
    "Start": 0.016818175,
    "End": 0.017369087,
    "Percent": 99.609375,
    "Count": "104"
   },
   {
    "Start": 0.017369087,
    "End": 0.017659903,
    "Percent": 99.6484375,
    "Count": "52"
   },
   {
    "Start": 0.017659903,
    "End": 0.018094079,
    "Percent": 99.6875,
    "Count": "52"
   },
   {
    "Start": 0.018094079,
    "End": 0.018557951,
    "Percent": 99.7265625,
    "Count": "52"
   },
   {
    "Start": 0.018557951,
    "End": 0.019201023,
    "Percent": 99.765625,
    "Count": "52"
   },
   {
    "Start": 0.019201023,
    "End": 0.019757055,
    "Percent": 99.8046875,
    "Count": "52"
   },
   {
    "Start": 0.019757055,
    "End": 0.020107263,
    "Percent": 99.82421875,
    "Count": "26"
   },
   {
    "Start": 0.020107263,
    "End": 0.020595711,
    "Percent": 99.84375,
    "Count": "26"
   },
   {
    "Start": 0.020595711,
    "End": 0.021017599,
    "Percent": 99.86328125,
    "Count": "26"
   },
   {
    "Start": 0.021017599,
    "End": 0.021704703,
    "Percent": 99.8828125,
    "Count": "26"
   },
   {
    "Start": 0.021704703,
    "End": 0.022589439,
    "Percent": 99.90234375,
    "Count": "26"
   },
   {
    "Start": 0.022589439,
    "End": 0.023239679,
    "Percent": 99.912109375,
    "Count": "13"
   },
   {
    "Start": 0.023239679,
    "End": 0.024006655,
    "Percent": 99.921875,
    "Count": "13"
   },
   {
    "Start": 0.024006655,
    "End": 0.024702975,
    "Percent": 99.931640625,
    "Count": "13"
   },
   {
    "Start": 0.024702975,
    "End": 0.025619455,
    "Percent": 99.94140625,
    "Count": "13"
   },
   {
    "Start": 0.025619455,
    "End": 0.026265599,
    "Percent": 99.951171875,
    "Count": "13"
   },
   {
    "Start": 0.026265599,
    "End": 0.026534911,
    "Percent": 99.9560546875,
    "Count": "6"
   },
   {
    "Start": 0.026534911,
    "End": 0.027142143,
    "Percent": 99.9609375,
    "Count": "7"
   },
   {
    "Start": 0.027142143,
    "End": 0.027958271,
    "Percent": 99.9658203125,
    "Count": "6"
   },
   {
    "Start": 0.027958271,
    "End": 0.030137343,
    "Percent": 99.970703125,
    "Count": "7"
   },
   {
    "Start": 0.030137343,
    "End": 0.030939135,
    "Percent": 99.9755859375,
    "Count": "6"
   },
   {
    "Start": 0.030939135,
    "End": 0.031284223,
    "Percent": 99.97802734375,
    "Count": "3"
   },
   {
    "Start": 0.031284223,
    "End": 0.032558079,
    "Percent": 99.98046875,
    "Count": "4"
   },
   {
    "Start": 0.032558079,
    "End": 0.032902143,
    "Percent": 99.98291015625,
    "Count": "3"
   },
   {
    "Start": 0.032902143,
    "End": 0.033282047,
    "Percent": 99.9853515625,
    "Count": "3"
   },
   {
    "Start": 0.033282047,
    "End": 0.034115583,
    "Percent": 99.98779296875,
    "Count": "3"
   },
   {
    "Start": 0.034115583,
    "End": 0.036368383,
    "Percent": 99.989013671875,
    "Count": "2"
   },
   {
    "Start": 0.036368383,
    "End": 0.036456447,
    "Percent": 99.990234375,
    "Count": "2"
   },
   {
    "Start": 0.036456447,
    "End": 0.037005311,
    "Percent": 99.991455078125,
    "Count": "1"
   },
   {
    "Start": 0.037005311,
    "End": 0.037130239,
    "Percent": 99.99267578125,
    "Count": "2"
   },
   {
    "Start": 0.037130239,
    "End": 0.037193727,
    "Percent": 99.993896484375,
    "Count": "1"
   },
   {
    "Start": 0.037193727,
    "End": 0.037388287,
    "Percent": 99.9945068359375,
    "Count": "1"
   },
   {
    "Start": 0.037388287,
    "End": 0.038311935,
    "Percent": 99.9951171875,
    "Count": "1"
   },
   {
    "Start": 0.038311935,
    "End": 0.041213951,
    "Percent": 99.9957275390625,
    "Count": "1"
   },
   {
    "Start": 0.041213951,
    "End": 0.042199039,
    "Percent": 99.996337890625,
    "Count": "1"
   },
   {
    "Start": 0.042199039,
    "End": 0.042199039,
    "Percent": 99.9969482421875,
    "Count": "0"
   },
   {
    "Start": 0.042199039,
    "End": 0.051259391,
    "Percent": 99.99725341796875,
    "Count": "1"
   },
   {
    "Start": 0.051259391,
    "End": 0.051259391,
    "Percent": 99.99755859375,
    "Count": "0"
   },
   {
    "Start": 0.051259391,
    "End": 0.053141503,
    "Percent": 99.99786376953125,
    "Count": "1"
   },
   {
    "Start": 0.053141503,
    "End": 0.053141503,
    "Percent": 99.9981689453125,
    "Count": "0"
   },
   {
    "Start": 0.053141503,
    "End": 0.053141503,
    "Percent": 99.99847412109375,
    "Count": "0"
   },
   {
    "Start": 0.053141503,
    "End": 0.053319679,
    "Percent": 99.998626708984375,
    "Count": "1"
   },
   {
    "Start": 0.053319679,
    "End": 0.053319679,
    "Percent": 99.998779296875,
    "Count": "0"
   },
   {
    "Start": 0.053319679,
    "End": 0.053319679,
    "Percent": 99.998931884765625,
    "Count": "0"
   },
   {
    "Start": 0.053319679,
    "End": 0.053319679,
    "Percent": 99.99908447265625,
    "Count": "0"
   },
   {
    "Start": 0.053319679,
    "End": 0.053319679,
    "Percent": 99.999237060546875,
    "Count": "0"
   },
   {
    "Start": 0.053319679,
    "End": 0.059467775,
    "Percent": 99.999313354492188,
    "Count": "1"
   },
   {
    "Start": 0.059467775,
    "End": 0.059467775,
    "Percent": 100,
    "Count": "0"
   }
  ],
  "Min": 0.000961791,
  "Max": 0.059467775,
  "Sum": 445.547126439,
  "Avg": 0.003350709,
  "StdDev": 0.002824306,
  "Percentiles": [
   {
    "Percentile": 50,
    "Value": 0.002409855
   },
   {
    "Percentile": 75,
    "Value": 0.003289983
   },
   {
    "Percentile": 80,
    "Value": 0.003809279
   },
   {
    "Percentile": 90,
    "Value": 0.007223551
   },
   {
    "Percentile": 95,
    "Value": 0.009506815
   },
   {
    "Percentile": 99,
    "Value": 0.014558207
   },
   {
    "Percentile": 99.9,
    "Value": 0.022589439
   }
  ]
 },
 "RetCodes": {
  "200": "132971"
 },
 "URL": "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com",
 "Version": "0.3.0",
 "Jitter": false,
 "RunType": "HTTP",
 "Sizes": {
  "Count": "132971",
  "Data": [],
  "Min": 128,
  "Max": 128,
  "Sum": 17020288,
  "Avg": 128,
  "StdDev": 0,
  "Percentiles": []
 },
 "HeaderSizes": {
  "Count": "132971",
  "Data": [],
  "Min": 289,
  "Max": 289,
  "Sum": 38428619,
  "Avg": 289,
  "StdDev": 0,
  "Percentiles": []
 },
 "BytesSent": "14494384",
 "BytesReceived": "61432602",
 "StartTime": "2022-08-15T19:31:42.247320975Z",
 "RequestedDuration": "60s"
}

Testing

Tested from command line and verified the results getting indexed on ES.

root@ip-10-0-137-160:/# export es=https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443; run_snafu -t nighthawk -s 3 --resourcetype pod -u testuid --user vchalla --url http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com --concurrency 8 --duration 5 --connections 80 --max-requests-per-connection 50 --rps 1000000 --create-archive --archive-file /tmp/nighthawk.archive
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: logging level is INFO
2022-08-18T02:34:59Z - INFO     - MainProcess - _load_benchmarks: Successfully imported 4 benchmark modules: coremarkpro, nighthawk, systemd_analyze, uperf
2022-08-18T02:34:59Z - INFO     - MainProcess - _load_benchmarks: Failed to import 0 benchmark modules: 
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: Using elasticsearch server with host: https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: Using index prefix for ES: snafu-nighthawk
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: Turning off TLS certificate verification
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: Connected to the elasticsearch cluster with info as follows:
2022-08-18T02:34:59Z - INFO     - MainProcess - run_snafu: {
    "name": "9ecb151f872c2ece36ad7a642a983be5",
    "cluster_name": "415909267177:perfscale-dev",
    "cluster_uuid": "Xz2IU4etSieAeaO2j-QCUw",
    "version": {
        "number": "7.10.2",
        "build_flavor": "oss",
        "build_type": "tar",
        "build_hash": "unknown",
        "build_date": "2022-07-20T07:43:57.819165Z",
        "build_snapshot": false,
        "lucene_version": "8.7.0",
        "minimum_wire_compatibility_version": "6.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1"
    },
    "tagline": "You Know, for Search"
}
2022-08-18T02:34:59Z - INFO     - MainProcess - py_es_bulk: Using streaming bulk indexer
2022-08-18T02:34:59Z - INFO     - MainProcess - wrapper_factory: identified nighthawk as the benchmark wrapper
2022-08-18T02:34:59Z - INFO     - MainProcess - _benchmark: Starting nighthawk wrapper.
2022-08-18T02:34:59Z - INFO     - MainProcess - _benchmark: Running setup tasks.
2022-08-18T02:34:59Z - INFO     - MainProcess - _benchmark: Collecting results from benchmark.
2022-08-18T02:34:59Z - INFO     - MainProcess - nighthawk: Collecting 3 samples of Nighthawk
2022-08-18T02:34:59Z - INFO     - MainProcess - nighthawk: Starting nighthawk sample 1 out of 3 with uuid testuid
2022-08-18T02:34:59Z - INFO     - MainProcess - nighthawk: nighthawk_client --concurrency 8 --duration 5 --connections 80 --max-requests-per-connection 4294937295 --rps 1000000 --output-format fortio http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com > nighthawk.json
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: Got sample result: BenchmarkResult(name='nighthawk', metadata={'user': 'vchalla', 'uuid': 'testuid'}, config={'concurrency': 8, 'duration': 5, 'connections': 80, 'max_requests_per_connection': 4294937295, 'rps': 1000000, 'kind': 'pod', 'url': 'http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'}, data={'workload': 'nighthawk', 'uuid': 'testuid', 'user': 'vchalla', 'cluster_name': 'mycluster', 'duration': 5, 'targets': ['http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'], 'concurrency': 8, 'connections': 80, 'max_requests_per_connection': 4294937295, 'hostname': 'ip-10-0-137-160', 'requested_qps': 8000000, 'throughput': 2538.4230306603, 'status_codes_1xx': 0, 'status_codes_2xx': 12692, 'status_codes_3xx': 0, 'status_codes_4xx': 0, 'status_codes_5xx': 0, 'p50_latency': 2.121855, 'p75_latency': 3.090687, 'p80_latency': 3.549055, 'p90_latency': 5.2065269999999995, 'p95_latency': 6.876671, 'p99_latency': 19.769343000000003, 'p99_9_latency': 33.908735, 'avg_latency': 3.038262, 'timestamp': '2022-08-18T02:34:59.629249652Z', 'bytes_in': 5863704.0, 'bytes_out': 1383755.0, 'iteration': 1}, labels={}, tag='results')
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: Finished executing nighthawk sample 1 out of 3
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: Starting nighthawk sample 2 out of 3 with uuid testuid
2022-08-18T02:35:32Z - INFO     - MainProcess - nighthawk: nighthawk_client --concurrency 8 --duration 5 --connections 80 --max-requests-per-connection 4294937295 --rps 1000000 --output-format fortio http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com > nighthawk.json
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: Got sample result: BenchmarkResult(name='nighthawk', metadata={'user': 'vchalla', 'uuid': 'testuid'}, config={'concurrency': 8, 'duration': 5, 'connections': 80, 'max_requests_per_connection': 4294937295, 'rps': 1000000, 'kind': 'pod', 'url': 'http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'}, data={'workload': 'nighthawk', 'uuid': 'testuid', 'user': 'vchalla', 'cluster_name': 'mycluster', 'duration': 5, 'targets': ['http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'], 'concurrency': 8, 'connections': 80, 'max_requests_per_connection': 4294937295, 'hostname': 'ip-10-0-137-160', 'requested_qps': 8000000, 'throughput': 2173.8396173900856, 'status_codes_1xx': 0, 'status_codes_2xx': 10864, 'status_codes_3xx': 0, 'status_codes_4xx': 0, 'status_codes_5xx': 0, 'p50_latency': 2.4620789999999997, 'p75_latency': 3.477631, 'p80_latency': 4.127743, 'p90_latency': 7.228927, 'p95_latency': 9.219071000000001, 'p99_latency': 13.818367, 'p99_9_latency': 19.082239, 'avg_latency': 3.469418, 'timestamp': '2022-08-18T02:35:32.922688493Z', 'bytes_in': 5019168.0, 'bytes_out': 1184939.0, 'iteration': 2}, labels={}, tag='results')
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: Finished executing nighthawk sample 2 out of 3
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: Starting nighthawk sample 3 out of 3 with uuid testuid
2022-08-18T02:36:08Z - INFO     - MainProcess - nighthawk: nighthawk_client --concurrency 8 --duration 5 --connections 80 --max-requests-per-connection 4294937295 --rps 1000000 --output-format fortio http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com > nighthawk.json
2022-08-18T02:36:31Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:36:31Z - INFO     - MainProcess - nighthawk: Got sample result: BenchmarkResult(name='nighthawk', metadata={'user': 'vchalla', 'uuid': 'testuid'}, config={'concurrency': 8, 'duration': 5, 'connections': 80, 'max_requests_per_connection': 4294937295, 'rps': 1000000, 'kind': 'pod', 'url': 'http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'}, data={'workload': 'nighthawk', 'uuid': 'testuid', 'user': 'vchalla', 'cluster_name': 'mycluster', 'duration': 5, 'targets': ['http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com'], 'concurrency': 8, 'connections': 80, 'max_requests_per_connection': 4294937295, 'hostname': 'ip-10-0-137-160', 'requested_qps': 8000000, 'throughput': 2181.002752194779, 'status_codes_1xx': 0, 'status_codes_2xx': 10909, 'status_codes_3xx': 0, 'status_codes_4xx': 0, 'status_codes_5xx': 0, 'p50_latency': 2.482943, 'p75_latency': 3.4507510000000003, 'p80_latency': 3.968895, 'p90_latency': 7.022847, 'p95_latency': 9.661951, 'p99_latency': 13.552639, 'p99_9_latency': 24.245247000000003, 'avg_latency': 3.476202, 'timestamp': '2022-08-18T02:36:09.056121405Z', 'bytes_in': 5039958.0, 'bytes_out': 1189190.0, 'iteration': 3}, labels={}, tag='results')
2022-08-18T02:36:31Z - INFO     - MainProcess - nighthawk: --------------------------------------------------
2022-08-18T02:36:31Z - INFO     - MainProcess - nighthawk: Finished executing nighthawk sample 3 out of 3
2022-08-18T02:36:31Z - INFO     - MainProcess - nighthawk: Successfully collected 3 samples of nighthawk.
2022-08-18T02:36:31Z - INFO     - MainProcess - _benchmark: Cleaning up
2022-08-18T02:36:32Z - INFO     - MainProcess - run_snafu: Indexed results - 3 success, 0 duplicates, 0 failures, with 0 retries.
2022-08-18T02:36:32Z - INFO     - MainProcess - run_snafu: Duration of execution - 0:01:33, with total size of 696 bytes

3 Sample execution es documents showing consistency:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "snafu-nighthawk-results",
        "_type" : "_doc",
        "_id" : "be14b4f1018d8954c370c1bf200f8229ca5e0db921d1a89abe7f629956997fca",
        "_score" : 1.0,
        "_source" : {
          "concurrency" : 8,
          "duration" : 5,
          "connections" : 80,
          "max_requests_per_connection" : 4294937295,
          "rps" : 1000000,
          "kind" : "pod",
          "url" : "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com",
          "workload" : "nighthawk",
          "uuid" : "testuid",
          "user" : "vchalla",
          "cluster_name" : "mycluster",
          "targets" : [
            "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com"
          ],
          "hostname" : "ip-10-0-137-160",
          "requested_qps" : 8000000,
          "throughput" : 2181.002752194779,
          "status_codes_1xx" : 0,
          "status_codes_2xx" : 10909,
          "status_codes_3xx" : 0,
          "status_codes_4xx" : 0,
          "status_codes_5xx" : 0,
          "p50_latency" : 2.482943,
          "p75_latency" : 3.4507510000000003,
          "p80_latency" : 3.968895,
          "p90_latency" : 7.022847,
          "p95_latency" : 9.661951,
          "p99_latency" : 13.552639,
          "p99_9_latency" : 24.245247000000003,
          "avg_latency" : 3.476202,
          "timestamp" : "2022-08-18T02:36:09.056121405Z",
          "bytes_in" : 5039958.0,
          "bytes_out" : 1189190.0,
          "iteration" : 3,
          "run_id" : "NA"
        }
      },
      {
        "_index" : "snafu-nighthawk-results",
        "_type" : "_doc",
        "_id" : "a6166fba0855e0119f7de52e89f6b7dd1cf78f36104d0ff21af1f5fd778243e1",
        "_score" : 1.0,
        "_source" : {
          "concurrency" : 8,
          "duration" : 5,
          "connections" : 80,
          "max_requests_per_connection" : 4294937295,
          "rps" : 1000000,
          "kind" : "pod",
          "url" : "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com",
          "workload" : "nighthawk",
          "uuid" : "testuid",
          "user" : "vchalla",
          "cluster_name" : "mycluster",
          "targets" : [
            "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com"
          ],
          "hostname" : "ip-10-0-137-160",
          "requested_qps" : 8000000,
          "throughput" : 2538.4230306603,
          "status_codes_1xx" : 0,
          "status_codes_2xx" : 12692,
          "status_codes_3xx" : 0,
          "status_codes_4xx" : 0,
          "status_codes_5xx" : 0,
          "p50_latency" : 2.121855,
          "p75_latency" : 3.090687,
          "p80_latency" : 3.549055,
          "p90_latency" : 5.2065269999999995,
          "p95_latency" : 6.876671,
          "p99_latency" : 19.769343000000003,
          "p99_9_latency" : 33.908735,
          "avg_latency" : 3.038262,
          "timestamp" : "2022-08-18T02:34:59.629249652Z",
          "bytes_in" : 5863704.0,
          "bytes_out" : 1383755.0,
          "iteration" : 1,
          "run_id" : "NA"
        }
      },
      {
        "_index" : "snafu-nighthawk-results",
        "_type" : "_doc",
        "_id" : "0004fe0bda7118f144d9b94cedfc5958e6645933c79cd2537258967e9ee6a6a8",
        "_score" : 1.0,
        "_source" : {
          "concurrency" : 8,
          "duration" : 5,
          "connections" : 80,
          "max_requests_per_connection" : 4294937295,
          "rps" : 1000000,
          "kind" : "pod",
          "url" : "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com",
          "workload" : "nighthawk",
          "uuid" : "testuid",
          "user" : "vchalla",
          "cluster_name" : "mycluster",
          "targets" : [
            "http://http-perf-vchalla-test.apps.vchalla-perfscale.perfscale.devcluster.openshift.com"
          ],
          "hostname" : "ip-10-0-137-160",
          "requested_qps" : 8000000,
          "throughput" : 2173.8396173900856,
          "status_codes_1xx" : 0,
          "status_codes_2xx" : 10864,
          "status_codes_3xx" : 0,
          "status_codes_4xx" : 0,
          "status_codes_5xx" : 0,
          "p50_latency" : 2.4620789999999997,
          "p75_latency" : 3.477631,
          "p80_latency" : 4.127743,
          "p90_latency" : 7.228927,
          "p95_latency" : 9.219071000000001,
          "p99_latency" : 13.818367,
          "p99_9_latency" : 19.082239,
          "avg_latency" : 3.469418,
          "timestamp" : "2022-08-18T02:35:32.922688493Z",
          "bytes_in" : 5019168.0,
          "bytes_out" : 1184939.0,
          "iteration" : 2,
          "run_id" : "NA"
        }
      }
    ]
  }
}
comet-perf-ci commented 2 years ago

Can one of the admins verify this patch?

codecov-commenter commented 2 years ago

Codecov Report

Merging #446 (739bac9) into master (c8372b5) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #446   +/-   ##
=======================================
  Coverage   39.35%   39.35%           
=======================================
  Files          15       15           
  Lines         869      869           
=======================================
  Hits          342      342           
  Misses        527      527
Flag Coverage Δ
gha 39.35% <ø> (ø)
python-3.6 39.35% <ø> (ø)
unit 39.35% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

dry923 commented 1 year ago

@rsevilla87 @jtaleric @mukrishn PTAL when you have a chance