search

cloud-bulldozer / e2e-benchmarking

Performance Tests for end Platforms
Apache License 2.0
40 stars 72 forks source link

router-perf-v2 touchstone_compare Error: Issue capturing results from elasticsearch using config #353

Open qiliRedHat opened 2 years ago

qiliRedHat commented 2 years ago

When I run router-perf in Jenkins https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/e2e-benchmarking-multibranch-pipeline/job/router-perf/351/console. Got the error below.

touchstone_compare --database elasticsearch -url xxxx:443 -u xxxx --config /home/jenkins/ws/workspace/multibranch-pipeline_router-perf/workloads/router-perf-v2/mb-touchstone.json -o csv --output-file /home/jenkins/ws/workspace/multibranch-pipeline_router-perf/workloads/router-perf-v2/ingress-performance.csv --rc 0
touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'test_type': 'http'}, 'buckets': ['routes', 'conn_per_targetroute', 'keepalive'], 'aggregations': {'requests_per_second': ['avg'], 'latency_95pctl': ['avg']}}
touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'test_type': 'edge'}, 'buckets': ['routes', 'conn_per_targetroute', 'keepalive'], 'aggregations': {'requests_per_second': ['avg'], 'latency_95pctl': ['avg']}}
touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'test_type': 'passthrough'}, 'buckets': ['routes', 'conn_per_targetroute', 'keepalive'], 'aggregations': {'requests_per_second': ['avg'], 'latency_95pctl': ['avg']}}
touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'test_type': 'reencrypt'}, 'buckets': ['routes', 'conn_per_targetroute', 'keepalive'], 'aggregations': {'requests_per_second': ['avg'], 'latency_95pctl': ['avg']}}
qiliRedHat commented 2 years ago

I read https://github.com/cloud-bulldozer/e2e-benchmarking/pull/332 and thought my Jenkins job may need update to reflect the change in https://github.com/jaredoconnell/e2e-benchmarking/blob/8af52a9d0e3c66d9df3b3c07acb29e746a58b7b8/workloads/router-perf-v2/env.sh#L24-L31

rsevilla87 commented 2 years ago

Hi @qiliRedHat, if you pay attention to the log lines, they say that touchstone couldn't find any metric using those filters, it seems like it was able to find mix termination metrics but it's missing the http, edge, passthrough and reencrypt ones.

Also, I've seen that the selected termination was mix, the problem is that the touchstone configuration file in this repo is meant to be used with all terminations, we would need to include a new one for each termination

qiliRedHat commented 2 years ago

@rsevilla87 As you said with all TERMINATIONS specified, the problem didn't reproduce.

04-07 18:15:20.013  touchstone_compare --database elasticsearch -url https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443 -u b85440ee-bfe9-4983-a3bf-5b3ef978932f --config /home/jenkins/ws/workspace/multibranch-pipeline_router-perf/workloads/router-perf-v2/mb-touchstone.json -o csv --output-file /home/jenkins/ws/workspace/multibranch-pipeline_router-perf/workloads/router-perf-v2/ingress-performance.csv --rc 0
04-07 18:15:25.241  Thu Apr  7 10:15:24 UTC 2022 Installing requirements to generate spreadsheet