The current test result - output of jenkins test run has detail error when it fail. However, there is no context around error/fail log, and no action could be take without deep knowledge about functional test and infra's integration test framework
partial error log., it is hard to tell what test is failed, what's context, and where is related code developer should look into to troubleshoot
2022-12-08 02:52:37 ERROR | functionalTestDashboards | with-security | FAIL | 1 |
2022-12-08 02:52:37 ERROR | functionalTestDashboards | without-security | FAIL | 1 |
/usr/share/opensearch/.local/share/virtualenvs/integ-test-opensearch-dashboards_2-NAv4layC/lib/python3.7/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
expected
test runner should be able to collect print, test run/config context information when throw error
add runbook to interpret the typical error, so it is actionable for developer when they got failed test run. they could quickly know it is real regression issue, or test environment issue, or config issue, or test code issue.
Is your feature request related to a problem?
The current test result - output of jenkins test run has detail error when it fail. However, there is no context around error/fail log, and no action could be take without deep knowledge about functional test and infra's integration test framework
e.g. one failed test run https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/1906/console
partial error log., it is hard to tell what test is failed, what's context, and where is related code developer should look into to troubleshoot
expected