opensearch-project / opensearch-dashboards-functional-test

Maintains functional tests for OpenSearch Dashboards and Dashboards plugins
Apache License 2.0
12 stars 112 forks source link

[FEATURE] make test result developer friendly and actionable #421

Open seraphjiang opened 1 year ago

seraphjiang commented 1 year ago

Is your feature request related to a problem?

The current test result - output of jenkins test run has detail error when it fail. However, there is no context around error/fail log, and no action could be take without deep knowledge about functional test and infra's integration test framework

e.g. one failed test run https://build.ci.opensearch.org/job/integ-test-opensearch-dashboards/1906/console

partial error log., it is hard to tell what test is failed, what's context, and where is related code developer should look into to troubleshoot

2022-12-08 02:52:37 ERROR    | functionalTestDashboards | with-security        | FAIL  |    1 |
2022-12-08 02:52:37 ERROR    | functionalTestDashboards | without-security     | FAIL  |    1 |
/usr/share/opensearch/.local/share/virtualenvs/integ-test-opensearch-dashboards_2-NAv4layC/lib/python3.7/site-packages/urllib3/connectionpool.py:1052: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,

expected

tianleh commented 1 year ago

This is about the text output from Cypress.