[Security Plugin] [2.x] Flaky behavior during `inaccessible_tenancy_features.js` test

peternied commented 5 months ago

When the test Test Dashboards tenancy features should not be accessible is running the expected UX indicating that multi-tenancy is not enabled appears [1] , but then quickly disappears showing a list of tenants [2]. I was only able to spot the error message in 2 frames of the record video [3].

It seems this test could pass depending on if cypresses catches this scenario; however it seems incorrect that the test then shows tenants appearing. This could be invalid cluster state or maybe there has been an issue with the test for a while as updates have been made.

Acceptance criteria

[ ] Determine that there is no bug in the security dashboard plugin with the multitenancy feature showing its disabled state.
[ ] Find the root cause of the test behavior - without an action on the the browser the list of tenants should not be shown
[ ] Update the test to pass 100% of the time

havior to be deterministic

Additional Context

The workflow failure was recorded on https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/8843007404?pr=1242

Screenshots

[1]
[2]

Test Run Video

[3] https://github.com/opensearch-project/opensearch-dashboards-functional-test/assets/2754967/48a4bd68-b2f5-488c-b3c7-02d216a7159a

Test report

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.0000" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2024-04-26T04:35:58" tests="0" file="cypress/integration/plugins/security-dashboards-plugin/inaccessible_tenancy_features.js" time="0.0000" failures="0">
  </testsuite>
  <testsuite name="Multi Tenancy Tests: " timestamp="2024-04-26T04:35:58" tests="1" time="0.0000" failures="1">
    <testcase name="Multi Tenancy Tests:  Test Dashboards tenancy features should not be accessible " time="0.0000" classname="Test Dashboards tenancy features should not be accessible ">
      <failure message="Timed out retrying after 60000ms: Expected to find content: &apos;You have not enabled multi tenancy&apos; but never did." type="AssertionError"><![CDATA[AssertionError: Timed out retrying after 60000ms: Expected to find content: 'You have not enabled multi tenancy' but never did.
    at Context.eval (http://localhost:5601/__cypress/tests?p=cypress/integration/plugins/security-dashboards-plugin/inaccessible_tenancy_features.js:114:57)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

dblock commented 3 months ago

@peternied Did the linked PR resolve this?

Catch All Triage - 1 2 3 4 5 6

peternied commented 3 months ago

Nope, this is still an outstanding issue in this repo.

opensearch-project / opensearch-dashboards-functional-test