red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
109 stars 166 forks source link

Execution fails before ODF installation step with StaleElementReferenceException on running RDR MultiCluster Job #9622

Closed keemano closed 1 month ago

keemano commented 5 months ago

Exception E selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found in the current frame 2024-04-03 12:48:52 E (Session info: chrome-headless-shell=123.0.6312.105) 2024-04-03 12:48:52 2024-04-03 12:48:52 venv/lib64/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py:242: StaleElementReferenceException

Job details - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-rdr-setup/658/console

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs.

sidhant-agrawal commented 1 month ago

Issue reproduced here: https://url.corp.redhat.com/fd4c08b

 01:12:38 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on Create cluster set
 01:12:38 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Send Cluster set name 'clusterset-submariner-61421bbbe15f4b2ebf'
 01:12:38 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on Create
 01:12:41 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on Manage resource assignments
 01:12:43 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Search and select cluster 'jnk-918-p'
 01:12:45 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Clear search by clicking on cross mark
 01:12:45 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Search and select cluster 'jnk-918-s'
 01:12:48 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Clear search by clicking on cross mark [2]
 01:12:48 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on 'Review'
 01:12:50 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on 'Save' to confirm the changes
 01:12:55 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on 'Submariner add-ons' tab
 01:12:55 - MainThread - ocs_ci.ocs.acm.acm - INFO - C[jnk-918-h] - Click on 'Install Submariner add-ons' button
 01:12:56 - MainThread - ocs_ci.framework.pytest_customization.reports - INFO - C[jnk-918-h] - duration reported by tests/functional/deployment/test_deployment.py::test_deployment immediately after test execution: 486.02
 ERROR
 ______________________ ERROR at setup of test_deployment _______________________

 request = <SubRequest 'cluster' for <Function test_deployment>>
 log_cli_level = 'INFO'
 record_testsuite_property = <bound method LogXML.add_global_property of <_pytest.junitxml.LogXML object at 0x7f9ca8f0e1f0>>
 set_live_must_gather_images = None

     @pytest.fixture(scope="session", autouse=True)
     def cluster(
         request, log_cli_level, record_testsuite_property, set_live_must_gather_images
     ):
         """
         This fixture initiates deployment for both OCP and OCS clusters.
         Specific platform deployment classes will handle the fine details
         of action
         """
         log.info(f"All logs located at {ocsci_log_path()}")

         teardown = ocsci_config.RUN["cli_params"]["teardown"]
         deploy = ocsci_config.RUN["cli_params"]["deploy"]
         if teardown or deploy:
             factory = dep_factory.DeploymentFactory()
             deployer = factory.get_deployment()

         # Add a finalizer to teardown the cluster after test execution is finished
         if teardown:

             def cluster_teardown_finalizer():
                 # If KMS is configured, clean up the backend resources
                 # we are doing it before OCP cleanup
                 if ocsci_config.DEPLOYMENT.get("kms_deployment"):
                     try:
                         kms = KMS.get_kms_deployment()
                         kms.cleanup()
                     except Exception as ex:
                         log.error(f"Failed to cleanup KMS. Exception is: {ex}")
                 deployer.destroy_cluster(log_cli_level)

             request.addfinalizer(cluster_teardown_finalizer)
             log.info("Will teardown cluster because --teardown was provided")

         # Download client
         if ocsci_config.DEPLOYMENT["skip_download_client"]:
             log.info("Skipping client download")
         else:
             force_download = (
                 ocsci_config.RUN["cli_params"].get("deploy")
                 and ocsci_config.DEPLOYMENT["force_download_client"]
             )
             get_openshift_client(force_download=force_download)

         # set environment variable for early testing of RHCOS
         if ocsci_config.ENV_DATA.get("early_testing"):
             release_img = ocsci_config.ENV_DATA["RELEASE_IMG"]
             log.info(f"Running early testing of RHCOS with release image: {release_img}")
             os.environ["RELEASE_IMG"] = release_img
             os.environ["OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE"] = release_img

         if deploy:
             # Deploy cluster
 >           deployer.deploy_cluster(log_cli_level)

 tests/conftest.py:1796: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 ocs_ci/deployment/deployment.py:652: in deploy_cluster
     self.do_deploy_submariner()
 ocs_ci/deployment/deployment.py:234: in do_deploy_submariner
     submariner.deploy()
 ocs_ci/deployment/acm.py:91: in deploy
     self.deploy_downstream()
 ocs_ci/deployment/acm.py:113: in deploy_downstream
     acm_obj.install_submariner_ui()
 ocs_ci/ocs/acm/acm.py:237: in install_submariner_ui
     self.do_click(self.page_nav["install-submariner-btn"], timeout=120)
 ocs_ci/ocs/ui/base_ui.py:216: in do_click
     _do_click(locator, timeout, enable_screenshot, copy_dom)
 ocs_ci/ocs/ui/base_ui.py:206: in _do_click
     element.click()
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webelement.py:80: in click
     self._execute(Command.CLICK_ELEMENT)
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webelement.py:633: in _execute
     return self._parent.execute(command, params)
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webdriver.py:321: in execute
     self.error_handler.check_response(response)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

 self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7f9c9fb23970>
 response = {'status': 404, 'value': '{"value":{"error":"stale element reference","message":"stale element reference: stale elemen...wn>\\n#20 0x5628f15077f0 \\u003Cunknown>\\n#21 0x5628f1516d8c \\u003Cunknown>\\n#22 0x7f68ff489c02 start_thread\\n"}}'}

     def check_response(self, response):
         """
         Checks that a JSON response from the WebDriver does not have an error.

         :Args:
          - response - The JSON response from the WebDriver server as a dictionary
            object.

         :Raises: If the response contains an error message.
         """
         status = response.get('status', None)
         if status is None or status == ErrorCode.SUCCESS:
             return
         value = None
         message = response.get("message", "")
         screen = response.get("screen", "")
         stacktrace = None
         if isinstance(status, int):
             value_json = response.get('value', None)
             if value_json and isinstance(value_json, basestring):
                 import json
                 try:
                     value = json.loads(value_json)
                     if len(value.keys()) == 1:
                         value = value['value']
                     status = value.get('error', None)
                     if status is None:
                         status = value["status"]
                         message = value["value"]
                         if not isinstance(message, basestring):
                             value = message
                             message = message.get('message')
                     else:
                         message = value.get('message', None)
                 except ValueError:
                     pass

         exception_class = ErrorInResponseException
         if status in ErrorCode.NO_SUCH_ELEMENT:
             exception_class = NoSuchElementException
         elif status in ErrorCode.NO_SUCH_FRAME:
             exception_class = NoSuchFrameException
         elif status in ErrorCode.NO_SUCH_WINDOW:
             exception_class = NoSuchWindowException
         elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
             exception_class = StaleElementReferenceException
         elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
             exception_class = ElementNotVisibleException
         elif status in ErrorCode.INVALID_ELEMENT_STATE:
             exception_class = InvalidElementStateException
         elif status in ErrorCode.INVALID_SELECTOR \
                 or status in ErrorCode.INVALID_XPATH_SELECTOR \
                 or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
             exception_class = InvalidSelectorException
         elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
             exception_class = ElementNotSelectableException
         elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
             exception_class = ElementNotInteractableException
         elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
             exception_class = InvalidCookieDomainException
         elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
             exception_class = UnableToSetCookieException
         elif status in ErrorCode.TIMEOUT:
             exception_class = TimeoutException
         elif status in ErrorCode.SCRIPT_TIMEOUT:
             exception_class = TimeoutException
         elif status in ErrorCode.UNKNOWN_ERROR:
             exception_class = WebDriverException
         elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
             exception_class = UnexpectedAlertPresentException
         elif status in ErrorCode.NO_ALERT_OPEN:
             exception_class = NoAlertPresentException
         elif status in ErrorCode.IME_NOT_AVAILABLE:
             exception_class = ImeNotAvailableException
         elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
             exception_class = ImeActivationFailedException
         elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
             exception_class = MoveTargetOutOfBoundsException
         elif status in ErrorCode.JAVASCRIPT_ERROR:
             exception_class = JavascriptException
         elif status in ErrorCode.SESSION_NOT_CREATED:
             exception_class = SessionNotCreatedException
         elif status in ErrorCode.INVALID_ARGUMENT:
             exception_class = InvalidArgumentException
         elif status in ErrorCode.NO_SUCH_COOKIE:
             exception_class = NoSuchCookieException
         elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
             exception_class = ScreenshotException
         elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
             exception_class = ElementClickInterceptedException
         elif status in ErrorCode.INSECURE_CERTIFICATE:
             exception_class = InsecureCertificateException
         elif status in ErrorCode.INVALID_COORDINATES:
             exception_class = InvalidCoordinatesException
         elif status in ErrorCode.INVALID_SESSION_ID:
             exception_class = InvalidSessionIdException
         elif status in ErrorCode.UNKNOWN_METHOD:
             exception_class = UnknownMethodException
         else:
             exception_class = WebDriverException
         if value == '' or value is None:
             value = response['value']
         if isinstance(value, basestring):
             if exception_class == ErrorInResponseException:
                 raise exception_class(response, value)
             raise exception_class(value)
         if message == "" and 'message' in value:
             message = value['message']

         screen = None
         if 'screen' in value:
             screen = value['screen']

         stacktrace = None
         if 'stackTrace' in value and value['stackTrace']:
             stacktrace = []
             try:
                 for frame in value['stackTrace']:
                     line = self._value_or_default(frame, 'lineNumber', '')
                     file = self._value_or_default(frame, 'fileName', '<anonymous>')
                     if line:
                         file = "%s:%s" % (file, line)
                     meth = self._value_or_default(frame, 'methodName', '<anonymous>')
                     if 'className' in frame:
                         meth = "%s.%s" % (frame['className'], meth)
                     msg = "    at %s (%s)"
                     msg = msg % (meth, file)
                     stacktrace.append(msg)
             except TypeError:
                 pass
         if exception_class == ErrorInResponseException:
             raise exception_class(response, message)
         elif exception_class == UnexpectedAlertPresentException:
             alert_text = None
             if 'data' in value:
                 alert_text = value['data'].get('text')
             elif 'alert' in value:
                 alert_text = value['alert'].get('text')
             raise exception_class(message, screen, stacktrace, alert_text)
 >       raise exception_class(message, screen, stacktrace)
 E       selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found in the current frame
 E         (Session info: chrome-headless-shell=126.0.6478.182)

 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py:242: StaleElementReferenceException
sidhant-agrawal commented 1 month ago

Observed StaleElementReferenceException during deployment when using PR #9646 : https://url.corp.redhat.com/e4cca64

 ______________________ ERROR at setup of test_deployment _______________________

 request = <SubRequest 'cluster' for <Function test_deployment>>
 log_cli_level = 'INFO'
 record_testsuite_property = <bound method LogXML.add_global_property of <_pytest.junitxml.LogXML object at 0x7fa7e8f97190>>
 set_live_must_gather_images = None

     @pytest.fixture(scope="session", autouse=True)
     def cluster(
         request, log_cli_level, record_testsuite_property, set_live_must_gather_images
     ):
         """
         This fixture initiates deployment for both OCP and OCS clusters.
         Specific platform deployment classes will handle the fine details
         of action
         """
         log.info(f"All logs located at {ocsci_log_path()}")

         teardown = ocsci_config.RUN["cli_params"]["teardown"]
         deploy = ocsci_config.RUN["cli_params"]["deploy"]
         if teardown or deploy:
             factory = dep_factory.DeploymentFactory()
             deployer = factory.get_deployment()

         # Add a finalizer to teardown the cluster after test execution is finished
         if teardown:

             def cluster_teardown_finalizer():
                 # If KMS is configured, clean up the backend resources
                 # we are doing it before OCP cleanup
                 if ocsci_config.DEPLOYMENT.get("kms_deployment"):
                     try:
                         kms = KMS.get_kms_deployment()
                         kms.cleanup()
                     except Exception as ex:
                         log.error(f"Failed to cleanup KMS. Exception is: {ex}")
                 deployer.destroy_cluster(log_cli_level)

             request.addfinalizer(cluster_teardown_finalizer)
             log.info("Will teardown cluster because --teardown was provided")

         # Download client
         if ocsci_config.DEPLOYMENT["skip_download_client"]:
             log.info("Skipping client download")
         else:
             force_download = (
                 ocsci_config.RUN["cli_params"].get("deploy")
                 and ocsci_config.DEPLOYMENT["force_download_client"]
             )
             get_openshift_client(force_download=force_download)

         # set environment variable for early testing of RHCOS
         if ocsci_config.ENV_DATA.get("early_testing"):
             release_img = ocsci_config.ENV_DATA["RELEASE_IMG"]
             log.info(f"Running early testing of RHCOS with release image: {release_img}")
             os.environ["RELEASE_IMG"] = release_img
             os.environ["OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE"] = release_img

         if deploy:
             # Deploy cluster
 >           deployer.deploy_cluster(log_cli_level)

 tests/conftest.py:1801: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 ocs_ci/deployment/deployment.py:668: in deploy_cluster
     self.do_deploy_submariner()
 ocs_ci/deployment/deployment.py:236: in do_deploy_submariner
     submariner.deploy()
 ocs_ci/deployment/acm.py:91: in deploy
     self.deploy_downstream()
 ocs_ci/deployment/acm.py:113: in deploy_downstream
     acm_obj.install_submariner_ui()
 ocs_ci/ocs/acm/acm.py:202: in install_submariner_ui
     self.navigate_clusters_page()
 ocs_ci/ocs/ui/acm_ui.py:85: in navigate_clusters_page
     self.do_click(locator=self.acm_page_nav["Clusters_page"], timeout=timeout)
 ocs_ci/ocs/ui/base_ui.py:216: in do_click
     _do_click(locator, timeout, enable_screenshot, copy_dom)
 ocs_ci/ocs/ui/base_ui.py:206: in _do_click
     element.click()
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webelement.py:80: in click
     self._execute(Command.CLICK_ELEMENT)
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webelement.py:633: in _execute
     return self._parent.execute(command, params)
 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/webdriver.py:321: in execute
     self.error_handler.check_response(response)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

 self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7fa7de22d670>
 response = {'status': 404, 'value': '{"value":{"error":"stale element reference","message":"stale element reference: stale elemen...wn>\\n#20 0x55fc570c9090 \\u003Cunknown>\\n#21 0x55fc570d84ec \\u003Cunknown>\\n#22 0x7fb199489c02 start_thread\\n"}}'}

     def check_response(self, response):
         """
         Checks that a JSON response from the WebDriver does not have an error.

         :Args:
          - response - The JSON response from the WebDriver server as a dictionary
            object.

         :Raises: If the response contains an error message.
         """
         status = response.get('status', None)
         if status is None or status == ErrorCode.SUCCESS:
             return
         value = None
         message = response.get("message", "")
         screen = response.get("screen", "")
         stacktrace = None
         if isinstance(status, int):
             value_json = response.get('value', None)
             if value_json and isinstance(value_json, basestring):
                 import json
                 try:
                     value = json.loads(value_json)
                     if len(value.keys()) == 1:
                         value = value['value']
                     status = value.get('error', None)
                     if status is None:
                         status = value["status"]
                         message = value["value"]
                         if not isinstance(message, basestring):
                             value = message
                             message = message.get('message')
                     else:
                         message = value.get('message', None)
                 except ValueError:
                     pass

         exception_class = ErrorInResponseException
         if status in ErrorCode.NO_SUCH_ELEMENT:
             exception_class = NoSuchElementException
         elif status in ErrorCode.NO_SUCH_FRAME:
             exception_class = NoSuchFrameException
         elif status in ErrorCode.NO_SUCH_WINDOW:
             exception_class = NoSuchWindowException
         elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
             exception_class = StaleElementReferenceException
         elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
             exception_class = ElementNotVisibleException
         elif status in ErrorCode.INVALID_ELEMENT_STATE:
             exception_class = InvalidElementStateException
         elif status in ErrorCode.INVALID_SELECTOR \
                 or status in ErrorCode.INVALID_XPATH_SELECTOR \
                 or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
             exception_class = InvalidSelectorException
         elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
             exception_class = ElementNotSelectableException
         elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
             exception_class = ElementNotInteractableException
         elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
             exception_class = InvalidCookieDomainException
         elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
             exception_class = UnableToSetCookieException
         elif status in ErrorCode.TIMEOUT:
             exception_class = TimeoutException
         elif status in ErrorCode.SCRIPT_TIMEOUT:
             exception_class = TimeoutException
         elif status in ErrorCode.UNKNOWN_ERROR:
             exception_class = WebDriverException
         elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
             exception_class = UnexpectedAlertPresentException
         elif status in ErrorCode.NO_ALERT_OPEN:
             exception_class = NoAlertPresentException
         elif status in ErrorCode.IME_NOT_AVAILABLE:
             exception_class = ImeNotAvailableException
         elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
             exception_class = ImeActivationFailedException
         elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
             exception_class = MoveTargetOutOfBoundsException
         elif status in ErrorCode.JAVASCRIPT_ERROR:
             exception_class = JavascriptException
         elif status in ErrorCode.SESSION_NOT_CREATED:
             exception_class = SessionNotCreatedException
         elif status in ErrorCode.INVALID_ARGUMENT:
             exception_class = InvalidArgumentException
         elif status in ErrorCode.NO_SUCH_COOKIE:
             exception_class = NoSuchCookieException
         elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
             exception_class = ScreenshotException
         elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
             exception_class = ElementClickInterceptedException
         elif status in ErrorCode.INSECURE_CERTIFICATE:
             exception_class = InsecureCertificateException
         elif status in ErrorCode.INVALID_COORDINATES:
             exception_class = InvalidCoordinatesException
         elif status in ErrorCode.INVALID_SESSION_ID:
             exception_class = InvalidSessionIdException
         elif status in ErrorCode.UNKNOWN_METHOD:
             exception_class = UnknownMethodException
         else:
             exception_class = WebDriverException
         if value == '' or value is None:
             value = response['value']
         if isinstance(value, basestring):
             if exception_class == ErrorInResponseException:
                 raise exception_class(response, value)
             raise exception_class(value)
         if message == "" and 'message' in value:
             message = value['message']

         screen = None
         if 'screen' in value:
             screen = value['screen']

         stacktrace = None
         if 'stackTrace' in value and value['stackTrace']:
             stacktrace = []
             try:
                 for frame in value['stackTrace']:
                     line = self._value_or_default(frame, 'lineNumber', '')
                     file = self._value_or_default(frame, 'fileName', '<anonymous>')
                     if line:
                         file = "%s:%s" % (file, line)
                     meth = self._value_or_default(frame, 'methodName', '<anonymous>')
                     if 'className' in frame:
                         meth = "%s.%s" % (frame['className'], meth)
                     msg = "    at %s (%s)"
                     msg = msg % (meth, file)
                     stacktrace.append(msg)
             except TypeError:
                 pass
         if exception_class == ErrorInResponseException:
             raise exception_class(response, message)
         elif exception_class == UnexpectedAlertPresentException:
             alert_text = None
             if 'data' in value:
                 alert_text = value['data'].get('text')
             elif 'alert' in value:
                 alert_text = value['alert'].get('text')
             raise exception_class(message, screen, stacktrace, alert_text)
 >       raise exception_class(message, screen, stacktrace)
 E       selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found in the current frame
 E         (Session info: chrome=127.0.6533.88)

 venv/lib64/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py:242: StaleElementReferenceException
am-agrawa commented 1 month ago

[RDR] Enable ACM observability for DR monitoring dashboard on RHACM console #9646

It's failing at a different place this time, should be fixed by this commit.