search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
109 stars 166 forks source link

pass storage namespace to all odf-cli commands that we run in automation #10446

Open DanielOsypenko opened 4 weeks ago

DanielOsypenko commented 4 weeks ago

test test_pvc_stale_volume_cleanup_cli fails, and all other tests will fail on managed platforms. openshift-storage namespace is not allowed on ROSA clusters because of "openshift" prefix in name

2024-09-06T11:19:23.149Z] 07:19:22 - MainThread - ocs_ci.framework.pytest_customization.reports - INFO  - duration reported by tests/functional/odf-cli/test_pvc_stale_volume_cleanup_cli.py::TestSubvolumesCommand::test_pvc_stale_volume_cleanup_cli immediately after test execution: 9.88
[2024-09-06T11:19:23.403Z] FAILED
[2024-09-06T11:19:23.403Z] ___________ TestSubvolumesCommand.test_pvc_stale_volume_cleanup_cli ____________
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z] self = <test_pvc_stale_volume_cleanup_cli.TestSubvolumesCommand object at 0x7f612d213910>
[2024-09-06T11:19:23.403Z] storageclass_factory = <function storageclass_factory_fixture.<locals>.factory at 0x7f60fb147c10>
[2024-09-06T11:19:23.403Z] pvc_factory = <function pvc_factory_fixture.<locals>.factory at 0x7f60fb147dc0>
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z]     @skipif_ocs_version("<4.15")
[2024-09-06T11:19:23.403Z]     @pytest.mark.polarion_id("OCS-5794")
[2024-09-06T11:19:23.403Z]     def test_pvc_stale_volume_cleanup_cli(self, storageclass_factory, pvc_factory):
[2024-09-06T11:19:23.403Z]         """
[2024-09-06T11:19:23.403Z]         1. Create a new PVC with Retain strategy.
[2024-09-06T11:19:23.403Z]         2. Delete the PVC
[2024-09-06T11:19:23.403Z]         3. Check for stale volumes
[2024-09-06T11:19:23.403Z]         4. Run the odf cli.
[2024-09-06T11:19:23.403Z]         5. Check for stale volumes
[2024-09-06T11:19:23.403Z]         6. No stale volumes should be present of the deleted PVC.
[2024-09-06T11:19:23.403Z]         """
[2024-09-06T11:19:23.403Z]         from pathlib import Path
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         if not Path(constants.CLI_TOOL_LOCAL_PATH).exists():
[2024-09-06T11:19:23.403Z]             retrieve_cli_binary(cli_type="odf")
[2024-09-06T11:19:23.403Z] >       output = run_cmd(cmd="odf-cli subvolume ls")
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z] tests/functional/odf-cli/test_pvc_stale_volume_cleanup_cli.py:37: 
[2024-09-06T11:19:23.403Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-09-06T11:19:23.403Z] ocs_ci/utility/utils.py:486: in run_cmd
[2024-09-06T11:19:23.403Z]     completed_process = exec_cmd(
[2024-09-06T11:19:23.403Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z] cmd = ['odf-cli', 'subvolume', 'ls'], secrets = None, timeout = 600
[2024-09-06T11:19:23.403Z] ignore_error = False, threading_lock = None, silent = False, use_shell = False
[2024-09-06T11:19:23.403Z] cluster_config = None, kwargs = {}, masked_cmd = 'odf-cli subvolume ls'
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z]     def exec_cmd(
[2024-09-06T11:19:23.403Z]         cmd,
[2024-09-06T11:19:23.403Z]         secrets=None,
[2024-09-06T11:19:23.403Z]         timeout=600,
[2024-09-06T11:19:23.403Z]         ignore_error=False,
[2024-09-06T11:19:23.403Z]         threading_lock=None,
[2024-09-06T11:19:23.403Z]         silent=False,
[2024-09-06T11:19:23.403Z]         use_shell=False,
[2024-09-06T11:19:23.403Z]         cluster_config=None,
[2024-09-06T11:19:23.403Z]         **kwargs,
[2024-09-06T11:19:23.403Z]     ):
[2024-09-06T11:19:23.403Z]         """
[2024-09-06T11:19:23.403Z]         Run an arbitrary command locally
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         If the command is grep and matching pattern is not found, then this function
[2024-09-06T11:19:23.403Z]         returns "command terminated with exit code 1" in stderr.
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         Args:
[2024-09-06T11:19:23.403Z]             cmd (str): command to run
[2024-09-06T11:19:23.403Z]             secrets (list): A list of secrets to be masked with asterisks
[2024-09-06T11:19:23.403Z]                 This kwarg is popped in order to not interfere with
[2024-09-06T11:19:23.403Z]                 subprocess.run(``**kwargs``)
[2024-09-06T11:19:23.403Z]             timeout (int): Timeout for the command, defaults to 600 seconds.
[2024-09-06T11:19:23.403Z]             ignore_error (bool): True if ignore non zero return code and do not
[2024-09-06T11:19:23.403Z]                 raise the exception.
[2024-09-06T11:19:23.403Z]             threading_lock (threading.RLock): threading.RLock object that is used
[2024-09-06T11:19:23.403Z]                 for handling concurrent oc commands
[2024-09-06T11:19:23.403Z]             silent (bool): If True will silent errors from the server, default false
[2024-09-06T11:19:23.403Z]             use_shell (bool): If True will pass the cmd without splitting
[2024-09-06T11:19:23.403Z]             cluster_config (MultiClusterConfig): In case of multicluster environment this object
[2024-09-06T11:19:23.403Z]                     will be non-null
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         Raises:
[2024-09-06T11:19:23.403Z]             CommandFailed: In case the command execution fails
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         Returns:
[2024-09-06T11:19:23.403Z]             (CompletedProcess) A CompletedProcess object of the command that was executed
[2024-09-06T11:19:23.403Z]             CompletedProcess attributes:
[2024-09-06T11:19:23.403Z]             args: The list or str args passed to run().
[2024-09-06T11:19:23.403Z]             returncode (str): The exit code of the process, negative for signals.
[2024-09-06T11:19:23.403Z]             stdout     (str): The standard output (None if not captured).
[2024-09-06T11:19:23.403Z]             stderr     (str): The standard error (None if not captured).
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         """
[2024-09-06T11:19:23.403Z]         masked_cmd = mask_secrets(cmd, secrets)
[2024-09-06T11:19:23.403Z]         log.info(f"Executing command: {masked_cmd}")
[2024-09-06T11:19:23.403Z]         if isinstance(cmd, str) and not kwargs.get("shell"):
[2024-09-06T11:19:23.403Z]             cmd = shlex.split(cmd)
[2024-09-06T11:19:23.403Z]         if config.RUN.get("custom_kubeconfig_location") and cmd[0] == "oc":
[2024-09-06T11:19:23.403Z]             if "--kubeconfig" in cmd:
[2024-09-06T11:19:23.403Z]                 cmd.pop(2)
[2024-09-06T11:19:23.403Z]                 cmd.pop(1)
[2024-09-06T11:19:23.403Z]             cmd = list_insert_at_position(cmd, 1, ["--kubeconfig"])
[2024-09-06T11:19:23.403Z]             cmd = list_insert_at_position(
[2024-09-06T11:19:23.403Z]                 cmd, 2, [config.RUN["custom_kubeconfig_location"]]
[2024-09-06T11:19:23.403Z]             )
[2024-09-06T11:19:23.403Z]         if cluster_config and cmd[0] == "oc" and "--kubeconfig" not in cmd:
[2024-09-06T11:19:23.403Z]             kubepath = cluster_config.RUN["kubeconfig"]
[2024-09-06T11:19:23.403Z]             kube_index = 1
[2024-09-06T11:19:23.403Z]             # check if we have an oc plugin in the command
[2024-09-06T11:19:23.403Z]             plugin_list = "oc plugin list"
[2024-09-06T11:19:23.403Z]             cp = subprocess.run(
[2024-09-06T11:19:23.403Z]                 shlex.split(plugin_list),
[2024-09-06T11:19:23.403Z]                 stdout=subprocess.PIPE,
[2024-09-06T11:19:23.403Z]                 stderr=subprocess.PIPE,
[2024-09-06T11:19:23.403Z]             )
[2024-09-06T11:19:23.403Z]             subcmd = cmd[1].split("-")
[2024-09-06T11:19:23.403Z]             if len(subcmd) > 1:
[2024-09-06T11:19:23.403Z]                 subcmd = "_".join(subcmd)
[2024-09-06T11:19:23.403Z]             if not isinstance(subcmd, str) and isinstance(subcmd, list):
[2024-09-06T11:19:23.403Z]                 subcmd = str(subcmd[0])
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]             for l in cp.stdout.decode().splitlines():
[2024-09-06T11:19:23.403Z]                 if subcmd in l:
[2024-09-06T11:19:23.403Z]                     # If oc cmdline has plugin name then we need to push the
[2024-09-06T11:19:23.403Z]                     # --kubeconfig to next index
[2024-09-06T11:19:23.403Z]                     kube_index = 2
[2024-09-06T11:19:23.403Z]                     log.info(f"Found oc plugin {subcmd}")
[2024-09-06T11:19:23.403Z]             cmd = list_insert_at_position(cmd, kube_index, ["--kubeconfig"])
[2024-09-06T11:19:23.403Z]             cmd = list_insert_at_position(cmd, kube_index + 1, [kubepath])
[2024-09-06T11:19:23.403Z]         if threading_lock and cmd[0] == "oc":
[2024-09-06T11:19:23.403Z]             threading_lock.acquire()
[2024-09-06T11:19:23.403Z]         completed_process = subprocess.run(
[2024-09-06T11:19:23.403Z]             cmd,
[2024-09-06T11:19:23.403Z]             stdout=subprocess.PIPE,
[2024-09-06T11:19:23.403Z]             stderr=subprocess.PIPE,
[2024-09-06T11:19:23.403Z]             stdin=subprocess.PIPE,
[2024-09-06T11:19:23.403Z]             timeout=timeout,
[2024-09-06T11:19:23.403Z]             **kwargs,
[2024-09-06T11:19:23.403Z]         )
[2024-09-06T11:19:23.403Z]         if threading_lock and cmd[0] == "oc":
[2024-09-06T11:19:23.403Z]             threading_lock.release()
[2024-09-06T11:19:23.403Z]         masked_stdout = mask_secrets(completed_process.stdout.decode(), secrets)
[2024-09-06T11:19:23.403Z]         if len(completed_process.stdout) > 0:
[2024-09-06T11:19:23.403Z]             log.debug(f"Command stdout: {masked_stdout}")
[2024-09-06T11:19:23.403Z]         else:
[2024-09-06T11:19:23.403Z]             log.debug("Command stdout is empty")
[2024-09-06T11:19:23.403Z]     
[2024-09-06T11:19:23.403Z]         masked_stderr = mask_secrets(completed_process.stderr.decode(), secrets)
[2024-09-06T11:19:23.403Z]         if len(completed_process.stderr) > 0:
[2024-09-06T11:19:23.403Z]             if not silent:
[2024-09-06T11:19:23.403Z]                 log.warning(f"Command stderr: {masked_stderr}")
[2024-09-06T11:19:23.403Z]         else:
[2024-09-06T11:19:23.403Z]             log.debug("Command stderr is empty")
[2024-09-06T11:19:23.403Z]         log.debug(f"Command return code: {completed_process.returncode}")
[2024-09-06T11:19:23.403Z]         if completed_process.returncode and not ignore_error:
[2024-09-06T11:19:23.403Z]             masked_stderr = bin_xml_escape(filter_out_emojis(masked_stderr))
[2024-09-06T11:19:23.403Z]             if (
[2024-09-06T11:19:23.403Z]                 "grep" in masked_cmd
[2024-09-06T11:19:23.403Z]                 and b"command terminated with exit code 1" in completed_process.stderr
[2024-09-06T11:19:23.403Z]             ):
[2024-09-06T11:19:23.403Z]                 log.info(f"No results found for grep command: {masked_cmd}")
[2024-09-06T11:19:23.403Z]             else:
[2024-09-06T11:19:23.403Z] >               raise CommandFailed(
[2024-09-06T11:19:23.403Z]                     f"Error during execution of command: {masked_cmd}."
[2024-09-06T11:19:23.403Z]                     f"\nError is {masked_stderr}"
[2024-09-06T11:19:23.403Z]                 )
[2024-09-06T11:19:23.403Z] E               ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: odf-cli subvolume ls.
[2024-09-06T11:19:23.403Z] E               Error is Error: Operator namespace 'openshift-storage' does not exist. namespaces "openshift-storage" not found
[2024-09-06T11:19:23.403Z] 
[2024-09-06T11:19:23.403Z] ocs_ci/utility/utils.py:704: CommandFailed

this issue opened following slack conversation https://ibm-systems-storage.slack.com/archives/C06EEKSUPPZ/p1725885411893119

DanielOsypenko commented 4 weeks ago

https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/41837/consoleText

DanielOsypenko commented 2 weeks ago

new failure - https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/42316/ @AviadP