search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
108 stars 168 forks source link

fix error in test 0.2 #10910

Open DanielOsypenko opened 4 days ago

DanielOsypenko commented 4 days ago

install_logging fixture creates resources that exist on ROSA HCP cluster.

[2024-11-22T10:28:56.481Z] 05:28:56 - MainThread - ocs_ci.utility.utils - WARNING  - Command stderr: Error from server (AlreadyExists): error when creating "/home/jenkins/workspace/qe-deploy-ocs-cluster/ocs-ci/ocs_ci/templates/openshift-infra/logging-deployment/clusterlogging_operator/cl-namespace.yaml": namespaces "openshift-logging" already exists
[2024-11-22T10:28:56.481Z] 
[2024-11-22T10:28:56.736Z] 05:28:56 - MainThread - ocs_ci.framework.pytest_customization.reports - INFO  - duration reported by tests/functional/workloads/ocp/logging/test_openshift-logging.py::Testopenshiftloggingonocs::test_create_new_project_to_verify_logging immediately after test execution: 51.11
[2024-11-22T10:28:56.736Z] ERROR
[2024-11-22T10:28:56.736Z] _ ERROR at setup of Testopenshiftloggingonocs.test_create_new_project_to_verify_logging _
[2024-11-22T10:28:56.736Z] 
[2024-11-22T10:28:56.736Z] request = <SubRequest 'install_logging' for <Function test_create_new_project_to_verify_logging>>
[2024-11-22T10:28:56.736Z] 
[2024-11-22T10:28:56.736Z]     @pytest.fixture(scope="class")
[2024-11-22T10:28:56.736Z]     def install_logging(request):
[2024-11-22T10:28:56.736Z]         """
[2024-11-22T10:28:56.736Z]         Setup and teardown
[2024-11-22T10:28:56.736Z]         * The setup will deploy openshift-logging in the cluster
[2024-11-22T10:28:56.736Z]         * The teardown will uninstall cluster-logging from the cluster
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         """
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         def finalizer():
[2024-11-22T10:28:56.736Z]             uninstall_cluster_logging()
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         request.addfinalizer(finalizer)
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         sub = ocp.OCP(
[2024-11-22T10:28:56.736Z]             kind=constants.SUBSCRIPTION,
[2024-11-22T10:28:56.736Z]             namespace=constants.OPENSHIFT_LOGGING_NAMESPACE,
[2024-11-22T10:28:56.736Z]         )
[2024-11-22T10:28:56.736Z]         logging_sub = sub.get().get("items")
[2024-11-22T10:28:56.736Z]         if logging_sub:
[2024-11-22T10:28:56.736Z]             log.info("Logging is already configured, Skipping Installation")
[2024-11-22T10:28:56.736Z]             return
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         log.info("Configuring Openshift-logging")
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Gets OCP version to align logging version to OCP version
[2024-11-22T10:28:56.736Z]         ocp_version = version.get_semantic_ocp_version_from_config()
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         logging_channel = "stable" if ocp_version >= version.VERSION_4_7 else ocp_version
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Creates namespace openshift-operators-redhat
[2024-11-22T10:28:56.736Z]         try:
[2024-11-22T10:28:56.736Z]             ocp_logging_obj.create_namespace(yaml_file=constants.EO_NAMESPACE_YAML)
[2024-11-22T10:28:56.736Z]         except CommandFailed as e:
[2024-11-22T10:28:56.736Z]             if "AlreadyExists" in str(e):
[2024-11-22T10:28:56.736Z]                 # on Rosa HCP the ns created from the deployment
[2024-11-22T10:28:56.736Z]                 log.info("Namespace openshift-operators-redhat already exists")
[2024-11-22T10:28:56.736Z]             else:
[2024-11-22T10:28:56.736Z]                 raise
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Creates an operator-group for elasticsearch
[2024-11-22T10:28:56.736Z]         assert ocp_logging_obj.create_elasticsearch_operator_group(
[2024-11-22T10:28:56.736Z]             yaml_file=constants.EO_OG_YAML, resource_name="openshift-operators-redhat"
[2024-11-22T10:28:56.736Z]         )
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Set RBAC policy on the project
[2024-11-22T10:28:56.736Z]         assert ocp_logging_obj.set_rbac(
[2024-11-22T10:28:56.736Z]             yaml_file=constants.EO_RBAC_YAML, resource_name="prometheus-k8s"
[2024-11-22T10:28:56.736Z]         )
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Creates subscription for elastic-search operator
[2024-11-22T10:28:56.736Z]         subscription_yaml = templating.load_yaml(constants.EO_SUB_YAML)
[2024-11-22T10:28:56.736Z]         subscription_yaml["spec"]["channel"] = logging_channel
[2024-11-22T10:28:56.736Z]         helpers.create_resource(**subscription_yaml)
[2024-11-22T10:28:56.736Z]         assert ocp_logging_obj.get_elasticsearch_subscription()
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Checks for Elasticsearch operator
[2024-11-22T10:28:56.736Z]         elastic_search_operator = OCP(
[2024-11-22T10:28:56.736Z]             kind=constants.POD, namespace=constants.OPENSHIFT_OPERATORS_REDHAT_NAMESPACE
[2024-11-22T10:28:56.736Z]         )
[2024-11-22T10:28:56.736Z]         elastic_search_operator.wait_for_resource(
[2024-11-22T10:28:56.736Z]             resource_count=1, condition=constants.STATUS_RUNNING, timeout=200, sleep=20
[2024-11-22T10:28:56.736Z]         )
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         # Creates a namespace openshift-logging
[2024-11-22T10:28:56.736Z] >       ocp_logging_obj.create_namespace(yaml_file=constants.CL_NAMESPACE_YAML)
[2024-11-22T10:28:56.736Z] 
[2024-11-22T10:28:56.736Z] tests/conftest.py:3328: 
[2024-11-22T10:28:56.736Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-11-22T10:28:56.736Z] ocs_ci/utility/deployment_openshift_logging.py:41: in create_namespace
[2024-11-22T10:28:56.736Z]     assert namespaces.create(yaml_file=yaml_file), "Failed to create namespace"
[2024-11-22T10:28:56.736Z] ocs_ci/ocs/ocp.py:442: in create
[2024-11-22T10:28:56.736Z]     output = self.exec_oc_cmd(command)
[2024-11-22T10:28:56.736Z] ocs_ci/ocs/ocp.py:212: in exec_oc_cmd
[2024-11-22T10:28:56.736Z]     out = run_cmd(
[2024-11-22T10:28:56.736Z] ocs_ci/utility/utils.py:488: in run_cmd
[2024-11-22T10:28:56.736Z]     completed_process = exec_cmd(
[2024-11-22T10:28:56.736Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-11-22T10:28:56.736Z] 
[2024-11-22T10:28:56.736Z] cmd = ['oc', '--kubeconfig', '/home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig', 'create', '-f', '/ho...ocs-cluster/ocs-ci/ocs_ci/templates/openshift-infra/logging-deployment/clusterlogging_operator/cl-namespace.yaml', ...]
[2024-11-22T10:28:56.736Z] secrets = None, timeout = 600, ignore_error = False, threading_lock = None
[2024-11-22T10:28:56.736Z] silent = False, use_shell = False
[2024-11-22T10:28:56.736Z] cluster_config = <ocs_ci.framework.MultiClusterConfig object at 0x7f628b297490>
[2024-11-22T10:28:56.736Z] lock_timeout = 7200, kwargs = {}
[2024-11-22T10:28:56.736Z] masked_cmd = 'oc create -f /home/jenkins/workspace/qe-deploy-ocs-cluster/ocs-ci/ocs_ci/templates/openshift-infra/logging-deployment/clusterlogging_operator/cl-namespace.yaml -o yaml'
[2024-11-22T10:28:56.736Z] kubepath = '/home/jenkins/current-cluster-dir/openshift-cluster-dir/auth/kubeconfig'
[2024-11-22T10:28:56.736Z] kube_index = 1, plugin_list = 'oc plugin list'
[2024-11-22T10:28:56.736Z] cp = CompletedProcess(args=['oc', 'plugin', 'list'], returncode=1, stdout=b'', stderr=b'Unable to read directory "/home/jen.../jenkins/.local/bin: no such file or directory. Skipping...\nerror: unable to find any kubectl plugins in your PATH\n')
[2024-11-22T10:28:56.736Z] subcmd = 'create'
[2024-11-22T10:28:56.736Z] 
[2024-11-22T10:28:56.736Z]     def exec_cmd(
[2024-11-22T10:28:56.736Z]         cmd,
[2024-11-22T10:28:56.736Z]         secrets=None,
[2024-11-22T10:28:56.736Z]         timeout=600,
[2024-11-22T10:28:56.736Z]         ignore_error=False,
[2024-11-22T10:28:56.736Z]         threading_lock=None,
[2024-11-22T10:28:56.736Z]         silent=False,
[2024-11-22T10:28:56.736Z]         use_shell=False,
[2024-11-22T10:28:56.736Z]         cluster_config=None,
[2024-11-22T10:28:56.736Z]         lock_timeout=7200,
[2024-11-22T10:28:56.736Z]         **kwargs,
[2024-11-22T10:28:56.736Z]     ):
[2024-11-22T10:28:56.736Z]         """
[2024-11-22T10:28:56.736Z]         Run an arbitrary command locally
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         If the command is grep and matching pattern is not found, then this function
[2024-11-22T10:28:56.736Z]         returns "command terminated with exit code 1" in stderr.
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         Args:
[2024-11-22T10:28:56.736Z]             cmd (str): command to run
[2024-11-22T10:28:56.736Z]             secrets (list): A list of secrets to be masked with asterisks
[2024-11-22T10:28:56.736Z]                 This kwarg is popped in order to not interfere with
[2024-11-22T10:28:56.736Z]                 subprocess.run(``**kwargs``)
[2024-11-22T10:28:56.736Z]             timeout (int): Timeout for the command, defaults to 600 seconds.
[2024-11-22T10:28:56.736Z]             ignore_error (bool): True if ignore non zero return code and do not
[2024-11-22T10:28:56.736Z]                 raise the exception.
[2024-11-22T10:28:56.736Z]             threading_lock (threading.RLock): threading.RLock object that is used
[2024-11-22T10:28:56.736Z]                 for handling concurrent oc commands
[2024-11-22T10:28:56.736Z]             silent (bool): If True will silent errors from the server, default false
[2024-11-22T10:28:56.736Z]             use_shell (bool): If True will pass the cmd without splitting
[2024-11-22T10:28:56.736Z]             cluster_config (MultiClusterConfig): In case of multicluster environment this object
[2024-11-22T10:28:56.736Z]                     will be non-null
[2024-11-22T10:28:56.736Z]             lock_timeout (int): maximum timeout to wait for lock to prevent deadlocks (default 2 hours)
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         Raises:
[2024-11-22T10:28:56.736Z]             CommandFailed: In case the command execution fails
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         Returns:
[2024-11-22T10:28:56.736Z]             (CompletedProcess) A CompletedProcess object of the command that was executed
[2024-11-22T10:28:56.736Z]             CompletedProcess attributes:
[2024-11-22T10:28:56.736Z]             args: The list or str args passed to run().
[2024-11-22T10:28:56.736Z]             returncode (str): The exit code of the process, negative for signals.
[2024-11-22T10:28:56.736Z]             stdout     (str): The standard output (None if not captured).
[2024-11-22T10:28:56.736Z]             stderr     (str): The standard error (None if not captured).
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         """
[2024-11-22T10:28:56.736Z]         masked_cmd = mask_secrets(cmd, secrets)
[2024-11-22T10:28:56.736Z]         log.info(f"Executing command: {masked_cmd}")
[2024-11-22T10:28:56.736Z]         if isinstance(cmd, str) and not kwargs.get("shell"):
[2024-11-22T10:28:56.736Z]             cmd = shlex.split(cmd)
[2024-11-22T10:28:56.736Z]         if config.RUN.get("custom_kubeconfig_location") and cmd[0] == "oc":
[2024-11-22T10:28:56.736Z]             if "--kubeconfig" in cmd:
[2024-11-22T10:28:56.736Z]                 cmd.pop(2)
[2024-11-22T10:28:56.736Z]                 cmd.pop(1)
[2024-11-22T10:28:56.736Z]             cmd = list_insert_at_position(cmd, 1, ["--kubeconfig"])
[2024-11-22T10:28:56.736Z]             cmd = list_insert_at_position(
[2024-11-22T10:28:56.736Z]                 cmd, 2, [config.RUN["custom_kubeconfig_location"]]
[2024-11-22T10:28:56.736Z]             )
[2024-11-22T10:28:56.736Z]         if cluster_config and cmd[0] == "oc" and "--kubeconfig" not in cmd:
[2024-11-22T10:28:56.736Z]             kubepath = cluster_config.RUN["kubeconfig"]
[2024-11-22T10:28:56.736Z]             kube_index = 1
[2024-11-22T10:28:56.736Z]             # check if we have an oc plugin in the command
[2024-11-22T10:28:56.736Z]             plugin_list = "oc plugin list"
[2024-11-22T10:28:56.736Z]             cp = subprocess.run(
[2024-11-22T10:28:56.736Z]                 shlex.split(plugin_list),
[2024-11-22T10:28:56.736Z]                 stdout=subprocess.PIPE,
[2024-11-22T10:28:56.736Z]                 stderr=subprocess.PIPE,
[2024-11-22T10:28:56.736Z]             )
[2024-11-22T10:28:56.736Z]             subcmd = cmd[1].split("-")
[2024-11-22T10:28:56.736Z]             if len(subcmd) > 1:
[2024-11-22T10:28:56.736Z]                 subcmd = "_".join(subcmd)
[2024-11-22T10:28:56.736Z]             if not isinstance(subcmd, str) and isinstance(subcmd, list):
[2024-11-22T10:28:56.736Z]                 subcmd = str(subcmd[0])
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]             for l in cp.stdout.decode().splitlines():
[2024-11-22T10:28:56.736Z]                 if subcmd in l:
[2024-11-22T10:28:56.736Z]                     # If oc cmdline has plugin name then we need to push the
[2024-11-22T10:28:56.736Z]                     # --kubeconfig to next index
[2024-11-22T10:28:56.736Z]                     kube_index = 2
[2024-11-22T10:28:56.736Z]                     log.info(f"Found oc plugin {subcmd}")
[2024-11-22T10:28:56.736Z]             cmd = list_insert_at_position(cmd, kube_index, ["--kubeconfig"])
[2024-11-22T10:28:56.736Z]             cmd = list_insert_at_position(cmd, kube_index + 1, [kubepath])
[2024-11-22T10:28:56.736Z]         try:
[2024-11-22T10:28:56.736Z]             if threading_lock and cmd[0] == "oc":
[2024-11-22T10:28:56.736Z]                 threading_lock.acquire(timeout=lock_timeout)
[2024-11-22T10:28:56.736Z]             completed_process = subprocess.run(
[2024-11-22T10:28:56.736Z]                 cmd,
[2024-11-22T10:28:56.736Z]                 stdout=subprocess.PIPE,
[2024-11-22T10:28:56.736Z]                 stderr=subprocess.PIPE,
[2024-11-22T10:28:56.736Z]                 stdin=subprocess.PIPE,
[2024-11-22T10:28:56.736Z]                 timeout=timeout,
[2024-11-22T10:28:56.736Z]                 **kwargs,
[2024-11-22T10:28:56.736Z]             )
[2024-11-22T10:28:56.736Z]         finally:
[2024-11-22T10:28:56.736Z]             if threading_lock and cmd[0] == "oc":
[2024-11-22T10:28:56.736Z]                 threading_lock.release()
[2024-11-22T10:28:56.736Z]         masked_stdout = mask_secrets(completed_process.stdout.decode(), secrets)
[2024-11-22T10:28:56.736Z]         if len(completed_process.stdout) > 0:
[2024-11-22T10:28:56.736Z]             log.debug(f"Command stdout: {masked_stdout}")
[2024-11-22T10:28:56.736Z]         else:
[2024-11-22T10:28:56.736Z]             log.debug("Command stdout is empty")
[2024-11-22T10:28:56.736Z]     
[2024-11-22T10:28:56.736Z]         masked_stderr = mask_secrets(completed_process.stderr.decode(), secrets)
[2024-11-22T10:28:56.736Z]         if len(completed_process.stderr) > 0:
[2024-11-22T10:28:56.736Z]             if not silent:
[2024-11-22T10:28:56.736Z]                 log.warning(f"Command stderr: {masked_stderr}")
[2024-11-22T10:28:56.736Z]         else:
[2024-11-22T10:28:56.736Z]             log.debug("Command stderr is empty")
[2024-11-22T10:28:56.736Z]         log.debug(f"Command return code: {completed_process.returncode}")
[2024-11-22T10:28:56.736Z]         if completed_process.returncode and not ignore_error:
[2024-11-22T10:28:56.736Z]             masked_stderr = bin_xml_escape(filter_out_emojis(masked_stderr))
[2024-11-22T10:28:56.736Z]             if (
[2024-11-22T10:28:56.736Z]                 "grep" in masked_cmd
[2024-11-22T10:28:56.736Z]                 and b"command terminated with exit code 1" in completed_process.stderr
[2024-11-22T10:28:56.736Z]             ):
[2024-11-22T10:28:56.736Z]                 log.info(f"No results found for grep command: {masked_cmd}")
[2024-11-22T10:28:56.736Z]             else:
[2024-11-22T10:28:56.736Z] >               raise CommandFailed(
[2024-11-22T10:28:56.736Z]                     f"Error during execution of command: {masked_cmd}."
[2024-11-22T10:28:56.736Z]                     f"\nError is {masked_stderr}"
[2024-11-22T10:28:56.736Z]                 )
[2024-11-22T10:28:56.737Z] E               ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc create -f /home/jenkins/workspace/qe-deploy-ocs-cluster/ocs-ci/ocs_ci/templates/openshift-infra/logging-deployment/clusterlogging_operator/cl-namespace.yaml -o yaml.
[2024-11-22T10:28:56.737Z] E               Error is Error from server (AlreadyExists): error when creating "/home/jenkins/workspace/qe-deploy-ocs-cluster/ocs-ci/ocs_ci/templates/openshift-infra/logging-deployment/clusterlogging_operator/cl-namespace.yaml": namespaces "openshift-logging" already exists
[2024-11-22T10:28:56.737Z] 
[2024-11-22T10:28:56.737Z] ocs_ci/utility/utils.py:710: CommandFailed
[2024-11-22T10:28:56.737Z]

https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/44277

openshift-ci[bot] commented 4 days ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: DanielOsypenko

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/red-hat-storage/ocs-ci/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
DanielOsypenko commented 2 days ago

execution job - https://url.corp.redhat.com/4929da9

DanielOsypenko commented 2 days ago

/cherry-pick release-4.17

openshift-cherrypick-robot commented 2 days ago

@DanielOsypenko: once the present PR merges, I will cherry-pick it on top of release-4.17 in a new PR and assign it to you.

In response to [this](https://github.com/red-hat-storage/ocs-ci/pull/10910#issuecomment-2500401947): >/cherry-pick release-4.17 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.