search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
365 stars 194 forks source link

OpenShift: Creating and deleting of TransportServers with F5 CIS Operator Version `2.18.1-ubi9` not working #3660

Open keennessch opened 4 days ago

keennessch commented 4 days ago

Setup Details

CIS Version : 2.18.1 Build: registry.connect.redhat.com/f5networks/cntr-ingress-svcs:2.18.1-ubi9 BIGIP Version: BIG-IP 17.1.1.3 Build 0.21.5 Engineering Hotfix AS3 Version: 3.48.0 Agent Mode: AS3 Orchestration: OSCP Orchestration Version: 4.16.17 Pool Mode: Cluster Additional Setup details:

Description

After updating the F5 CIS Operator to 2.18.1-ubi9, existing TransportServer resources change to Pending status. New TransportServers immediately show Pending status, stay in this state and are never created on the load balancer. Deleting TransportServer resources are deleted from the OpenShift cluster but not from the load balancer. Switching the Operator's version back to 2.18.0-ubi9 by defining it in the .spec.version field of the F5BigIpCtlr resource, TransportServer resources change to Ok status after a couple of seconds. Creating and deleting TransportServers works as expected and have the desired effect on the load balancer.

Steps To Reproduce

1) Change the .spec.version field to 2.18.1-ubi9 in the F5BigIpCtlr resource on the OCP cluster 2) Wait for the new f5-bigip-ctlr pod to successfully start

Expected Result

TransportServer resources show the correct state corresponding to their virtual server counterparts on the load balancer. New TransportServer resources can be created and show up as virtual servers on the load balancer. Existing TransportServer resources can be deleted and are deleted on the load balancer.

Actual Result

TransportServer resources show Pending state, regardless of their actual state on the load balancer. New TransportServer resources can be created on the OCP cluster but are not created on the load balancer. Existing TransportServer resources can be deleted on the OCP cluster but are not deleted on the load balancer.

Diagnostic Information

The used F5BigIpCtlr definition while collecting logs for this issue:

apiVersion: cis.f5.com/v1
kind: F5BigIpCtlr
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: in-cluster-f5-cis-instance:cis.f5.com/F5BigIpCtlr:f5-cis/f5-bigip-ctlr
    operator-sdk/primary-resource: /f5-bigip-ctlr
    operator-sdk/primary-resource-type: ClusterRoleBinding.rbac.authorization.k8s.io
  creationTimestamp: "2024-10-29T13:05:16Z"
  finalizers:
  - helm.sdk.operatorframework.io/uninstall-release
  name: f5-bigip-ctlr
  namespace: f5-cis
spec:
  args:
    agent: as3
    as3_validation: true
    bigip_partition: <REDACTED-PARTITION>
    bigip_url: <REDACTED-URL>
    custom_resource_mode: true
    default_route_domain: <REDACTED-ROUTE-DOMAIN>
    http_client_metrics: true
    insecure: false
    log_as3_response: true
    log_level: AS3DEBUG
    orchestration_cni: ovn-k8s
    pool_member_type: cluster
    static_routing_mode: true
    trusted-certs-cfgmap: f5-cis/user-ca-bundle
  bigip_login_secret: f5-bigip-ctlr-login
  image:
    pullPolicy: Always
    repo: f5networks/cntr-ingress-svcs
    user: registry.connect.redhat.com
  ingressClass:
    create: false
    defaultController: false
    ingressClassName: f5
  namespace: f5-cis
  podSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
      - ALL
    runAsGroup: null
    runAsNonRoot: true
    runAsUser: null
  rbac:
    create: true
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsUser: 1000
  serviceAccount:
    create: true
    name: f5-bigip-ctrl-sa
  version: 2.18.1-ubi9
status:
  conditions:
  - lastTransitionTime: "2024-10-29T13:05:33Z"
    status: "True"
    type: Initialized
  - lastTransitionTime: "2024-10-29T13:05:35Z"
    message: |+
      Container Ingress Services controller: f5-bigip-ctlr

      Controller Documentation:
      - Kubernetes: https://clouddocs.f5.com/containers/latest/userguide/kubernetes/
      - OpenShift: https://clouddocs.f5.com/containers/latest/userguide/openshift/

    reason: UpgradeSuccessful
    status: "True"
    type: Deployed
  deployedRelease:
    manifest: ...
    name: f5-bigip-ctlr

Logs from the f5-bigip-ctlr pod. Repeating lines have been replaced by [...].


2024/11/25 10:44:42 [DEBUG] No url in credentials directory, falling back to CLI argument
2024/11/25 10:44:42 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.18.1, BuildInfo: azure-6775-e3c0ec4ef25fc2fd3b994c05b63317aed2662622
2024/11/25 10:44:42 [DEBUG] [DEBUG] Adding https at the beginning of the GTM BIG IP URL as it does not start with https.
2024/11/25 10:44:43 [DEBUG] [BIGIP] Http client instrumented with metrics!
2024/11/25 10:44:43 [INFO] ConfigWriter started: 0xc0003345d0
2024/11/25 10:44:43 [WARNING] Creating GTM with default bigip credentials as GTM BIGIP Url or GTM BIGIP Username or GTM BIGIP Password is missing on CIS args.
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) writing section name global
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) successfully wrote section (global)
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) writing section name bigip
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) successfully wrote section (bigip)
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) writing section name gtm_bigip
2024/11/25 10:44:43 [DEBUG] [CCCL] ConfigWriter (0xc0003345d0) successfully wrote section (gtm_bigip)
2024/11/25 10:44:43 [INFO] Started config driver sub-process at pid: 17
2024/11/25 10:44:43 [DEBUG] [AS3] posting GET BIGIP AS3 Version request on https://<REDACTED-URL>/mgmt/shared/appsvcs/info
2024/11/25 10:44:43 [INFO] [CORE] Registered BigIP Metrics
2024/11/25 10:44:44 [DEBUG] /usr/local/bin/bigipconfigdriver.py:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2024/11/25 10:44:44 [DEBUG]   __import__('pkg_resources').require('f5-ctlr-agent==0.1.0')
2024/11/25 10:44:46 [DEBUG] [AS3] BIGIP is serving with AS3 version: 3.48.0
2024/11/25 10:44:46 [DEBUG] Controller Created
2024/11/25 10:44:46 [DEBUG] Client Created
2024/11/25 10:44:46 [DEBUG] No namespaces provided. Watching all namespaces
2024/11/25 10:44:46 [DEBUG] Creating Common Resource Informers for Namespace: 
2024/11/25 10:44:46 [DEBUG] Creating Custom Resource Informers for Namespace: 
2024/11/25 10:44:46 [DEBUG] [AS3] Posting GET BIGIP Reg Key request on https://<REDACTED-URL>/mgmt/tm/shared/licensing/registration
2024/11/25 10:44:46 [INFO] Starting Controller
2024/11/25 10:44:46 [INFO] Starting  Node Informer
I1125 10:44:46.716073       1 shared_informer.go:240] Waiting for caches to sync for F5 CIS Ingress Controller
2024/11/25 10:44:46 [DEBUG]  Initialising controller monitored kubernetes nodes from cluster: local
[...]
2024/11/25 10:44:47 [DEBUG]  Initialising controller monitored kubernetes nodes from cluster: local
I1125 10:44:47.016185       1 shared_informer.go:247] Caches are synced for F5 CIS Ingress Controller 
2024/11/25 10:44:47 [DEBUG] Successfully synced node informer cache
2024/11/25 10:44:47 [INFO] Starting ExternalDNS Informer
I1125 10:44:47.016232       1 shared_informer.go:240] Waiting for caches to sync for F5 CIS Ingress Controller
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
2024/11/25 10:44:48 [DEBUG] Enqueueing Endpoints: <REDACTED>
[...]
2024/11/25 10:44:48 [DEBUG] Enqueueing Endpoints: <REDACTED>
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
* Port 8443 provides access to the OpenShift resource metrics. This port is for internal use, and no other usage is guaranteed.
* Port 9443 provides access to the internal `openshift-state-metrics` metrics. This port is for internal use, and no other usage is guaranteed. operator.openshift.io/spec-hash:700eac9efd8a924fdb3738d2a7935c5f8c8de046a1c3806df1ca3c92076a4ecc service.alpha.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212 service.beta.openshift.io/serving-cert-secret-name:openshift-state-metrics-tls service.beta.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212] [] []  [{service-ca-operator Update v1 2024-04-25 12:24:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:service.alpha.openshift.io/serving-cert-signed-by":{},"f:service.beta.openshift.io/serving-cert-signed-by":{}}}}} {operator Update v1 2024-10-28 11:00:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/description":{},"f:operator.openshift.io/spec-hash":{},"f:service.beta.openshift.io/serving-cert-secret-name":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:k8s-app":{}}},"f:spec":{"f:clusterIP":{},"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":8443,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9443,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:https-main,Protocol:TCP,Port:8443,TargetPort:{1 0 https-main},NodePort:0,AppProtocol:nil,},ServicePort{Name:https-self,Protocol:TCP,Port:9443,TargetPort:{1 0 https-self},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/component: exporter,app.kubernetes.io/name: openshift-state-metrics,},ClusterIP:None,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[None],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},} from cluster: local
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
* Port 9091 provides access to all the Prometheus endpoints. Granting access requires binding a user to the `cluster-monitoring-view` cluster role.
* Port 9092 provides access the `/metrics` and `/federate` endpoints only. This port is for internal use, and no other usage is guaranteed. operator.openshift.io/spec-hash:44db31cd73c886ad771ecc7e4731887da474737fa746e940e1abe3283798b76f service.alpha.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212 service.beta.openshift.io/serving-cert-secret-name:prometheus-k8s-tls service.beta.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212] [] []  [{service-ca-operator Update v1 2024-04-25 12:24:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:service.alpha.openshift.io/serving-cert-signed-by":{},"f:service.beta.openshift.io/serving-cert-signed-by":{}}}}} {operator Update v1 2024-10-28 11:00:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/description":{},"f:operator.openshift.io/spec-hash":{},"f:service.beta.openshift.io/serving-cert-secret-name":{}},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9091,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9092,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:sessionAffinityConfig":{".":{},"f:clientIP":{".":{},"f:timeoutSeconds":{}}},"f:type":{}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:web,Protocol:TCP,Port:9091,TargetPort:{1 0 web},NodePort:0,AppProtocol:nil,},ServicePort{Name:metrics,Protocol:TCP,Port:9092,TargetPort:{1 0 metrics},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/component: prometheus,app.kubernetes.io/instance: k8s,app.kubernetes.io/name: prometheus,app.kubernetes.io/part-of: openshift-monitoring,},ClusterIP:<REDACTED-IP>,Type:ClusterIP,ExternalIPs:[],SessionAffinity:ClientIP,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:&SessionAffinityConfig{ClientIP:&ClientIPConfig{TimeoutSeconds:*10800,},},TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[<REDACTED-IP>],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},} from cluster: local
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
[...]
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
* Port 9091 provides access to all the Thanos Querier endpoints. Granting access requires binding a user to the `cluster-monitoring-view` cluster role.
* Port 9092 provides access to the `/api/v1/query`, `/api/v1/query_range/, `/api/v1/labels`, `/api/v1/label/*/values`, and `/api/v1/series` endpoints restricted to a given project. Granting access requires binding a user to the `view` cluster role in the project.
* Port 9093 provides access to the `/api/v1/alerts`, and `/api/v1/rules` endpoints restricted to a given project. Granting access requires binding a user to the `monitoring-rules-edit` cluster role or `monitoring-edit` cluster role or `monitoring-rules-view` cluster role in the project.
* Port 9094 provides access to the `/metrics` endpoint only. This port is for internal use, and no other usage is guaranteed. operator.openshift.io/spec-hash:0c29478f63d34bf0dd2a114c15b5518c2e967a61c11e61f2676c05a268c8b7f8 service.alpha.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212 service.beta.openshift.io/serving-cert-secret-name:thanos-querier-tls service.beta.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212] [] []  [{service-ca-operator Update v1 2024-04-25 12:24:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:service.alpha.openshift.io/serving-cert-signed-by":{},"f:service.beta.openshift.io/serving-cert-signed-by":{}}}}} {operator Update v1 2024-10-28 11:00:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/description":{},"f:operator.openshift.io/spec-hash":{},"f:service.beta.openshift.io/serving-cert-secret-name":{}},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9091,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9092,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9093,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9094,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:web,Protocol:TCP,Port:9091,TargetPort:{1 0 web},NodePort:0,AppProtocol:nil,},ServicePort{Name:tenancy,Protocol:TCP,Port:9092,TargetPort:{1 0 tenancy},NodePort:0,AppProtocol:nil,},ServicePort{Name:tenancy-rules,Protocol:TCP,Port:9093,TargetPort:{1 0 tenancy-rules},NodePort:0,AppProtocol:nil,},ServicePort{Name:metrics,Protocol:TCP,Port:9094,TargetPort:{1 0 metrics},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/component: query-layer,app.kubernetes.io/instance: thanos-querier,app.kubernetes.io/name: thanos-query,app.kubernetes.io/part-of: openshift-monitoring,},ClusterIP:<REDACTED-IP>,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[<REDACTED-IP>],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},} from cluster: local
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
[...]
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
* Port 9091 provides access to the `/metrics` endpoint only. This port is for internal use, and no other usage is guaranteed.
* Port 9092 provides access to the `/federate` endpoint only. Granting access requires binding a user to the `cluster-monitoring-view` cluster role.

This also exposes the `/metrics` endpoint of the Thanos sidecar web server on port 10902. This port is for internal use, and no other usage is guaranteed. operator.openshift.io/spec-hash:5d24d54c21bc80e08e61fb22543775331c03d254b249a560e583818a38ccf404 service.alpha.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212 service.beta.openshift.io/serving-cert-secret-name:prometheus-user-workload-tls service.beta.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212] [] []  [{service-ca-operator Update v1 2024-04-25 13:08:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:service.alpha.openshift.io/serving-cert-signed-by":{},"f:service.beta.openshift.io/serving-cert-signed-by":{}}}}} {operator Update v1 2024-10-28 11:00:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/description":{},"f:operator.openshift.io/spec-hash":{},"f:service.beta.openshift.io/serving-cert-secret-name":{}},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9091,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9092,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":10902,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:sessionAffinityConfig":{".":{},"f:clientIP":{".":{},"f:timeoutSeconds":{}}},"f:type":{}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:metrics,Protocol:TCP,Port:9091,TargetPort:{1 0 metrics},NodePort:0,AppProtocol:nil,},ServicePort{Name:federate,Protocol:TCP,Port:9092,TargetPort:{1 0 federate},NodePort:0,AppProtocol:nil,},ServicePort{Name:thanos-proxy,Protocol:TCP,Port:10902,TargetPort:{1 0 thanos-proxy},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/component: prometheus,app.kubernetes.io/instance: user-workload,app.kubernetes.io/name: prometheus,app.kubernetes.io/part-of: openshift-monitoring,},ClusterIP:<REDACTED-IP>,Type:ClusterIP,ExternalIPs:[],SessionAffinity:ClientIP,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:&SessionAffinityConfig{ClientIP:&ClientIPConfig{TimeoutSeconds:*10800,},},TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[<REDACTED-IP>],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},} from cluster: local
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
* Port 9091 provides access to all Thanos Ruler endpoints. Granting access requires binding a user to the `cluster-monitoring-view` cluster role.
* Port 9092 provides access to the `/metrics` endpoint only. This port is for internal use, and no other usage is guaranteed.

This also exposes the gRPC endpoints on port 10901. This port is for internal use, and no other usage is guaranteed. operator.openshift.io/spec-hash:958429f802736f0fb134982be126b4f7bc3ed0485008e5efc3040433d3dc36fc service.alpha.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212 service.beta.openshift.io/serving-cert-secret-name:thanos-ruler-tls service.beta.openshift.io/serving-cert-signed-by:openshift-service-serving-signer@1714047212] [] []  [{service-ca-operator Update v1 2024-04-25 13:08:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:service.alpha.openshift.io/serving-cert-signed-by":{},"f:service.beta.openshift.io/serving-cert-signed-by":{}}}}} {operator Update v1 2024-10-28 11:00:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:openshift.io/description":{},"f:operator.openshift.io/spec-hash":{},"f:service.beta.openshift.io/serving-cert-secret-name":{}},"f:labels":{".":{},"f:app.kubernetes.io/component":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9091,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":9092,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}},"k:{\"port\":10901,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:sessionAffinityConfig":{".":{},"f:clientIP":{".":{},"f:timeoutSeconds":{}}},"f:type":{}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:web,Protocol:TCP,Port:9091,TargetPort:{1 0 web},NodePort:0,AppProtocol:nil,},ServicePort{Name:metrics,Protocol:TCP,Port:9092,TargetPort:{1 0 metrics},NodePort:0,AppProtocol:nil,},ServicePort{Name:grpc,Protocol:TCP,Port:10901,TargetPort:{1 0 grpc},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/name: thanos-ruler,thanos-ruler: user-workload,},ClusterIP:<REDACTED-IP>,Type:ClusterIP,ExternalIPs:[],SessionAffinity:ClientIP,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:&SessionAffinityConfig{ClientIP:&ClientIPConfig{TimeoutSeconds:*10800,},},TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[<REDACTED-IP>],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},} from cluster: local
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
[...]
2024/11/25 10:44:48 [DEBUG] Enqueueing Service: <REDACTED>
2024/11/25 10:44:48 [DEBUG] Enqueueing Endpoints: <REDACTED>
[...]
2024/11/25 10:44:48 [DEBUG] Enqueueing Endpoints: <REDACTED>
2024/11/25 10:44:49 [DEBUG] [2024-11-25 10:44:49,618 icontrol.session DEBUG] get WITH uri: https://<REDACTED-URL>:443/mgmt/tm/sys/ AND suffix:  AND kwargs: {}
2024/11/25 10:44:49 [DEBUG] [2024-11-25 10:44:49,619 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): <REDACTED-URL>:443
2024/11/25 10:44:49 [DEBUG]  Initialising controller monitored kubernetes nodes from cluster: local
2024/11/25 10:44:50 [DEBUG]  Initialising controller monitored kubernetes nodes from cluster: local
2024/11/25 10:44:51 [DEBUG] [2024-11-25 10:44:51,101 urllib3.connectionpool DEBUG] https://<REDACTED-URL>:443 "POST /mgmt/shared/authn/login HTTP/1.1" 200 825
2024/11/25 10:44:51 [DEBUG] [2024-11-25 10:44:51,102 icontrol.authtoken DEBUG] Wait for 1 sec after login...
2024/11/25 10:44:52 [DEBUG] [2024-11-25 10:44:52,117 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): <REDACTED-URL>:443
2024/11/25 10:44:52 [DEBUG] [2024-11-25 10:44:52,738 urllib3.connectionpool DEBUG] https://<REDACTED-URL>:443 "GET /mgmt/tm/sys/ HTTP/1.1" 200 4136
2024/11/25 10:44:52 [DEBUG] [2024-11-25 10:44:52,739 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json;charset=utf-8 Content-Encoding: None Text: '{"kind":"tm:sys:syscollectionstate","selfLink":"https://localhost/mgmt/tm/sys?ver=17.1.1.3","items":[{"reference":{"link":"https://localhost/mgmt/tm/sys/application?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/crypto?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-log-settings?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/diags?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/disk?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dynad?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ecm?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/file?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/fpga?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icall?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ipfix?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-config?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/pfman?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sflow?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/software?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/turboflex?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/url-db?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/aom?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/autoscale-group?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/cluster?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/config?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/core?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-ha?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/datastor?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/db?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dns?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/feature-module?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/folder?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/global-settings?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ha-group?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/httpd?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/iapp-restricted-key?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icontrol-soap?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/internal-proxy?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-rotate?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-dhcp?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ip?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ovsdb?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-proxy-config?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-route?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ntp?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/outbound-smtp?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/provision?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/scriptd?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/service?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/smtp-server?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/snmp?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sshd?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/state-mirroring?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/syslog?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/telemd?ver=17.1.1.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ucs?ver=17.1.1.3"}}]}'
2024/11/25 10:44:52 [DEBUG] [2024-11-25 10:44:52,916 f5_cccl DEBUG] F5CloudServiceManager initialize
2024/11/25 10:44:52 [DEBUG] [2024-11-25 10:44:52,917 f5_cccl.bigip DEBUG] BigIPProxy.__init__()
2024/11/25 10:44:53 [DEBUG] [2024-11-25 10:44:53,018 icontrol.session DEBUG] get WITH uri: https://<REDACTED-URL>:443/mgmt/tm/sys/ AND suffix:  AND kwargs: {}
2024/11/25 10:44:53 [DEBUG] Enqueueing Secrets: <REDACTED>
[...]

cut because of maximum 65536 characters --> see file
[f5-case_only_log.txt](https://github.com/user-attachments/files/17935180/f5-case_only_log.txt)

#### Observations (if any)
trinaths commented 4 days ago

@keennessch Please share sample transport server resources and complete CIS logs to automation_toolchain_pm@f5.com

abvinodu2003 commented 3 days ago

i too have the same issue i'm running OCP 4.14.12 please find the arguments i'm using for CIS 

args: [
              "--bigip-username=$(BIGIP_USERNAME)",
              "--bigip-password=$(BIGIP_PASSWORD)",
              "--bigip-url=192.168.110.167",
              "--bigip-partition=ocpclusterpart",
              "--pool-member-type=cluster",
              "--log-level=AS3DEBUG",
              "--insecure=true",
              "--as3-validation=true",
              "--log-as3-response=true",
              "--static-routing-mode=true",
              "--orchestration-cni=ovn-k8s",
              "--agent=as3",
              "--custom-resource-mode=true",
          ]

please find the transportserver configuration

[cislogs.txt](https://github.com/user-attachments/files/17949850/cislogs.txt)
StevenBarre commented 2 days ago

I'm also experiencing this issue. I've emailed the logs as requested.