search

F5Networks / k8s-bigip-ctlr

Repository for F5 Container Ingress Services for Kubernetes & OpenShift.
Apache License 2.0
357 stars 195 forks source link

CIS 2.16.1 crash in startup after using namespace-label #3434

Closed kylinsoong closed 4 months ago

kylinsoong commented 4 months ago

Setup Details

CIS Version :2.16.1
Build: f5networks/k8s-bigip-ctlr:2.16.1
BIGIP Version: Big IP v15.1 AS3 Version: 3.50 Agent Mode: AS3 Orchestration: K8S Pool Mode: Cluster

Description

CIS 2.16.1 crash in startup with below error:

2024/05/23 10:46:28 [INFO] [INIT] Starting: Container Ingress Services - Version: 2.16.1, BuildInfo: azure-5932-c0934efcc07227fdc64bd9c8e17e8cc21a4bc3b2
2024/05/23 10:46:28 [INFO] ConfigWriter started: 0xc0005d5cb0
2024/05/23 10:46:28 [DEBUG] [CCCL] ConfigWriter (0xc0005d5cb0) writing section name global
2024/05/23 10:46:28 [DEBUG] [CCCL] ConfigWriter (0xc0005d5cb0) successfully wrote section (global)
2024/05/23 10:46:28 [DEBUG] [CCCL] ConfigWriter (0xc0005d5cb0) writing section name bigip
2024/05/23 10:46:28 [DEBUG] [CCCL] ConfigWriter (0xc0005d5cb0) successfully wrote section (bigip)
2024/05/23 10:46:28 [INFO] Started config driver sub-process at pid: 11
2024/05/23 10:46:28 [INFO] [INIT] Creating Agent for as3
2024/05/23 10:46:28 [INFO] [AS3] Initializing AS3 Agent
2024/05/23 10:46:28 [DEBUG] [INIT] Invalid trusted-certs-cfgmap option provided.
2024/05/23 10:46:28 [DEBUG] [CORE] Agent Response Worker started and blocked on channel  0xc0004c60c0
2024/05/23 10:46:28 [DEBUG] [AS3] No certs appended, using only system certs
2024/05/23 10:46:28 [DEBUG] [AS3] Validating AS3 schema with  as3-schema-3.50.0-5-cis.json
2024/05/23 10:46:28 [DEBUG] [AS3] posting GET BIGIP AS3 Version request on https://192.168.71.100/mgmt/shared/appsvcs/info
2024/05/23 10:46:29 [DEBUG] [2024-05-23 10:46:29,459 icontrol.session DEBUG] get WITH uri: https://192.168.71.100:443/mgmt/tm/sys/ AND suffix:  AND kwargs: {}
2024/05/23 10:46:29 [DEBUG] [2024-05-23 10:46:29,461 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): 192.168.71.100:443
2024/05/23 10:46:30 [DEBUG] [2024-05-23 10:46:30,088 urllib3.connectionpool DEBUG] https://192.168.71.100:443 "POST /mgmt/shared/authn/login HTTP/1.1" 200 725
2024/05/23 10:46:30 [DEBUG] [2024-05-23 10:46:30,089 icontrol.authtoken DEBUG] Wait for 1 sec after login...
2024/05/23 10:46:30 [DEBUG] [AS3] BIGIP is serving with AS3 version: 3.50.2
2024/05/23 10:46:30 [DEBUG] [AS3] Cleaning Partition k8s 

2024/05/23 10:46:30 [DEBUG] [AS3] posting request to https://192.168.71.100/mgmt/shared/appsvcs/declare/k8s
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,093 urllib3.connectionpool DEBUG] Starting new HTTPS connection (1): 192.168.71.100:443
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,181 urllib3.connectionpool DEBUG] https://192.168.71.100:443 "GET /mgmt/tm/sys/ HTTP/1.1" 200 4100
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,182 icontrol.session DEBUG] RESPONSE::STATUS: 200 Content-Type: application/json;charset=utf-8 Content-Encoding: None Text: '{"kind":"tm:sys:syscollectionstate","selfLink":"https://localhost/mgmt/tm/sys?ver=15.1.10.3","items":[{"reference":{"link":"https://localhost/mgmt/tm/sys/application?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/crypto?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-log-settings?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/diags?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/disk?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dynad?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ecm?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/file?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/fpga?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icall?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ipfix?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-config?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/pfman?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sflow?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/software?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/turboflex?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/url-db?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/aom?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/autoscale-group?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/cluster?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/config?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/core?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/daemon-ha?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/datastor?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/db?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/dns?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/feature-module?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/folder?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/global-settings?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ha-group?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/httpd?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/icontrol-soap?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/internal-proxy?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/log-rotate?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-dhcp?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ip?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-ovsdb?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-proxy-config?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/management-route?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ntp?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/outbound-smtp?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/provision?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/scriptd?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/service?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/smtp-server?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/snmp?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/sshd?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/state-mirroring?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/syslog?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/telemd?ver=15.1.10.3"}},{"reference":{"link":"https://localhost/mgmt/tm/sys/ucs?ver=15.1.10.3"}}]}'
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,184 __main__ DEBUG] config handler thread start
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,187 __main__ DEBUG] config handler woken for reset
2024/05/23 10:46:31 [DEBUG] [2024-05-23 10:46:31,187 __main__ DEBUG] loaded configuration file successfully
2024/05/23 10:46:31 [INFO] [2024-05-23 10:46:31,189 __main__ INFO] entering inotify loop to watch /tmp/k8s-bigip-ctlr.config1074436415/config.json
2024/05/23 10:46:37 [DEBUG] [AS3] Raw response from Big-IP: map[declaration:map[class:ADC controls:map[archiveTimestamp:2024-05-23T10:46:31.009Z class:Controls userAgent:CIS/v2.16.1 K8S/v1.16.2] id:urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d k8s:map[Shared:map[class:Application template:shared] class:Tenant defaultRouteDomain:0] label:CIS Declaration remark:Auto-generated by CIS schemaVersion:3.50.0 updateMode:selective] results:[map[code:200 declarationId:urn:uuid:85626792-9ee7-46bb-8fc8-4ba708cfdc1d host:localhost message:no change runTime:800 tenant:k8s]]] 
2024/05/23 10:46:37 [DEBUG] [AS3] Response from BIG-IP: code: 200 --- tenant:k8s --- message: no change
2024/05/23 10:46:37 [DEBUG] Posting GET BIGIP Reg Key request on https://192.168.71.100/mgmt/tm/shared/licensing/registration
2024/05/23 10:46:37 [INFO] [CORE] Registered BigIP Metrics
2024/05/23 10:46:37 [DEBUG] [CORE] Creating new app informer
2024/05/23 10:46:37 [INFO] [CORE] Not watching Ingress resources.
2024/05/23 10:46:37 [INFO] [CORE] Watching ConfigMap resources.
2024/05/23 10:46:37 [INFO] [CORE] Handling ConfigMap resource events.
2024/05/23 10:46:37 [INFO] [CORE] Not handling Ingress resource events.
2024/05/23 10:46:37 [INFO] [CORE] Not watching Ingress resources.
2024/05/23 10:46:37 [INFO] [CORE] Watching ConfigMap resources.
2024/05/23 10:46:37 [INFO] [CORE] Handling ConfigMap resource events.
2024/05/23 10:46:37 [DEBUG] [CORE] Finished syncing namespace bigip-ctlr-ns-17 (101.029515ms)
2024/05/23 10:46:37 [DEBUG] [CORE] Creating new app informer
2024/05/23 10:46:37 [INFO] [CORE] Not handling Ingress resource events.
2024/05/23 10:46:37 [INFO] [CORE] Not watching Ingress resources.
2024/05/23 10:46:37 [INFO] [CORE] Watching ConfigMap resources.
2024/05/23 10:46:37 [INFO] [CORE] Handling ConfigMap resource events.
2024/05/23 10:46:37 [DEBUG] [CORE] Finished syncing namespace bigip-ctlr-ns-5 (101.190166ms)
2024/05/23 10:46:37 [DEBUG] [CORE] Creating new app informer
2024/05/23 10:46:37 [INFO] [CORE] Not handling Ingress resource events.
2024/05/23 10:46:38 [DEBUG] [CORE] Finished syncing namespace bigip-ctlr-ns-7 (600.741439ms)
2024/05/23 10:46:38 [DEBUG] [CORE] Creating new app informer
2024/05/23 10:46:38 [INFO] [CORE] Not watching Ingress resources.
2024/05/23 10:46:38 [INFO] [CORE] Watching ConfigMap resources.
2024/05/23 10:46:38 [INFO] [CORE] Handling ConfigMap resource events.
2024/05/23 10:46:38 [INFO] [CORE] Not handling Ingress resource events.
2024/05/23 10:46:39 [DEBUG] [CORE] Finished syncing namespace bigip-ctlr-ns-14 (801.02231ms)
2024/05/23 10:46:39 [DEBUG] [CORE] Creating new app informer
E0523 10:46:39.315020       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 138 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x16c0da0?, 0x27057f0})
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0004e1380?})
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x16c0da0, 0x27057f0})
    /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).getQueueLength(0xc0000f2380)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1297 +0xd5
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).processNextVirtualServer(0xc0000f2380)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1366 +0x1b7
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).virtualServerWorker(0xc0002d06a0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1262 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
2024/05/23 10:46:39 [INFO] [CORE] Not watching Ingress resources.
2024/05/23 10:46:39 [INFO] [CORE] Watching ConfigMap resources.
2024/05/23 10:46:39 [INFO] [CORE] Handling ConfigMap resource events.
2024/05/23 10:46:39 [INFO] [CORE] Not handling Ingress resource events.
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x1b39060, 0xc0009daf30}, 0x1, 0xc000323020)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002d07b0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x898165?, 0x0?, 0x0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).runImpl
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1207 +0x24a
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x140fd15]

goroutine 138 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0004e1380?})
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd7
panic({0x16c0da0, 0x27057f0})
    /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).getQueueLength(0xc0000f2380)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1297 +0xd5
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).processNextVirtualServer(0xc0000f2380)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1366 +0x1b7
github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).virtualServerWorker(0xc0002d06a0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1262 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x1b39060, 0xc0009daf30}, 0x1, 0xc000323020)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0002d07b0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x898165?, 0x0?, 0x0?)
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/F5Networks/k8s-bigip-ctlr/v2/pkg/appmanager.(*Manager).runImpl
    /go/src/github.com/F5Networks/k8s-bigip-ctlr/pkg/appmanager/appManager.go:1207 +0x24a

Steps To Reproduce

1) Use the below yaml to start up CIS

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bigip-ctlr
  labels:
    app: bigip-ctlr
  namespace: bigip-ctlr
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bigip-ctlr
  template:
    metadata:
      name: bigip-ctlr
      labels:
        app: bigip-ctlr
    spec:
      serviceAccountName: bigip-ctlr
      containers:
        - name: bigip-ctlr
          #image: "cloudadc/k8s-bigip-ctlr:2.16.1.2"
          image: "f5networks/k8s-bigip-ctlr:2.16.1"
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            exec:
              command:
              - curl
              - -k
              - -s
              - -o
              - /dev/null
              - https://192.168.71.100/mgmt/shared/appsvcs/info
            initialDelaySeconds: 15
            periodSeconds: 15
            timeoutSeconds: 5
            successThreshold: 1
          env:
            - name: BIGIP_USERNAME
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: username
            - name: BIGIP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: bigip-login
                  key: password
          command: ["/app/bin/k8s-bigip-ctlr"]
          args: [
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--log-level=DEBUG",
            "--manage-ingress=false",
            "--log-as3-response=true",
            "--manage-configmaps=true",
            "--bigip-url=192.168.71.100",
            "--insecure=true",
            "--hubmode=true",
            "--node-poll-interval=360",
            "--periodic-sync-interval=36000",
            "--filter-tenants=true",
            "--log-as3-response=true",
            "--bigip-partition=k8s",
            "--pool-member-type=cluster",
            "--namespace-label=cis.f5.com/zone=zone-1",
            ]
trinaths commented 4 months ago

Created [CONTCNTR-4745] for internal tracking.

vklohiya commented 4 months ago

Fixed the issue. It will be available in CIS 2.17.0