search

F5Networks / f5-ipam-controller

The F5 IPAM Controller runs in an orchestration environment like Kubernetes to allocate IP addresses from an IPAM system to BIG-IP Virtual Servers. The purpose is to abstract complexity related to setting up BIG-IP from a networking perspective
Apache License 2.0
10 stars 16 forks source link

FIC f5-bigip-ctlr.k8s.ipam not found #77

Closed andreasjacobi closed 2 years ago

andreasjacobi commented 2 years ago

Setup Details

FIC Version : 0.1.5 CIS Version : 2.6.0
FIC Build: f5networks/f5-ipam-controller:0.1.5 CIS Build: f5networks/k8s-bigip-ctlr:2.6.0
BIGIP Version: Big IP 15.0
AS3 Version: 3.28
Orchestration: K8S
Orchestration Version:
Additional Setup details: Cilium

Description

Deploying a type loadbalancer with the IPAM label results in the following FIC error: [ERROR] Unable to Update IPAM: kube-system/f5-bigip-ctlr.k8s.ipam Error: ipams.fic.f5.com "f5-bigip-ctlr.k8s.ipam" not found

And due to the above the CIS message: IP address not available, yet, for service service: nginx/nginx-lb

Steps To Reproduce

CIS is installed using Helm and all objects that does not require FIC works when manually specifying the IP. Roles are updated according to the doc.

kubectl describe clusterrole f5-bigip-ctlr

  ipams.fic.f5.com/status                         []                 []              [get list watch update create patch delete]
  ipams.fic.f5.com                                []                 []              [get list watch update create patch delete]

kubectl describe clusterrole ipam-ctlr-clusterrole

  ipams.fic.f5.com/status  []                 []              [get list watch update patch delete create]
  ipams.fic.f5.com         []                 []              [get list watch update patch delete create]

kubectl describe ipam -n kube-system 

Name:         f5-bigip-ctlr.k8s.ipam
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>
API Version:  fic.f5.com/v1
Kind:         IPAM
Metadata:
  Creation Timestamp:  2021-10-12T15:36:24Z
  Generation:          45
  Managed Fields:
    API Version:  fic.f5.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:hostSpecs:
      f:status:
    Manager:         k8s-bigip-ctlr.real
    Operation:       Update
    Time:            2021-10-13T08:21:41Z
  Resource Version:  3617083
  UID:               7a38d2ae-77bd-4de7-85d3-f219fa4674ed
Spec:
  Host Specs:
    Ipam Label:  Production
    Key:         nginx/nginx-lb_svc
Status:
Events:  <none>

kubectl describe svc -n nginx

Name:              nginx
Namespace:         nginx
Labels:            name=nginx
                   role=public
Annotations:       <none>
Selector:          name=nginx
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.103.63.53
IPs:               10.103.63.53
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         10.245.2.250:80,10.245.2.254:80,10.245.2.66:80
Session Affinity:  None
Events:            <none>

Name:                     nginx-lb
Namespace:                nginx
Labels:                   app=nginx
Annotations:              cis.f5.com/health: {"interval": 10, "timeout": 31}
                          cis.f5.com/ipamLabel: Production
Selector:                 name=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.105.127.243
IPs:                      10.105.127.243
Port:                     svc-lb1-80  80/TCP
TargetPort:               80/TCP
NodePort:                 svc-lb1-80  32479/TCP
Endpoints:                10.245.2.250:80,10.245.2.254:80,10.245.2.66:80
Port:                     svc-lb1-8080  8080/TCP
TargetPort:               8080/TCP
NodePort:                 svc-lb1-8080  30797/TCP
Endpoints:                10.245.2.250:8080,10.245.2.254:8080,10.245.2.66:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

FIC logs:

2021/10/13 14:51:18 [INFO] [INIT] Starting: F5 IPAM Controller - Version: 0.1.5, BuildInfo: azure-1035-1bb5b0bc70546b7546ad2b1f42405b9aa867de2e
2021/10/13 14:51:18 [DEBUG] Creating IPAM Kubernetes Client
2021/10/13 14:51:18 [DEBUG] [ipam] Creating Informers for Namespace kube-system
2021/10/13 14:51:18 [DEBUG] Created New IPAM Client
2021/10/13 14:51:18 [DEBUG] [MGR] Creating Manager with Provider: f5-ip-provider
2021/10/13 14:51:18 [DEBUG] [STORE] Using IPAM DB file from mount path
2021/10/13 14:51:18 [DEBUG] [STORE] [ipaddress status ipam_label reference]
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.209 1 Stage Uv38ByGCZU8WP18P
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.210 1 Stage lWbHTRADfE17uwQH
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.211 1 Stage gYVa2GgdDYbR6R4A
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.212 1 Stage ZpTSxCKs0gigByk5
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.213 1 Stage 650YpEeEBF2H88Z8
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.214 1 Stage la9aJTZ5Ubqi/2zU
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.215 1 Stage X7kLrbN8WCG22VUm
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.201 0 Production nginx/nginx-lb_svc
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.202 1 Production YyUlP+xzjdep4ov5
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.203 1 Production DwcCRIYVu9oIMT9q
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.204 1 Production C/UFmHWSHmaKW98s
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.205 1 Production ktJXK80GaNLWxS9Q
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.206 1 Production a/hMcXTLdHY2TMPb
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.207 1 Production Fy7YV5S7NYsMO1Jd
2021/10/13 14:51:18 [DEBUG] [STORE] 10.100.100.208 1 Production /wlCedsZROvXoZ0P
2021/10/13 14:51:18 [DEBUG] [PROV] Provider Initialised
2021/10/13 14:51:18 [INFO] [CORE] Controller started
2021/10/13 14:51:18 [INFO] Starting IPAMClient Informer
I1013 14:51:18.234151       1 shared_informer.go:240] Waiting for caches to sync for F5 IPAMClient Controller
2021/10/13 14:51:18 [DEBUG] Enqueueing on Create: kube-system/f5-bigip-ctlr.k8s.ipam
I1013 14:51:18.334297       1 shared_informer.go:247] Caches are synced for F5 IPAMClient Controller 
2021/10/13 14:51:18 [DEBUG] K8S Orchestrator Started
2021/10/13 14:51:18 [DEBUG] Starting Response Worker
2021/10/13 14:51:18 [DEBUG] Starting Custom Resource Worker
2021/10/13 14:51:18 [DEBUG] Processing Key: &{0xc0004d6420 <nil> Create}
2021/10/13 14:51:18 [ERROR] Unable to Update IPAM: kube-system/f5-bigip-ctlr.k8s.ipam    Error: ipams.fic.f5.com "f5-bigip-ctlr.k8s.ipam" not found
2021/10/13 14:51:18 [DEBUG] Updated: kube-system/f5-bigip-ctlr.k8s.ipam with Status. With IP: 10.100.100.201 for Request: 
Hostname:   Key: nginx/nginx-lb_svc IPAMLabel: Production   IPAddr:     Operation: Create
morishitaf5 commented 2 years ago

got same issue.

writemike commented 2 years ago

Same issue here as well. Is anyone looking at this? Does any additional information need to be provided?

trinaths commented 2 years ago

@andreasjacobi CIS is not validated on Cilium CNI. @morishitaf5 and @writemike do you see this issue the same Cilium based setup. Unable to reproduce this issue. However, delete IPAM CRD and restart CIS. after restart CIS will create it again.

andreasjacobi commented 2 years ago

@trinaths Should the CNI affect the operation of CIS? Anyway I tried to delete the CRD and restart CIS and IPAM but that just gave me this: E1102 09:54:45.566901 1 reflector.go:138] github.com/F5Networks/f5-ipam-controller/pkg/ipammachinery/informers.go:35: Failed to watch *v1.IPAM: failed to list *v1.IPAM: the server could not find the requested resource (get ipams.fic.f5.com)

The CRD is not re-created.

writemike commented 2 years ago

@trinaths I'm using a k3s deployment with flannel. Built using the "NGINX Ingress Controller Lab" UDF.

writemike commented 2 years ago

After a few f5-ipam-controller and k8s-bigip-ctlr pods restarts the error went away and IngressLink with ipamLabels started working. Wish I knew what fixed it. Anything I can provide here to help @andreasjacobi ?

CIS ARGs:

args: [
            "--bigip-username=$(BIGIP_USERNAME)",
            "--bigip-password=$(BIGIP_PASSWORD)",
            "--bigip-url=https://10.1.20.10",
            "--insecure=true",
            "--custom-resource-mode=true",
            "--bigip-partition=kubernetes",
            "--pool-member-type=cluster",
            "--flannel-name=/Common/flannel_vxlan",
            "--log-level=INFO",
            "--as3-validation=true",
            "--log-as3-response=true",
            "--ipam=true",
            "--disable-teems"
          ]

IPAM ARGs:

args:
            - --orchestration
            - kubernetes
            - --ip-range
            - '{"Production":"10.1.10.15-10.1.10.16","Default":"10.1.10.17-10.1.10.18"} '
            - --log-level
            - DEBUG`

k3s node info:

kg nodes -o wide

NAME      STATUS     ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
bigip1    NotReady   <none>                 7h58m                  <none>        <none>        <unknown>            <unknown>         <unknown>
knode2    Ready      <none>                 2d18h   v1.21.5+k3s2   10.1.20.111   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   containerd://1.4.11-k3s1
knode1    Ready      <none>                 2d18h   v1.21.5+k3s2   10.1.20.110   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   containerd://1.4.11-k3s1
kmaster   Ready      control-plane,master   2d18h   v1.21.5+k3s2   10.1.20.109   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   containerd://1.4.11-k3s1
andreasjacobi commented 2 years ago

@writemike Thanks for your effort. I couldn't get it to work so I gave up on IPAM for now. I noticed that in my lab I run k8s 1.22 which I guess is not supported so that could be something. I'm not sure if @morishitaf5 made any progress?

posteingang commented 2 years ago

I have quite the exact same setup (except k3s v.1.19) like @writemike with the exact same error. Any help/solution would be great. Everything is working, except services tpye LoadBalancer. :(

mikeoleary commented 2 years ago

I have the same error and I cannot work around it. I get it whether I am using a VirtualServer or a IngressLink.

@writemike you mentioned that the error went away after some restarts. Is it possible you rolled back to FIC version 0.1.4 and edited your clusterrole and ipam crd schema accordingly to make this work?

FIC Build: f5networks/f5-ipam-controller:0.1.5 CIS Build: f5networks/k8s-bigip-ctlr:2.6.0 BIGIP version: 16.1.1 - shouldnt matter, error is in K8s AS3 version: 3.25 - shouldnt matter, error is in K8s Orchestration: K8S Additional Setup details: AKS

trinaths commented 2 years ago

@andreasjacobi - Is CIS upgraded in your setup from an older version to 2.6 ? if so, please delete the ipam crd and restart CIS. CIS will recreate that again.

andreasjacobi commented 2 years ago

@trinaths - You are correct, it is upgraded from older version. I just now tried deleting the CRD and restart the CIS but it doesn't re-create the CRD. Now I just get the CIS error message:

 E1201 08:08:18.868382       1 reflector.go:138] github.com/F5Networks/f5-ipam-controller/pkg/ipammachinery/informers.go:35: Failed to watch *v1.IPAM: failed to list *v1.IPAM: the server could not │
│  find the requested resource (get ipams.fic.f5.com)

kubectl get crd ipams.fic.f5.com Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "ipams.fic.f5.com" not found

I tried uninstall and re-install the controller using Helm but the CRD doesn't get installed.

trinaths commented 2 years ago

@andreasjacobi can you share the come log rather than this 1 line from log.

posteingang commented 2 years ago

I used CIS 2.6.1 at begining. If I delete the IPAM CRD it will re-create after a CIS restart. But IPAM throw the error message above. :(

mikeoleary commented 2 years ago

@trinaths I can easily reproduce this error but cannot work around it. I've emailed my complete logs from CIS and FIC to you, and supporting resource definitions.

andreasjacobi commented 2 years ago

@trinaths Logs emailed to you.

mikeoleary commented 2 years ago

Update. @mdditt2000 helped me solve this. FIC and CIS should both run with the same svc account. I had created a svc account for FIC and another for CIS, with 2 different clusterroles and 2 different clusterrolebindings. After fixing this, my errors went away. I created a small repo with a working demo: https://github.com/mikeoleary/ipam-troubleshooting

mdditt2000 commented 2 years ago

@mikeoleary glad that resolved the issue. Thanks @nandakishorepeddi fir your help!!

mdditt2000 commented 2 years ago

Issue is a misconfiguration of the ServiceAccount and ClusterRoleBindings. I have created the following YouTube video and document on how to troubleshoot and resolve this configuration issue.

https://youtu.be/Zt_fEife19Y

https://github.com/mdditt2000/kubernetes-1-19/blob/master/cis%202.7/ipam/README.md

CIS 2.7 documentation will be updated for additional information. Closing this issue as resolved.