costela / hcloud-ip-floater

k8s controller for Hetzner Cloud floating IPs
GNU General Public License v3.0
58 stars 11 forks source link

support for kubernets 1.20 #23

Closed ghorofamike closed 3 years ago

ghorofamike commented 3 years ago

Im unable to get this to work on kubernetes 1.20, its unable to assign the ip's. is support for this coming in future? here is the current error:

W1230 10:06:57.131557       1 reflector.go:289] k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: watch of *v1.Pod ended with: too old resource version: 3584 (4248)
W1230 10:27:05.142240       1 reflector.go:289] k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: watch of *v1.Pod ended with: too old resource version: 7132 (11402)
[root@prod-ghorofa-124-fra1o62b7tq1tp relations]# kubectl  logs -n hcloud-ip-floater hcloud-ip-floater-6cb5d5c58b-wtvkd 
W1230 10:06:57.131557       1 reflector.go:289] k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: watch of *v1.Pod ended with: too old resource version: 3584 (4248)
W1230 10:27:05.142240       1 reflector.go:289] k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: watch of *v1.Pod ended with: too old resource version: 7132 (11402)
costela commented 3 years ago

I unfortunately haven't tested this with 1.20 yet. These log entries aren't necessarily a sign of problem though. Do you have any other issues, or a more complete log?

ghorofamike commented 3 years ago

i get no other logs, iv been looking thru the code, and im not seeing any of the log lines i would expect, it literary prints only those lines. so something changed somewhere and its not getting events or so, and no normal execution paths happen that trigger logging. what are your suggestions?

ghorofamike commented 3 years ago

So theres is some operator error involved, i cant tell where, i had not requested debug logs, here they are after changing configs:
Here is a link to the whole logs: fpaste, and here is a snippet:

time="2021-01-06T19:12:11Z" level=info msg="service update" namespace=ingress-nginx service=nginx-ingress
time="2021-01-06T19:12:11Z" level=info msg="service unchanged" namespace=ingress-nginx service=nginx-ingress
time="2021-01-06T19:12:12Z" level=debug msg="--- Response:" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="HTTP/2.0 200 OK" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Access-Control-Allow-Credentials: true" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Access-Control-Allow-Origin: *" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Content-Type: application/json" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Date: Wed, 06 Jan 2021 19:12:12 GMT" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Link: <https://api.hetzner.cloud/v1/floating_ips?label_selector=hcloud%3Dfloater&page=1>; rel=last" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Ratelimit-Limit: 3600" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Ratelimit-Remaining: 3599" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Ratelimit-Reset: 1609960333" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Strict-Transport-Security: max-age=15724800; includeSubDomains" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="Vary: Accept-Encoding" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="X-Correlation-Id: f4b3f932-197a-4532-a6e6-e29cd06b5425" component=hcloud
time="2021-01-06T19:12:12Z" level=debug component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="{" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="  \"floating_ips\": [" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="    {" component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="      \"id\": 272829," component=hcloud
time="2021-01-06T19:12:12Z" level=debug msg="      \"name\": \"lb1-prod\"," component=hcloud
time="2021-01-06T19:12:12Z" level=info msg="floating IPs changed" component=fipcontroller
time="2021-01-06T19:12:12Z" level=info msg="starting reconciliation" component=fipcontroller
time="2021-01-06T19:12:12Z" level=debug msg="ignoring unattached floating IP" component=fipcontroller fip=ggg.ggg.ggg.ggg
time="2021-01-06T19:12:12Z" level=debug msg="ignoring unattached floating IP" component=fipcontroller fip=fff.fff.fff.fff
time="2021-01-06T19:12:12Z" level=info msg="reconciliation done" component=fipcontroller

im unsure what is going on.

ghorofamike commented 3 years ago

This portion:

time="2021-01-08T06:00:22Z" level=debug component=hcloud
time="2021-01-08T06:00:22Z" level=debug component=hcloud
time="2021-01-08T06:00:22Z" level=info msg="floating IPs changed" component=fipcontroller
time="2021-01-08T06:00:22Z" level=info msg="starting reconciliation" component=fipcontroller
time="2021-01-08T06:00:22Z" level=debug msg="ignoring unattached floating IP" component=fipcontroller fip=XXX.XXX.XXX.XXX
time="2021-01-08T06:00:22Z" level=debug msg="ignoring unattached floating IP" component=fipcontroller fip=YYY.YYY.YYY.YYY
time="2021-01-08T06:00:22Z" level=info msg="reconciliation done" component=fipcontroller

Does it mean its the interaction with metallb thats problematic? it seems to see that any of my IP's are attached.

costela commented 3 years ago

Yes, this looks like it could be a metallb issue. What does kubectl get svc say? Is k8s getting IPs from metallb?

And what version of hcloud-ip-floater are you using?

ghorofamike commented 3 years ago

I think i have it resolved, its like most issues, operator error, i mixed up my service "selector", (reversed some hyphenated words). so false alarm. without the selectors, this tool could not see live pods for the service, so it would not attach the ips. its easily confused to be a metallb issue, but its not.