MirtoBusico
commented
2 years ago Hi @tao12345666333 the problem appeared after 5 cluster restarts How I've done the tests: 1) start the cluster and wait everything goes green 2) inspect the upstream address with apisix dashboard 3) shutdown the cluster.
I've done this 5 times. In the first four times in the upstream there is the service address; but in the fifth attempt appears the POD address
In apisix dashboard
The kubernetes addresses
Nothing in the apisix log (UTC time -1 hour respect to dashboard):
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/f3118c88 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/23ef4d0 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/4a5806af HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/6b312a6d HTTP/1.1" 404 39 0.003 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/7dc55665 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"Nothing in the apisix ingress controller log:
2022-02-16T13:05:35.738Z | [GIN] 2022/02/16 - 21:05:35 | 200 | 38.15µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:39.633Z | [GIN] 2022/02/16 - 21:05:39 | 200 | 27.66µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:45.738Z | [GIN] 2022/02/16 - 21:05:45 | 200 | 38.349µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:49.633Z | [GIN] 2022/02/16 - 21:05:49 | 200 | 39.56µs | 127.0.0.6 | GET "/healthz"Recap of framework and installation:
1) three virtual machines cluster with K3S
sysop@m01serv:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
m01kw1 Ready <none> 28d v1.22.5+k3s1 192.168.102.122 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1
m01kw2 Ready <none> 28d v1.22.5+k3s1 192.168.102.123 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1
m01km Ready control-plane,master 28d v1.22.5+k3s1 192.168.102.121 <none> Ubuntu 20.04.3 LTS 5.4.0-99-generic containerd://1.5.8-k3s1
sysop@m01serv:~$ 2) Istio version 1.12.2
sysop@m01serv:~$ istioctl version
client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (16 proxies)
sysop@m01serv:~$ 3) Helm chart version
sysop@m01serv:~$ helm show chart apisix/apisix
apiVersion: v2
appVersion: 2.12.0
dependencies:
- condition: etcd.enabled
name: etcd
repository: https://charts.bitnami.com/bitnami
version: 6.2.6
- alias: dashboard
condition: dashboard.enabled
name: apisix-dashboard
repository: https://charts.apiseven.com
version: 0.4.0
- alias: ingress-controller
condition: ingress-controller.enabled
name: apisix-ingress-controller
repository: https://charts.apiseven.com
version: 0.9.0
description: A Helm chart for Apache APISIX
icon: https://apache.org/logos/res/apisix/apisix.png
maintainers:
- name: tao12345666333
name: apisix
type: application
version: 0.8.2
sysop@m01serv:~$ 4) changes in values.yaml
...
gateway:
type: LoadBalancer
...
tls:
enabled: true
servicePort: 443
containerPort: 9443
existingCASecret: "m01cacert"
certCAFilename: "cert"
http2:
enabled: true
...
discovery:
enabled: true
registry:
dns:
servers:
- "10.43.0.10:53"
...
dashboard:
enabled: true
ingress-controller:
enabled: true5) Apisix installation:
kubectl create ns apisix
kubectl label namespace apisix istio-injection=enabled
kubectl -n apisix create secret generic m01cacert --from-file=cert=./m01ca.pem
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--set ingress-controller.config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 \
--namespace apisix6) route definition:
sysop@m01serv:~/software/apisisx$ cat productpage-ar.yaml
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: productpage
spec:
http:
- name: rule1
match:
hosts:
- www2.m01.net
paths:
- /*
backends:
- serviceName: productpage
servicePort: 9080
resolveGranularity: service
sysop@m01serv:~/software/apisisx$ 7) route installation:
kubectl -n bookinfo apply -f productpage-ar.yaml8) Get upstream from apisix pod shell
bash-5.1# curl http://127.0.0.1:9180/apisix/admin/upstreams/49b914ed -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X GET
{"action":"get","count":1,"node":{"value":{"name":"bookinfo_productpage_9080","update_time":1645015594,"labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"port":9080,"weight":100,"priority":0,"host":"10.42.1.111"}],"id":"49b914ed","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","hash_on":"vars","scheme":"http","type":"roundrobin","pass_host":"pass","create_time":1644952229},"key":"\/apisix\/upstreams\/49b914ed"}}
bash-5.1# Hope this helps. Can I inspect anithing else?
tao12345666333
I found that
resolveGranularity: serviceno longer takes effect after multiple restarts of the APISIX Ingress controllerxref: #853