search

apache / apisix

The Cloud-Native API Gateway
https://apisix.apache.org/blog/
Apache License 2.0
14.5k stars 2.52k forks source link

bug: stream_proxy upstream cannot resolve properly #8334

Closed wey-gu closed 1 year ago

wey-gu commented 1 year ago

Current Behavior

They could be resolved in apisix pod(am I right?)

bash-5.1# ping nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local -c 1
PING nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local (172.17.0.13): 56 data bytes
64 bytes from 172.17.0.13: seq=0 ttl=64 time=0.123 ms

--- nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.123/0.123/0.123 ms
bash-5.1# nslookup nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local
Server:     10.96.0.10
Address:    10.96.0.10:53

*** Can't find nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local: No answer

Name:   nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local
Address: 172.17.0.13

bash-5.1# hostname
apisix-6d89854bc5-5m788

While it ended up with following errors:

2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] resolver.lua:47: parse_domain(): failed to parse domain: nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local, error: failed to query the DNS server: dns client error: 101 empty record received while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779
2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] upstream.lua:79: parse_domain_for_nodes(): dns resolver domain: nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local error: failed to query the DNS server: dns client error: 101 empty record received while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779
2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] init.lua:965: stream_preread_phase(): failed to set upstream: no valid upstream node while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779

Expected Behavior

Those stream_routes with upstream nodes like nebula-storaged-2.nebula-storaged-headless.default.svc.cluster.local could resolved

Error Logs

2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] resolver.lua:47: parse_domain(): failed to parse domain: nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local, error: failed to query the DNS server: dns client error: 101 empty record received while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779
2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] upstream.lua:79: parse_domain_for_nodes(): dns resolver domain: nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local error: failed to query the DNS server: dns client error: 101 empty record received while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779
2022/11/15 12:26:59 [error] 44#44: *9538531 stream [lua] init.lua:965: stream_preread_phase(): failed to set upstream: no valid upstream node while prereading client data, client: 172.17.0.1, server: 0.0.0.0:9779

Steps to Reproduce

minikube start --addons="default-storageclass" --extra-config=apiserver.service-node-port-range=1-65535
# NebulaGraph
curl -sL nebula-kind.siwei.io/install-on-k8s.sh | bash

#apisix
helm repo add apisix https://charts.apiseven.com
helm repo add bitnami https://charts.bitnami.com/bitnami

helm repo update

helm install apisix apisix/apisix \
  --set gateway.type=NodePort \
  --set gateway.stream.enabled=true \
  --set ingress-controller.enabled=true

kubectl edit ConfigMap apisix
# https://github.com/apache/apisix-helm-chart/issues/348
#      stream_proxy:                 # TCP/UDP proxy
#        only: false
#        tcp:                        # TCP proxy port list
#          - addr: 9100
#            tls: true
#          - addr: 9779
#            tls: true
#          - addr: 9559
#            tls: true

apisix_pod=$(kubectl get po -l \
    "app.kubernetes.io/name=apisix" -o name)

kubectl delete $apisix_pod

helm install apisix-dashboard apisix/apisix-dashboard 

# sign certs in host of minikube
mkcert -CAROOT
mkcert \
  'demo.siwei.io' localhost 127.0.0.1 ::1 \
  $(minikube ip) 10.1.1.168 \
  '*.nebula-storaged-headless.default.svc.cluster.local' \
  '*.nebula-metad-headless.default.svc.cluster.local'

# now add the certs in apisix siwth dashboard

apisix_pod=$(kubectl get po -l \
    "app.kubernetes.io/name=apisix" -o name)

kubectl exec -it $apisix_pod -- \
    curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 \
    -H "X-API-KEY: $apisix_api_key" -X PUT -d \
'{
    "sni": "nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local",
    "upstream": {
        "nodes": {
            "nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local:9779": 1
        },
        "type": "roundrobin"
    }
}'

kubectl exec -it $apisix_pod -- \
    curl http://127.0.0.1:9180/apisix/admin/stream_routes/2 \
    -H "X-API-KEY: $apisix_api_key" -X PUT -d \
'{
    "sni": "nebula-storaged-1.nebula-storaged-headless.default.svc.cluster.local",
    "upstream": {
        "nodes": {
            "nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local:9779": 1
        },
        "type": "roundrobin"
    }
}'

kubectl exec -it $apisix_pod -- \
    curl http://127.0.0.1:9180/apisix/admin/stream_routes/3 \
    -H "X-API-KEY: $apisix_api_key" -X PUT -d \
'{
    "sni": "nebula-storaged-2.nebula-storaged-headless.default.svc.cluster.local",
    "upstream": {
        "nodes": {
            "nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local:9779": 1
        },
        "type": "roundrobin"
    }
}'

kubectl apply -f svc.yaml

# svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: apisix-svc
  labels:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: apisix
spec:
  selector:
    app.kubernetes.io/instance: apisix
    app.kubernetes.io/name: apisix
  ports:
    - protocol: TCP
      port: 9779
      targetPort: 9779
      name: thrift
      nodePort: 9779
  type: NodePort
###########
# expose the service
minikube service apisix-svc

Then call tcp over tls from the host like nebula-storaged-0.nebula-storaged-headless.default.svc.cluster.local:9779

BTW. After I changed upstream to the IP address, everything worked like a charm.

Environment

❯ kubectl exec -it $apisix_pod -- uname -a
Defaulted container "apisix" out of: apisix, wait-etcd (init)
Linux apisix-6d89854bc5-5m788 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 Linux
tzssangglass commented 1 year ago

this is a known issue, ref: https://github.com/apache/apisix/issues/7733#issuecomment-1220852352

spacewander commented 1 year ago

Let's track them in one issue: #7733