search

apache / apisix

The Cloud-Native API Gateway
https://apisix.apache.org/blog/
Apache License 2.0
14.3k stars 2.49k forks source link

help request: apisix connet etcd ipv6 address fail #9737

Closed stubbornTanzhe closed 1 week ago

stubbornTanzhe commented 1 year ago

Description

stateful-set etcd cluster with ipv6 address, when apisix pod init, it failed because connect to etcd domain url(headless service) fail. It shows 

request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known
Warning! Request etcd endpoint 'http://etcd-0.etcd-headless.apisix.svc.cluster.local:2379/version' error, host or service not provided, or not known, retry time=1

curl the etcd url in apisix pod is success. and I change the etcd url from string to ip-address(in the configmap), it shows

/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported
request etcd endpoint 'http://[fd00:c5a6::1b:7478]:2379/version' error, Address family for hostname not supported
request etcd endpoint 'http://[fd00:c5a6::27:3dfb]:2379/version' error, Address family for hostname not supported
all etcd nodes are unavailable
Warning! Request etcd endpoint 'http://[fd00:c5a6::13:61c]:2379/version' error, Address family for hostname not supported, retry time=1

and the openresty version is with ipv6:

[root@5927f3188b04 resty]# /usr/local/openresty/bin/openresty -p /usr/local/apisix -V
nginx version: openresty/1.21.4.1
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1s  1 Nov 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.7 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/
local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/
local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit
-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-m
isc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../sr
cache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0
.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-mod
ule=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpat
h,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl1
11/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.
OdVMn6DSl0/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 -
-add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../a
pisix-nginx-module-1.12.0/src/stream --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../apisix-nginx-module-1.12.0/src/meta --add-mod
ule=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../wasm-nginx-module-0.6.4 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../lua-var-ngin
x-module-v0.5.3 --add-module=/tmp/tmp.OdVMn6DSl0/openresty-1.21.4.1/../grpc-client-nginx-module-v0.4.2 --with-poll_module --with-pcre-jit
 --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_
imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_a
uth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module -
-with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-ipv6 --
with-stream --with-http_ssl_module

now i am confused

  1. init-etcd seems that it just use socket http lib send request(https://github.com/apache/apisix/blob/master/apisix/cli/etcd.lua#L135)
  2. i found some issue which with upstream ipv6 and etcd ipv6 ,but close them already(https://github.com/apache/apisix/issues/7100), i do not get the right solution. in my version(2.15.3), it just NOT work.

etcd yaml:

apiVersion: v1
data:
  jwt-token.pem: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS2dJQkFBS0NBZ0VBN1d6LzVNb3AzVHF0QWNMeFVMU2p5Mm9SbmhJSXBvTndBV2pLb2Uxb08reDFlKzMzCjc1dG1BTU91SmhxS0c3TGhIdFVxWENiQXZlYWRyZ09QTnJ4MGVLS0J2Ni9sY0FBbUh2Y2FVdyt3d2lYNFFxWXgKSWE0NTVGODkyL3pBT2YwUm5UMVVjUXJQbDR3SDZWWndvY3QrNzhZL3lDaWRQcGpoWTNra0IveWszUmlxUVFMMwpqR3NHMXZLYURNUEl3bFgxc0xkRUdzTjhjN1pvd0x5SWtCc1BiSWlYSHdPeWE3Mkp2cVRZc2dydWw5Ri8yRFl1ClBWZEhaMzREdTBpd1NVNCt1N1lQS0lFYUp6RmdSd3c2czJXYlZBQXVEN1h6LzU2anMvTm5uZEFNWHlYT1RReGsKTnQwZlZVS2R5SWNPdGJGK2w0blFnNG1RVXBDS3MvdGhHYzFzY1hMWHpqWVJaclo3NjFkVFRYSm15UWRWbWhOYgpaemZ2SzJEZ3ppYkZBcStPQ29PaWE2TlByYUhTZEFsejlWcS92MFB4T0RlbXZDR2Q1clg2VHF4eUdqcGtGcWxOCmdNbVpWR2pHdVBEWXN5TVpjUDZOWEVoUFFhWmRWNzY5Q1kyY0xlSWVsWk1LNC9TUDNOL0RiU3E2cEdZaDhOV1kKV3pyV2k0UWtyUVI0UnBlOGhGL1FRbmo5RnpVTUxlR05zUG91L0JnTGJCbDIyN2E3NmdlUEsrMmp2VW5UMXlCVApLbzJmWFlnYTgxVEhpYzhOR2VCSDlrU1VQMThXVUVTcGI2cXVJTVQ5dHAzTmx6SDdYc3E4TnNqQlEwMVJqb2VCCm9Tc1hYcDZteWhJdU90QjZINkFLbTJwTitCU0lrRW9XL0RUYlJwTllkdE5FSjRpT3BWZFNYaFE1d1dzQ0F3RUEKQVFLQ0FnRUE0R3VHV2NGSjN3WkdZQUxmVExUaW1qU3hZR09WYjZlUHFKVmY2NzFYS0VVeks2aGRFa3dtRjVUZgpPTDhmU2wxRDgzdjBaVlJ3SFZBdVBUeTZFemdCTThJUHVKdlAySlViYzJ1SW8zaVNVSUhKYjE1ZDFJTTF6NHJECkV2eU95b0ZPTS8yem9vSmdoTlBrNm9jOVFZSHpnenMyTU1GTEdNOHpSRzVPUTRwb1gyQ2dGWUZ6Y25QNzdCTTIKejlnRUhNVlp6ZWI4NWZ0ZjJKUlQ0ZEtUbHZzdEVoN3VFVy9TSjRycDk1MGtBdTdZWmdLdHdEWStjTmJkOUpJbApmdEtQWjBzaGV4YjByd21SS2ZpL0U0a1l0MWZVZ1lzRi91MnpMQ3pNZWgwYUZRNkJrekpPRllBZi9jZjNoUVRKCjFiRHJrNHlwcHgycVRac3JJdy9qTzRyaVZweFk0ZlVjOW5MeVJPeG1CaC9uTHlFNFVEQWhZSGMwZjZhUWxYSVYKcGtpZ3d4ZENrditRNm9uaUZLUFU0RDF3VlhQOVVxZ2ltaFJ5V2tPcmxXUGZWa09laWIwek03Z3NvVXBtRGJ4TAoyOXAweHB6WVBPQ3REbXVXTEZxZWZmSUFFM0g5UFowWHVRUlJPQTV3cHlBdUVqK1U1cjJPMGQ5T0ZLQVA0akpXCmM3aHEwK29pZHhHNmZMK3VJZ0d1T1hPZW9uYU5jNFQ2NVZPbG9EQWkrdkZycUlXVktSbm9qMjFLdytyYzg0aHoKK2tPTktnbHVjZ2JjbUgrb1AvanU3dklTdW8raXpRUFViZnZ2alhwRThxc1l0eW1UY1J2UW5pd2I2c3h0ekNBVgpuZVE5ZElZMGYzT2JvQitucHBQMUFIZWszZlJLN1hZZ2c5eTJ4eE1IdFpyRitvUXNpaGtDZ2dFQkFPNHNzVnVLCmhLcWNJcmdkZUc2OEg0TGNtT09GWHRyZlRMNTVzTS83STI5UWhjMmczc3pYYXZoM1BndEo1ekxJeDMwdWdKdWYKN1h6Snh2Tjl2RGs1SUlVTXc3SnRJYXZKNGVQWks2L0lHZUVkOXhWTUF2NTIrVE5rOFJSbW5oeW5ua0FPNlkxcwpGM21GTURmblVxbFhzODRCVlM1N1Y3Z3RZMXV2TWpOMHRSdmdjNkxFeStpd1NQYTRnUXM2aFY4R3dMSVk3ekQvCmJHWU93bVgxNjhwbVBhbXFxOXIyUU1ubnNiWFhJb3R5aVZ1YzFwY3B5aFdqOHI1MWw4RFR6eVpEVUZWcTlXcC8KOFdkYXZ3OURwS0srckIxcEM5dFJadXF2ZjNQTkJ2UjdXdnZNS3FXTCtGZi9LdXhTMk9yenRCSXBSeVJJQTNwNgowTnBDbVVNenJObDRBRGNDZ2dFQkFQOHg5Y255dlVXbHl4b01xKzVLaTVpS3ppVzBBb1loWlZVajFLN3BRNktPCkwxeEs1UHg0WS9aTmNlNGs0c3F4azBkZmpCT3J1eFdYSk1oMGhYSHhPVzhBUDBSYU1waWc5YVh4MWkwVXp6anIKbEZLZWtoNzNrQWpFcHBYN0FGcVRHOHF2NUJ2azArRWI1cGhPNFM4ZFlSQngxdzdSKzYvVisxS3JuYm1LOTE1RQoxbVBWTWtzQ2dlNnFwUGs4aHhWeHVhcStrOWR3eVVVSXhQR2w1NW5BU0hyN1JYaldYa3pkallIZk8vZHFxSGwwCkc3MWx2QmN5dUMwOERsMW5LVE5KQlAvY0pEeGVNVlVha25NM00yN2FNeitPSmRnTUE2YVV1V3pYS3NIWHdBeHkKeWM0ZWJwZEdKcVkrRGRhNGNLckV6VTQ4T2hTM1l6cGxZcXlyemtqeXBtMENnZ0VCQU5PSFUvbEdHNjlldmlNZwo0VjRQckRqUDdPVmRCVEtFVWFkMWNvZzB0bkxkWkFpTlVITkMrdGt6SmlKbWYzU0dCMDV4WjMxUDIxOFI0YVZOClRVYlJLc1dmNFlwdERCT0RXQ0RCTnVDR2FoMmFQR0Jvd3R2M0ZEb0Y4MnEzL01MY1Ixc0RJNEFidVBtUFJaVFEKMklSWHhQUTRFTXdZZFg2NHFONjd5VzBUd09uQ1BWRVpVVFFXcEthaWxORkJKMFNQUGNLdkRIaEswWjRPaTY1OAphKzYxUVJ5WDNNdTZHeTUyblVrdmlabXBucjZEbTc1ak9meGdRRzNSQ0hjVVpveklDZWZpOTVXZmplbktHWmZYCjdyY3Zlck1nSVl5bVRpNHVzWlpLU0Z3SDFuVjlEWDg3d2ROOVowUytDN01Yd0J6RkZrN3R1bEVrRkt0VHJNU1gKZDNNbzlaOENnZ0VCQUptaFhCVHRtMkI0aFNMemlmWDB6c0ZRbnZZM3ZtVTlhYUd0NW5ZK1c4ZGF6Y3hFRWtLagp2NW9oMlhyQ25mM2tsWU9jVTlucndyOG10TEF3NWIxSXVZakhuMDdvOWhqSW5kbi9FeThrbmZQb2J1eW1KZFdhCnVBMXZSZEo0dnlmSDlDMDdZcG9nVWlYdEJBK2hQUk4rSGxjbUVaQU1mZmJIWFh2UmNTeW9LbStJQllFb2NoU1MKTitLVXZLaUc5ZFBGR1Q5ZlorV0ZNc1hxbDVYYWlPa1l4d215aDRwTms0dTJ2Vi80SWtPNjVCM3A5bUU1QTlqNwpZY0ZwckVReXZLenhRcDg5eWxyRllmR1lBNTUxRnZPZlRNYndMbnc0RDJLVG1tV1p5MVhUS3Z6VGhnWWViL21GCjgvS1UzUVB0R1hiVTc3d3ZxYythNzVQU1FXc3VLd3ZqZUtrQ2dnRUFWMHlBbG9YQ2hicnQrYml3MTVsek50dmwKUktWbFlyeXNoWlJ4VTNoRU9yTllPY042WkwxazZLWnUySVhWckJBVjlQNlVPbmdaK1VxUVpibHluSGVZcTZteApDWU1HSjVaNVZkYzJEYXVVaUNkczRybkd5U0pKWGFpUC82WVlxbEJvWkMvdnlHK3dBTy9ybThtdFlrR1hhdEc5CkRqMkJqd2l5Z0ROQUlKZ2l0OHZYTWM0RFlvK3Z5WkJUZVRNb21RNkFITDJiNUc2NE81ZWxHU08yZzd3SnJ6QysKdjhCZUIyTCs3U0x0amNVTEsydktoWFNnVE9EZTUwWWU4UVFBVTRYM1lIQStPc3pGa1pLWVNGaEtnV3hzNDdzUApIRGU1ZS92NjFNRDFnbkdaVjBrS1hIRHhwVWExNkpTKzlya3Z2UEF6aVQwNzltUnJ4YTliUnhmd2lZS3FQUT09Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  labels:
    app.kubernetes.io/instance: etcd
    app.kubernetes.io/name: etcd
  name: etcd-jwt-token
  namespace: apisix
type: Opaque
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  generation: 1
  labels:
    app.kubernetes.io/instance: etcd
    app.kubernetes.io/name: etcd
  name: etcd
  namespace: apisix
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: etcd
      app.kubernetes.io/name: etcd
  serviceName: etcd-headless
  template:
    metadata:
      annotations:
        checksum/token-secret: 60156acd10c1cc700c72beb227921dd1c20bf596700362f42d7ab98b32f67a44
        prometheus.io/port: "2379"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/instance: etcd
        app.kubernetes.io/name: etcd
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: etcd
                  app.kubernetes.io/name: etcd
              namespaces:
              - apisix
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: ETCDCTL_API
          value: "3"
        - name: ETCD_ON_K8S
          value: "yes"
        - name: ETCD_START_FROM_SNAPSHOT
          value: "no"
        - name: ETCD_DISASTER_RECOVERY
          value: "no"
        - name: ETCD_NAME
          value: $(MY_POD_NAME)
        - name: ETCD_DATA_DIR
          value: /bitnami/etcd/data
        - name: ETCD_LOG_LEVEL
          value: info
        - name: ALLOW_NONE_AUTHENTICATION
          value: "yes"
        - name: ETCD_AUTH_TOKEN
          value: jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
        - name: ETCD_ADVERTISE_CLIENT_URLS
          value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2379
        - name: ETCD_LISTEN_CLIENT_URLS
          value: http://[::]:2379
        - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
          value: http://$(MY_POD_NAME).etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380
        - name: ETCD_LISTEN_PEER_URLS
          value: http://[::]:2380
        - name: ETCD_INITIAL_CLUSTER_TOKEN
          value: etcd-cluster-k8s
        - name: ETCD_INITIAL_CLUSTER_STATE
          value: new
        - name: ETCD_INITIAL_CLUSTER
          value: etcd-0=http://etcd-0.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local:2380
        - name: ETCD_CLUSTER_DOMAIN
          value: etcd-headless.$(MY_POD_NAMESPACE).svc.cluster.local
        - name: MY_STS_NAME
          value: etcd
        image:  {{ .harbor_k_host }}/{{ .image_etcd }}
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /opt/bitnami/scripts/etcd/prestop.sh
        livenessProbe:
          failureThreshold: 5
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          tcpSocket:
            port: 2379
          timeoutSeconds: 5
        name: etcd
        ports:
        - containerPort: 2379
          name: client
          protocol: TCP
        - containerPort: 2380
          name: peer
          protocol: TCP
        resources:
          requests:
            cpu: 1
            memory: 1G
            ephemeral-storage: 10Gi
          limits:
            cpu: 1
            memory: 1G
            ephemeral-storage: 10Gi
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami/etcd
          name: data
        - mountPath: /opt/bitnami/etcd/certs/token/
          name: etcd-jwt-token
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etcd-jwt-token
        secret:
          defaultMode: 256
          secretName: etcd-jwt-token
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
      storageClassName: nfs-controller
      volumeMode: Filesystem
    status:
      phase: Pending
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  labels:
    app.kubernetes.io/instance: etcd
    app.kubernetes.io/name: etcd
  name: etcd-headless
  namespace: apisix
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: client
    port: 2379
    protocol: TCP
    targetPort: client
  - name: peer
    port: 2380
    protocol: TCP
    targetPort: peer
  selector:
    app.kubernetes.io/instance: etcd
    app.kubernetes.io/name: etcd
  sessionAffinity: None
  type: ClusterIP

Environment

Revolyssup commented 1 year ago

@stubbornTanzhe This fix/feat https://github.com/apache/apisix/issues/7100 is not available in 2.15.3. Here is a request to cherry pick this to 2.15.3 - https://github.com/apache/apisix/pull/8245#issuecomment-1504445848

Revolyssup commented 1 year ago

@stubbornTanzhe This is available in 3.2.0

moonming commented 1 year ago

@stubbornTanzhe do you have time to backport this to 2.15.3?

stubbornTanzhe commented 1 year ago

@stubbornTanzhe This is available in 3.2.0

thank you so much

stubbornTanzhe commented 1 year ago

@stubbornTanzhe do you have time to backport this to 2.15.3?

really not, and I found that maybe there is some tricky thing with the backport, such as i just replace the patch.lua(just with the change code) and not work.

I will REALLY appreciate that if any of you backport this code(and update the rpm)

because 2.15.3 is a big version with no aggressive step and ipv6 feature is a big section in Independent and Controllable Nationalization ;)

stubbornTanzhe commented 1 year ago

can you do the backport thing? or can you give some advice about the patch.lua enable mechanism? @moonming @Revolyssup

github-actions[bot] commented 3 weeks ago

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

github-actions[bot] commented 1 week ago

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.