search

apache / apisix-helm-chart

Apache APISIX Helm Chart
https://apisix.apache.org/
Apache License 2.0
231 stars 210 forks source link

Help: Upgrading using Helm Chart 2.0.0 giving ETCD errors in decoupled mode #566

Open NNicoletti opened 1 year ago

NNicoletti commented 1 year ago

EDIT: Added control plane helm chart to better understand the problem.

I was attempting to do 2 things:

  1. Upgrade my data and control planes using 2.0.0, but I'm running into etcd errors on the data_plane side. The control_plane updated just fine.
  2. Add the mqtt-proxy plugin and turn on stream_proxy so I can use it

Error I get is:

2023-06-22T18:18:48.604511973-04:00 request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, temporary failure in name resolution
2023-06-22T18:18:48.604569105-04:00 all etcd nodes are unavailable
2023-06-22T18:18:48.604547077-04:00 Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, temporary failure in name resolution, retry time=1
2023-06-22T18:18:48.604585549-04:00 Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, temporary failure in name resolution, retry time=2

I realize this is a pretty common error, and I ran into it when figuring out the decoupled deployment a lot. I had a working decoupled deployment and didn't think that either of the 2 things I was attempting to change would cause this.

Any help?

In creating this issue, I noticed that there are two deployment spots: apisix.deployment and deployment. Is that on purpose? Which one is the correct to use when using Helm Chart 2.0.0?

Helm charts for both data plane and control plane below. Control plan was successful and I see 3 etcd pods in my cluster.

ONE MORE EDIT: I attempted to install by updating apisix.deployment to use the correct certs/host as well as under deployment and it still failed. Not sure what is correct usage here. Fwiw it also failed when trying to downgrade using Helm Chart 1.4.0 AFTER attempting to use 2.0.0.

Data Plane (failed):

affinity: {}
apisix:
  admin:
    allow:
      ipList:
        - 127.0.0.1/24
    cors: true
    credentials:
      admin: edd1c9f034335f136f87ad84b625c8f1
      secretName: ''
      viewer: 4054f7cf07e344346cd3f287985e76a2
    enabled: true
    externalIPs: []
    ingress:
      annotations: {}
      enabled: false
      hosts:
        - host: apisix-admin.local
          paths:
            - /apisix
      tls: []
    ip: 0.0.0.0
    port: 9180
    servicePort: 9180
    type: ClusterIP
  customPlugins:
    enabled: false
    luaPath: /opts/custom_plugins/?.lua
    plugins:
      - attrs: {}
        configMap:
          mounts:
            - key: the-file-name
              path: mount-path
          name: configmap-name
        name: plugin-name
  deployment:
    certs:
      cert: ''
      cert_key: ''
      certsSecret: ''
      mTLSCACert: ''
      mTLSCACertSecret: ''
    controlPlane:
      cert: ''
      certKey: ''
      certsSecret: ''
      confServerPort: '9280'
    dataPlane:
      controlPlane:
        host: []
        prefix: /apisix
        timeout: 30
    mode: traditional
    role: traditional
  discovery:
    enabled: false
    registry: {}
  dns:
    resolvers:
      - 127.0.0.1
      - 172.20.0.10
      - 114.114.114.114
      - 223.5.5.5
      - 1.1.1.1
      - 8.8.8.8
    timeout: 5
    validity: 30
  enableIPv6: true
  enableServerTokens: true
  extPlugin:
    cmd:
      - /path/to/apisix-plugin-runner/runner
      - run
    enabled: false
  fullCustomConfig:
    config: {}
    enabled: false
  luaModuleHook:
    configMapRef:
      mounts:
        - key: ''
          path: ''
      name: ''
    enabled: false
    hookPoint: ''
    luaPath: ''
  nginx:
    configurationSnippet:
      httpAdmin: ''
      httpEnd: ''
      httpSrv: ''
      httpStart: ''
      main: ''
      stream: ''
    customLuaSharedDicts: []
    enableCPUAffinity: true
    envs: []
    logs:
      accessLog: /dev/stdout
      accessLogFormat: >-
        $remote_addr - $remote_user [$time_local] $http_host \"$request\"
        $status $body_bytes_sent $request_time \"$http_referer\"
        \"$http_user_agent\" $upstream_addr $upstream_status
        $upstream_response_time
        \"$upstream_scheme://$upstream_host$upstream_uri\"
      accessLogFormatEscape: default
      enableAccessLog: true
      errorLog: /dev/stderr
      errorLogLevel: warn
    workerConnections: '10620'
    workerProcesses: auto
    workerRlimitNofile: '20480'
  pluginAttrs: {}
  plugins: []
  prometheus:
    containerPort: 9091
    enabled: false
    metricPrefix: apisix_
    path: /apisix/prometheus/metrics
  router:
    http: radixtree_host_uri
  setIDFromPodUID: false
  ssl:
    additionalContainerPorts: []
    certCAFilename: ''
    containerPort: 9443
    enabled: false
    existingCASecret: ''
    http2:
      enabled: true
    sslProtocols: TLSv1.2 TLSv1.3
  stream_plugins: []
  vault:
    enabled: false
    host: ''
    prefix: ''
    timeout: 10
    token: ''
  wasm:
    enabled: false
    plugins: []
autoscaling:
  enabled: false
  maxReplicas: 100
  minReplicas: 1
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80
  version: v2
dashboard:
  config:
    conf:
      etcd:
        endpoints:
          - apisix-etcd:2379
        password: null
        prefix: /apisix
        username: null
  enabled: false
etcd:
  auth:
    rbac:
      create: false
      rootPassword: ''
    tls:
      certFilename: ''
      certKeyFilename: ''
      enabled: false
      existingSecret: ''
      sni: ''
      verify: true
  enabled: false
  prefix: /apisix
  replicaCount: 3
  service:
    port: 2379
  timeout: 30
externalEtcd:
  existingSecret: ''
  host:
    - http://etcd.host:2379
  password: ''
  secretPasswordKey: etcd-root-password
  user: root
extraEnvVars: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: ''
global:
  imagePullSecrets: []
  cattle:
    systemProjectId: p-bs7km
hostNetwork: false
image:
  pullPolicy: IfNotPresent
  repository: apache/apisix
  tag: 3.3.0-debian
ingress:
  annotations: {}
  enabled: false
  hosts:
    - host: apisix.local
      paths: []
  tls: []
ingress-controller:
  config:
    apisix:
      adminAPIVersion: v3
  enabled: false
initContainer:
  image: busybox
  tag: 1.28
metrics:
  serviceMonitor:
    annotations: {}
    enabled: false
    interval: 15s
    labels: {}
    name: ''
    namespace: ''
nameOverride: ''
nodeSelector: {}
podAnnotations: {}
podDisruptionBudget:
  enabled: false
  maxUnavailable: 1
  minAvailable: 90%
podSecurityContext: {}
priorityClassName: ''
rbac:
  create: true
replicaCount: 1
resources: {}
securityContext: {}
service:
  externalIPs: []
  externalTrafficPolicy: Cluster
  http:
    additionalContainerPorts: []
    containerPort: 9080
    enabled: true
    servicePort: 80
  labelsOverride: {}
  stream:
    enabled: false
    only: false
    tcp:
      - '9100'
    udp: []
  tls:
    servicePort: 443
  type: NodePort
serviceAccount:
  annotations: {}
  create: false
  name: ''
timezone: ''
tolerations: []
updateStrategy: {}
useDaemonSet: false
admin:
  enabled: false
deployment:
  certs:
    cert: tls.crt
    cert_key: tls.key
    certsSecret: apisix-mtls-client-crt
    mTLSCACert: tls.crt
    mTLSCACertSecret: apisix-ca-bundle
  dataPlane:
    controlPlane:
      host:
        - https://apisix-control-plane-control-plane:9280
  mode: decoupled
  role: data_plane
gateway:
  tls:
    certCAFilename: tls.crt
    enabled: true
    existingCASecret: apisix-ca-bundle
plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - file-logger
  - grpc-transcode
  - grpc-web
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
  - server-info
  - mqtt-proxy
serviceMonitor:
  enabled: true
  namespace: api-services

Control Plane (success):

affinity: {}
apisix:
  admin:
    allow:
      ipList:
        - 127.0.0.1/24
    cors: true
    credentials:
      admin: edd1c9f034335f136f87ad84b625c8f1
      secretName: ''
      viewer: 4054f7cf07e344346cd3f287985e76a2
    enabled: true
    externalIPs: []
    ingress:
      annotations: {}
      enabled: false
      hosts:
        - host: apisix-admin.local
          paths:
            - /apisix
      tls: []
    ip: 0.0.0.0
    port: 9180
    servicePort: 9180
    type: ClusterIP
  customPlugins:
    enabled: false
    luaPath: /opts/custom_plugins/?.lua
    plugins:
      - attrs: {}
        configMap:
          mounts:
            - key: the-file-name
              path: mount-path
          name: configmap-name
        name: plugin-name
  deployment:
    certs:
      cert: tls.crt
      cert_key: tls.key
      certsSecret: apisix-mtls-server-crt
      mTLSCACert: ''
      mTLSCACertSecret: ''
    controlPlane:
      cert: tls.crt
      certKey: tls.key
      certsSecret: apisix-mtls-server-crt
      confServerPort: '9280'
    dataPlane:
      controlPlane:
        host: []
        prefix: /apisix
        timeout: 30
    mode: decoupled
    role: control_plane
  discovery:
    enabled: false
    registry: {}
  dns:
    resolvers:
      - 127.0.0.1
      - 172.20.0.10
      - 114.114.114.114
      - 223.5.5.5
      - 1.1.1.1
      - 8.8.8.8
    timeout: 5
    validity: 30
  enableIPv6: true
  enableServerTokens: true
  extPlugin:
    cmd:
      - /path/to/apisix-plugin-runner/runner
      - run
    enabled: false
  fullCustomConfig:
    config: {}
    enabled: false
  luaModuleHook:
    configMapRef:
      mounts:
        - key: ''
          path: ''
      name: ''
    enabled: false
    hookPoint: ''
    luaPath: ''
  nginx:
    configurationSnippet:
      httpAdmin: ''
      httpEnd: ''
      httpSrv: ''
      httpStart: ''
      main: ''
      stream: ''
    customLuaSharedDicts: []
    enableCPUAffinity: true
    envs: []
    logs:
      accessLog: /dev/stdout
      accessLogFormat: >-
        $remote_addr - $remote_user [$time_local] $http_host \"$request\"
        $status $body_bytes_sent $request_time \"$http_referer\"
        \"$http_user_agent\" $upstream_addr $upstream_status
        $upstream_response_time
        \"$upstream_scheme://$upstream_host$upstream_uri\"
      accessLogFormatEscape: default
      enableAccessLog: true
      errorLog: /dev/stderr
      errorLogLevel: warn
    workerConnections: '10620'
    workerProcesses: auto
    workerRlimitNofile: '20480'
  pluginAttrs: {}
  plugins: []
  prometheus:
    containerPort: 9091
    enabled: false
    metricPrefix: apisix_
    path: /apisix/prometheus/metrics
  router:
    http: radixtree_host_uri
  setIDFromPodUID: false
  ssl:
    additionalContainerPorts: []
    certCAFilename: ''
    containerPort: 9443
    enabled: false
    existingCASecret: ''
    http2:
      enabled: true
    sslProtocols: TLSv1.2 TLSv1.3
  stream_plugins: []
  vault:
    enabled: false
    host: ''
    prefix: ''
    timeout: 10
    token: ''
  wasm:
    enabled: false
    plugins: []
autoscaling:
  enabled: false
  maxReplicas: 100
  minReplicas: 1
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80
  version: v2
dashboard:
  config:
    conf:
      etcd:
        endpoints:
          - apisix-etcd:2379
        password: null
        prefix: /apisix
        username: null
  enabled: false
etcd:
  auth:
    rbac:
      create: false
      rootPassword: ''
    tls:
      certFilename: ''
      certKeyFilename: ''
      enabled: false
      existingSecret: ''
      sni: ''
      verify: true
  enabled: true
  prefix: /apisix
  replicaCount: 3
  service:
    port: 2379
  timeout: 30
  host:
    - https://etcd.local:2379
externalEtcd:
  existingSecret: ''
  host:
    - http://etcd.host:2379
  password: ''
  secretPasswordKey: etcd-root-password
  user: root
extraEnvVars: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: ''
global:
  imagePullSecrets: []
  cattle:
    systemProjectId: p-bs7km
hostNetwork: false
image:
  pullPolicy: IfNotPresent
  repository: apache/apisix
  tag: 3.3.0-debian
ingress:
  annotations: {}
  enabled: false
  hosts:
    - host: apisix.local
      paths: []
  tls: []
ingress-controller:
  config:
    apisix:
      adminAPIVersion: v3
  enabled: false
initContainer:
  image: busybox
  tag: 1.28
metrics:
  serviceMonitor:
    annotations: {}
    enabled: false
    interval: 15s
    labels: {}
    name: ''
    namespace: ''
nameOverride: ''
nodeSelector: {}
podAnnotations: {}
podDisruptionBudget:
  enabled: false
  maxUnavailable: 1
  minAvailable: 90%
podSecurityContext: {}
priorityClassName: ''
rbac:
  create: true
replicaCount: 1
resources: {}
securityContext: {}
service:
  externalIPs: []
  externalTrafficPolicy: Cluster
  http:
    additionalContainerPorts: []
    containerPort: 9080
    enabled: true
    servicePort: 80
  labelsOverride: {}
  stream:
    enabled: true
    only: false
    tcp:
      - '9100'
    udp: []
  tls:
    servicePort: 443
  type: NodePort
serviceAccount:
  annotations: {}
  create: false
  name: ''
timezone: ''
tolerations: []
updateStrategy: {}
useDaemonSet: false
admin:
  allow:
    ipList:
      - 127.0.0.1/24
      - 0.0.0.0/0
  ingress:
    hosts:
      - host: apisix-control-plane-admin.local
        paths:
          - /apisix
    tls:
      - hosts:
          - apisix-control-plane-admin.local
        secretName: ''
deployment:
  certs:
    cert: tls.crt
    cert_key: tls.key
    certsSecret: apisix-mtls-client-crt
  controlPlane:
    cert: tls.crt
    certKey: tls.key
    certsSecret: apisix-mtls-server-crt
  mode: decoupled
  role: control_plane
gateway:
  ingress:
    hosts:
      - host: apisix.local
        paths:
          - /apisix
    tls:
      - hosts:
          - apisix.local
        secretName: ''
  tls:
    certCAFilename: tls.crt
    enabled: true
    existingCASecret: apisix-ca-bundle
plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - file-logger
  - grpc-transcode
  - grpc-web
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
  - server-info
  - mqtt-proxy
serviceMonitor:
  enabled: true
tao12345666333 commented 1 year ago

Helm Chart v2.x has many breaking changes, it can't upgrade from v1.x directly

NNicoletti commented 1 year ago

@tao12345666333 ok thank you. I will downgrade back to 1.4.0 to get it up for now. I'll figure out what needs to be done to upgrade it later and do a fresh install once we're ready!