search

Kong / charts

Helm chart for Kong
Apache License 2.0
248 stars 480 forks source link

Kong install fail with error Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "rm": executable file not found in $PATH: unknown #668

Closed kaviap closed 1 year ago

kaviap commented 2 years ago

Events: Type Reason Age From Message

Warning BackOff 18m (x4785 over 17h) kubelet Back-off restarting failed container

randmonkey commented 2 years ago

Could you please tell us which version of kong chart and what values are used in your deployment?

kaviap commented 2 years ago

Sorry for the late reply ...

The Kong version is kong-2.12.0.

Controlplane-Values.yaml


image:
  repository: cscoreeusghacr01.azurecr.io/csdevkong1/kong
  tag: 2.12.0
  pullPolicy: IfNotPresent
  pullSecrets:
   - acrsecret

deployment:
  kong:
    enabled: true

env:
  database: "postgres"
  role: control_plane
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  nginx_worker_processes: "2"
  proxy_access_log: /dev/stdout
  admin_access_log: /dev/stdout
  admin_gui_access_log: /dev/stdout
  portal_api_access_log: /dev/stdout
  proxy_error_log: /dev/stderr
  admin_error_log: /dev/stderr
  admin_gui_error_log: /dev/stderr
  portal_api_error_log: /dev/stderr
  prefix: /usr/local/kong
  PATH: "$PATH:/usr/bin"
  pg_host: "csdeveusapipostgres01.postgres.database.azure.com" # Azure PostgreSQL DB connection
  pg_user:
    valueFrom:
      secretKeyRef:
        name: postgres-db-secrets
        key: user
  pg_password:
    valueFrom:
      secretKeyRef:
        name: postgres-db-secrets
        key: password
  password:
    valueFrom:
      secretKeyRef:
        name: admin-token-secret
        key: admin-token-key
  admin_api_uri: "https://admin.csdev.cp.clearsense.com"
  admin_gui_url: "https://manager.csdev.cp.clearsense.com"
  proxy_url: "https://api.csdev.cp.clearsense.com"
  portal_api_url: "https://devportalapi.csdev.cp.clearsense.com"
  portal_gui_host: "devportal.csdev.cp.clearsense.com"

  portal_gui_protocol: "https"
  portal_gui_use_subdomains: false 
  portal_auth: "basic-auth"
  portal_session_conf:
    valueFrom:
      secretKeyRef:
        name: kong-session-config
        key: portal_gui_session_conf
  #portal_cors_origins: "*"

  smtp_starttls: on

admin:
  enabled: true
  type: NodePort
  annotations:
    konghq.com/protocol: "https"

  http:
    enabled: true
    servicePort: 8001
    containerPort: 8001
    parameters: []

  tls:
    enabled: true
    servicePort: 8444
    containerPort: 8444
    parameters: []
      # - http2

status:
  enabled: true
  http:
    enabled: true
    containerPort: 8100
    parameters: []

  tls:
    enabled: true
    containerPort: 8543
    parameters: []

proxy:
  enabled: false
  type: NodePort
  annotations:
    konghq.com/protocol: "https"

  http:
    enabled: true
    servicePort: 8000
    containerPort: 8000
    parameters: []

  tls:
    enabled: true
    servicePort: 8443
    containerPort: 8443
    parameters:
      - http2
  stream: []

ingressController:
  enabled: true
  env:
    kong_admin_token:
      valueFrom:
        secretKeyRef:
          name: admin-token-secret
          key: admin-token-key

  admissionWebhook:
    enabled: false
    failurePolicy: Fail
    port: 8080

  ingressClass: kong

  rbac:
    # Specifies whether RBAC resources should be created
    create: true

  serviceAccount:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the fullname template
    name:
    # The annotations for service account
    annotations: {}

  installCRDs: false

  # general properties
  livenessProbe:
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 5
    timeoutSeconds: 5
    periodSeconds: 10
    successThreshold: 1
    failureThreshold: 3
  readinessProbe:
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 5
    timeoutSeconds: 5
    periodSeconds: 10
    successThreshold: 1
    failureThreshold: 3
  resources:
    limits:
      cpu: 300m
      memory: 512Mi
    requests:
       cpu: 200m
       memory: 256Mi
  affinity: 
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - clearsense-cp-kong
          topologyKey: kubernetes.io/hostname
        weight: 10  
# -----------------------------------------------------------------------------
# Postgres sub-chart parameters
# -----------------------------------------------------------------------------

postgresql:
  enabled: true # Turned off subchart to use external postgres
  postgresqlUsername: kong_app_sa
  postgresqlDatabase: kong_db
  postgresqlPassword: admin@123
  service:
    port: 5432

# -----------------------------------------------------------------------------
# Miscellaneous parameters
# -----------------------------------------------------------------------------

waitImage:
  repository: bash
  tag: 5
  pullPolicy: IfNotPresent

readinessProbe:
  httpGet:
    path: "/status"
    port: status
    scheme: HTTP
  initialDelaySeconds: 5
  timeoutSeconds: 5
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: "/status"
    port: status
    scheme: HTTP
  initialDelaySeconds: 5
  timeoutSeconds: 5
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "/bin/sleep 15 && kong quit"]

cluster:
  enabled: true
  tls:
    enabled: true
    servicePort: 8005
    containerPort: 8005
clustertelemetry:
  enabled: true
  tls:
    enabled: true
    servicePort: 8006
    containerPort: 8006
secretVolumes:
  - kong-cluster-cert

enterprise:
  enabled: true
  license_secret: kong-enterprise-license
  vitals:
    enabled: true
  portal:
    enabled: true
  rbac:
    enabled: true
    admin_gui_auth: basic-auth
    # If RBAC is enabled, this Secret must contain an admin_gui_session_conf key
    # The key value must be a secret configuration, following the example at
    # https://docs.konghq.com/enterprise/latest/kong-manager/authentication/sessions
    session_conf_secret: kong-session-config
    # If admin_gui_auth is not set to basic-auth, provide a secret name which
    # has an admin_gui_auth_conf key containing the plugin config JSON
    # admin_gui_auth_conf_secret: admin-gui-auth-conf
  # For configuring emails and SMTP, please read through:
  # https://docs.konghq.com/enterprise/latest/developer-portal/configuration/smtp
  # https://docs.konghq.com/enterprise/latest/kong-manager/networking/email
  smtp:
    enabled: false
    portal_emails_from: developer@email.com
    portal_emails_reply_to: developer-support@email.com
    admin_emails_from: noreply-developer@email.com
    admin_emails_reply_to: noreply-developer@email.com
    smtp_admin_emails: noreply-developer@email.com
    smtp_host: smtp.sendgrid.net
    smtp_port: 587
    smtp_auth_type: "login"
    smtp_ssl: nil
    smtp_starttls: true
    auth:
      # If your SMTP server does not require authentication, this section can
      # be left as-is. If smtp_username is set to anything other than an empty
      # string, you must create a Secret with an smtp_password key containing
      # your SMTP password and specify its name here.
      smtp_username: "apikey"
      smtp_password_secret: smtp-password-secret

manager:
  enabled: true
  type: NodePort
  annotations:
    konghq.com/protocol: "https"

  http:
    enabled: true
    servicePort: 8002
    containerPort: 8002
    parameters: []

  tls:
    enabled: true
    servicePort: 8445
    containerPort: 8445
    parameters:
      - http2

portal:
  enabled: true
  type: NodePort
  annotations:
    konghq.com/protocol: "https"

  http:
    enabled: true
    servicePort: 8003
    containerPort: 8003
    parameters: []

  tls:
    enabled: true
    servicePort: 8446
    containerPort: 8446
    parameters:
      - http2

portalapi:
  enabled: true
  type: NodePort
  annotations:
    konghq.com/protocol: "https"

  http:
    enabled: true
    servicePort: 8004
    containerPort: 8004
    parameters: []

  tls:
    enabled: true
    servicePort: 8447
    containerPort: 8447
    parameters:
      - http2

plugins:
  configMaps:
  - name: kong-plugin-license-validate
    pluginName: license-validate

Dataplane-Values.yaml

image:
        repository: cscoreeusghacr01.azurecr.io/csdevkong/kong
  tag: 2.12.0
  pullPolicy: IfNotPresent

env:
  prefix: /kong_prefix/
  database: "off"
  role: data_plane
  lua_ssl_trusted_certificate: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert: /etc/secrets/kong-cluster-cert/tls.crt
  cluster_cert_key: /etc/secrets/kong-cluster-cert/tls.key
  cluster_server_name: clearsense-cpdev-kong-cluster.kong.svc.cluster.local
  cluster_control_plane: clearsense-cpdev-kong-cluster.kong.svc.cluster.local:8005
  cluster_telemetry_endpoint: clearsense-cpdev-kong-clustertelemetry.kong.svc.cluster.local:8006

secretVolumes:
  - kong-cluster-cert

ingressController:
  enabled: false
  installCRDs: false

admin:
  enabled: false
  http:
    enabled: true
proxy:
  enabled: true
portal:
  enabled: false
portalapi:
  enabled: false
manager:
  enabled: false
enterprise:
  enabled: true
  license_secret: kong-enterprise-license
  vitals:
    enabled: true
  portal:
    enabled: false
  rbac:
    enabled: false

# deployment:
#   userDefinedVolumeMounts:
#     - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
#       name: default-token
#   userDefinedVolumes:
#     - name: default-token
#       projected:
#         sources:
#           - serviceAccountToken:
#               expirationSeconds: 7200
#               path: token

# topologySpreadConstraints:
#   - maxSkew: 1
#     topologyKey: topology.kubernetes.io/zone
#     whenUnsatisfiable: ScheduleAnyway
#     labelSelector:
#       matchLabels:
#         app.kubernetes.io/instance: clearsense-dp
#         app.kubernetes.io/name: kong
# resources:
#   limits:
#     cpu: 1
#     memory: 1Gi
#   requests:
#     cpu: 500m
#     memory: 500M
# may need to adjust numbers in prod, but have low for testing
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80
# affinity:
#   podAntiAffinity:
#     preferredDuringSchedulingIgnoredDuringExecution:
#     - podAffinityTerm:
#         labelSelector:
#           matchExpressions:
#           - key: app
#             operator: In
#             values:
#             - clearsense-dp-kong
#         topologyKey: kubernetes.io/hostname
#       weight: 10

plugins:
  configMaps:
  - name: kong-plugin-license-validate
    pluginName: license-validate
randmonkey commented 2 years ago

Edited your reply to make it looks in the right format. the error happened because there was a container wanted to execute rm, but the command does not exist in the image. Could you know which container failed to start because of this error, probably the clear-stale-pid init container in kong deployment? This is the only place we call an rm command in container in charts. The container will use the same image as used in kong-proxy container, which should be the kong gateway image. Seems that you were using your own kong image, which is cscoreeusghacr01.azurecr.io/csdevkong1/kong:2.12.0 (on azure container registry?), is it the same as the our kong image in docker hub? Or you built it yourself? We do not have a kong 2.12.0 image, our default kong image in charts 2.12.0 is kong:2.8. Could you please confirm again what is the image tag and chart version you actually used?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

munjalpatel commented 1 year ago

@randmonkey I have the same issue when I try to run on arm64 nodes.

Here is how my image override looks:

"image": {
  "repository": "kong/kong-gateway",
  "tag": "3.1"
},