apache / apisix-ingress-controller

APISIX Ingress Controller for Kubernetes
https://apisix.apache.org/
Apache License 2.0
1.02k stars 343 forks source link

bug: plugin_config_name is not working in ApisixRoute CRD #1693

Closed renatomattos2912 closed 1 year ago

renatomattos2912 commented 1 year ago

Current Behavior

When applying CRD for ApisixPluginConfig and ApisixRoute using plugin_config_name field to bound plugins to route, the bound is not being made after all, so each CRD is applied individually in kubernetes, no errors on that, but the plugin_config_name field is being ignored on route.

This problem is very similar to that one: https://github.com/apache/apisix-ingress-controller/issues/1003, but the solution pointed not worked for me.

Expected Behavior

I expect that ApisixRoute bound with ApisixPluginConfig works when plugin_config_name is correctly added to route.

Error Logs

No errors logs

Steps to Reproduce

  1. Run APISIX in a kubernete cluster using Helm with values bellow:
global:
  imagePullSecrets: []

apisix:  
  enabled: true
  enableIPv6: true
  enableServerTokens: true
  setIDFromPodUID: false  
  customLuaSharedDicts:
    []    
  luaModuleHook:
    enabled: false    
    luaPath: ""    
    hookPoint: ""    
    configMapRef:      
      name: ""      
      mounts:        
        - key: ""          
          path: ""

  httpRouter: radixtree_uri

  enableCustomizedConfig: false
  customizedConfig: {}    

  image:    
    repository: apache/apisix    
    pullPolicy: IfNotPresent    
    tag: 3.1.0-debian

  kind: Deployment
  replicaCount: 1  
  priorityClassName: ""  
  podAnnotations: {}
  podSecurityContext:
    {}    
  securityContext:
    {}   

  podDisruptionBudget:
    enabled: false    
    minAvailable: 90%    
    maxUnavailable: 1

  resources:
    {}

  hostNetwork: false
  nodeSelector: {}
  tolerations: []
  affinity: {} 
  timezone: ""
  extraEnvVars: []

nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: false
  annotations: {}
  name: ""

rbac:
  create: false

gateway:
  type: LoadBalancer
  externalTrafficPolicy: Cluster  
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb    
  externalIPs: []  
  http:
    enabled: true
    servicePort: 80
    containerPort: 9080  
  tls:
    enabled: true
    servicePort: 443
    containerPort: 9080    
    existingCASecret: ""    
    certCAFilename: ""
    http2:
      enabled: true    
    sslProtocols: "TLSv1.2 TLSv1.3"  
  stream:
    enabled: false
    only: false
    tcp: []
    udp: []  
  ingress:
    enabled: false    
    annotations:
      {}      
    hosts:
      - host: apisix.local
        paths: []
    tls: []  

admin:  
  enabled: true  
  type: ClusterIP  
  externalIPs: []  
  port: 9180  
  servicePort: 9180  
  cors: true  
  credentials:    
    admin: edd1c9f034335f136f87ad84b625c8f1    
    viewer: 4054f7cf07e344346cd3f287985e76a2

  allow:    
    ipList:
      - 127.0.0.1/24  
  ingress:
    enabled: false    
    annotations:
      {}      
    hosts:
      - host: apisix-admin.local
        paths:
          - "/apisix"
    tls: []  

nginx:
  workerRlimitNofile: "20480"
  workerConnections: "10620"
  workerProcesses: auto
  enableCPUAffinity: true
  envs: []

plugins:
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  - echo
  - fault-injection
  - file-logger
  - grpc-transcode
  - hmac-auth
  - http-logger
  - ip-restriction
  - ua-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  - node-status
  - openid-connect
  - authz-casbin
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - traffic-split
  - gzip
  - real-ip
  - ext-plugin-pre-req
  - ext-plugin-post-req
stream_plugins:
  - mqtt-proxy
  - ip-restriction
  - limit-conn

pluginAttrs: {}

extPlugin:
  enabled: false  
  cmd: ["/path/to/apisix-plugin-runner/runner", "run"]

wasmPlugins:  
  enabled: false
  plugins: []

customPlugins:
  enabled: false  
  luaPath: "/opts/custom_plugins/?.lua"
  plugins:    
    - name: ""      
      attrs: {}      
      configMap:        
        name: ""        
        mounts:
          - key: ""
            path: ""
          - key: ""
            path: ""

updateStrategy:
  {}

extraVolumes: []

extraVolumeMounts: []

extraInitContainers: []

discovery:  
  enabled: false  
  registry:
    {}

logs:  
  enableAccessLog: true  
  accessLog: "/dev/stdout"  
  accessLogFormat: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'  
  accessLogFormatEscape: default
  errorLog: "/dev/stderr"
  errorLogLevel: "warn"

dns:
  resolvers:
    - 127.0.0.1
    - 172.20.0.10
    - 114.114.114.114
    - 223.5.5.5
    - 1.1.1.1
    - 8.8.8.8
  validity: 30
  timeout: 5

initContainer:  
  image: busybox  
  tag: 1.28

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 100
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

configurationSnippet:
  main: |

  httpStart: |

  httpEnd: |

  httpSrv: |

  httpAdmin: |

  stream: |

serviceMonitor:
  enabled: false  
  namespace: ""  
  name: ""  
  interval: 15s  
  path: /apisix/prometheus/metrics  
  metricPrefix: apisix_  
  containerPort: 9091  
  labels:
    {}    
  annotations:
    {}    
etcd:  
  enabled: true  
  host:    
    - http://etcd.host:2379  
  prefix: "/apisix"
  timeout: 30

  auth:
    rbac:      
      create: false      
      user: ""      
      password: ""
    tls:      
      enabled: false    
      existingSecret: ""      
      certFilename: ""      
      certKeyFilename: ""      
      verify: true
      sni: ""

  service:
    port: 2379

  replicaCount: 3

ingress-controller:
  enabled: true
  config:
    apisix:
      adminAPIVersion: "v3"
      serviceNamespace: apisix

vault:
  enabled: false
  host: ""
  timeout: 10
  token: ""
  prefix: ""
  1. Apply CRD to create ApisixPluginConfig

    apiVersion: apisix.apache.org/v2
    kind: ApisixPluginConfig
    metadata:
    name: public-endpoints
    namespace: apisix
    spec:
    plugins:
    - name: cors
      enable: true
      config:
        allow_origins: http://localhost:3000
        allow_credential: true
        allow_methods: "**"
        allow_headers: "**"
        expose_headers: "**"    
  2. Apply CRD to create ApisixRoute bounded to ApisixPluginConfig

    apiVersion: apisix.apache.org/v2
    kind: ApisixRoute
    metadata:
    name: route-test
    namespace: apisix
    spec:
    http:
    - name: rule1
      match:
        hosts:
          - localhost
        paths:
          - /test
        methods:
          - OPTIONS
          - GET
      backends:
        - serviceName: httpbin
          servicePort: 80      
      plugin_config_name: public-endpoints 
  3. Check if plugins is applied or not.

Environment

yujinchoi-94 commented 1 year ago

@tao12345666333 Same here. Here's what I've configured.

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: openapi-alimi-route
  namespace: ingress-apisix
spec:
  http:
  - name: rule-***
    priority: 1
    match:
      hosts:
        {{- include "openapi.hosts" . | indent 8 }}
      paths:
        - *** 
    upstreams:
      - name: cv-upstream
    plugin_config_name: openapi-plugin-config
apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  name: openapi-plugin-config
  namespace: ingress-apisix
spec:
  plugins:
  - name: forward-auth
    enable: true
    config:
      request_method: GET
      uri: ***
      request_headers: ***
      upstream_headers: ***
  - name: prometheus
    enable: true
    config:
      disable: false
      prefer_name: true
  - name: http-logger
    enable: true
    config:
      uri: ***
  - name: proxy-rewrite
    enable: true
    config:
      uri: ***
      headers:
        ***

When I call check a route using ADMIN API neither plugin_config_name nor plugin_config_id appears.

curl 10.192.145.209:9180/apisix/admin/routes/31ceeb97 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' | jq
{
  "value": {
    "labels": {
      "managed-by": "apisix-ingress-controller"
    },
    "hosts": [
      ***
    ],
    "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
    "id": "31ceeb97",
    "update_time": 1679539025,
    "status": 1,
    "create_time": 1679536996,
    "uris": [
      ***
    ],
    "upstream_id": "281b5cf0",
    "name": "ingress-apisix_openapi-alimi-route_rule-alimi",
    "priority": 1
  },
  "modifiedIndex": 62324,
  "key": "/apisix/routes/31ceeb97",
  "createdIndex": 62188
}

However, when I describe ApisixRoute using kubectl, plugin_config_name appears.

k describe apisixroute openapi-alimi-route
Name:         openapi-alimi-route
Namespace:    ingress-apisix
Labels:       argocd.argoproj.io/instance=dev-apisix
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixRoute
Metadata:
  Creation Timestamp:  2023-03-23T02:03:16Z
  Generation:          3
  Managed Fields:
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      apisix-ingress-controller
    Operation:    Update
    Time:         2023-03-23T02:03:16Z
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:argocd.argoproj.io/instance:
      f:spec:
        .:
        f:http:
    Manager:         argocd-application-controller
    Operation:       Update
    Time:            2023-03-23T02:03:16Z
  Resource Version:  542685578
  UID:               9be6e2bc-1b42-4942-91cd-fc5118743cfe
Spec:
  Http:
    Match:
      Hosts:
        ***
      Paths:
        ***
    Name:                ***
    plugin_config_name:  openapi-plugin-config
    Priority:            1
    Upstreams:
      Name:  cv-upstream
Status:
  Conditions:
    Message:              Sync Successfully
    Observed Generation:  3
    Reason:               ResourcesSynced
    Status:               True
    Type:                 ResourcesAvailable
Events:
  Type    Reason           Age                   From           Message
  ----    ------           ----                  ----           -------
  Normal  ResourcesSynced  4m32s (x15 over 48m)  ApisixIngress  ApisixIngress synced successfully

Also, when I manually copied configuration in ApisixPluginConfig and pasted it in ApisixRoute, it works fine.

AlinsRan commented 1 year ago

You need to create an ApisixPluginConfig before creating an ApisixRoute, otherwise this may be due to ApisixRoute not being able to find an ApisixPluginConfig, which may not work properly.

yujinchoi-94 commented 1 year ago

@AlinsRan

I've just tried like you suggested (Create ApisixPluginConfig first and then ApisixRoute) However, it isn't working. Here's what I've tried.

k describe apisixpluginconfig openapi-alimi-plugin-config
Name:         openapi-alimi-plugin-config
Namespace:    ingress-apisix
Labels:       argocd.argoproj.io/instance=dev-apisix
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixPluginConfig
Metadata:
  Creation Timestamp:  2023-03-23T05:42:39Z # created first.
  Generation:          1
  Managed Fields:
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      apisix-ingress-controller
    Operation:    Update
    Time:         2023-03-23T05:42:39Z
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:argocd.argoproj.io/instance:
      f:spec:
        .:
        f:plugins:
    Manager:         argocd-application-controller
    Operation:       Update
    Time:            2023-03-23T05:42:39Z
  Resource Version:  542795381
  UID:               36adc3ef-1b45-456d-9d08-3d1ce4ba36a5
Spec:
  Plugins:
    Config:
      request_headers:
        ***
      request_method:  GET
      upstream_headers:
        ***
      Uri:   ***
    Enable:  true
    Name:    forward-auth
    Config:
      Disable:      false
      prefer_name:  true
    Enable:         true
    Name:           prometheus
    Config:
      Uri:   http://openapi-logging.openapi-logging.svc.cluster.local:8080/v1/logging
    Enable:  true
    Name:    http-logger
    Config:
      Headers:
        ***
      Uri:                            ***
    Enable:                           true
    Name:                             proxy-rewrite
Status:
  Conditions:
    Message:              Sync Successfully
    Observed Generation:  1
    Reason:               ResourcesSynced
    Status:               True
    Type:                 ResourcesAvailable
Events:
  Type    Reason           Age                 From           Message
  ----    ------           ----                ----           -------
  Normal  ResourcesSynced  2m4s (x4 over 11m)  ApisixIngress  ApisixIngress synced successfully
k describe apisixroute openapi-alimi-route
Name:         openapi-alimi-route
Namespace:    ingress-apisix
Labels:       argocd.argoproj.io/instance=dev-apisix
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixRoute
Metadata:
  Creation Timestamp:  2023-03-23T05:53:13Z
  Generation:          1
  Managed Fields:
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:      apisix-ingress-controller
    Operation:    Update
    Time:         2023-03-23T05:53:13Z
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
        f:labels:
          .:
          f:argocd.argoproj.io/instance:
      f:spec:
        .:
        f:http:
    Manager:         argocd-application-controller
    Operation:       Update
    Time:            2023-03-23T05:53:13Z
  Resource Version:  542802079
  UID:               b8e9a2e9-29d1-42c6-83c9-a866b52b010e
Spec:
  Http:
    Match:
      Hosts:
        ***
      Paths:
        ***
    Name:                rule-alimi
    plugin_config_name:  openapi-alimi-plugin-config
    Priority:            1
    Upstreams:
      Name:  cv-upstream
Status:
  Conditions:
    Message:              Sync Successfully
    Observed Generation:  1
    Reason:               ResourcesSynced
    Status:               True
    Type:                 ResourcesAvailable
Events:
  Type    Reason           Age                From           Message
  ----    ------           ----               ----           -------
  Normal  ResourcesSynced  29s (x2 over 29s)  ApisixIngress  ApisixIngress synced successfully

Even though ApisixRoute created first, I think it should work normally after ApisixPluginConfig is created. Also, it would be good if i could know that kind of situation through events or at least logs. I can't find any logs related with that.

AlinsRan commented 1 year ago

@yujinchoi-94 I saw that you referenced upstream in ApisixRoute, and I think it's related to this issue.

renatomattos2912 commented 1 year ago

@renatomattos2912 I saw that you referenced upstream in ApisixRoute, and I think it's related to this issue.

Great, it is exactly that, but i think the fix was not released yet cause as we can see in ingress controller helm and in docker repo, it is still in version 1.6.0 that is the buggy version:

https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/values.yaml

https://hub.docker.com/r/apache/apisix-ingress-controller/tags

I hope it will be fixed in version 1.6.1 soon.

github-actions[bot] commented 1 year ago

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.