search

apache / apisix

The Cloud-Native API Gateway
https://apisix.apache.org/blog/
Apache License 2.0
14.43k stars 2.5k forks source link

bug: About the instability of Apisix's proxying of gRPC services. #9240

Open FlyTOmeLight opened 1 year ago

FlyTOmeLight commented 1 year ago

Current Behavior

I am using apisix to proxy my grpc service, and I found the following issue while doing benchmark. This is the access log and the error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type returned by the grpc client. This error is very frequent and seriously affects the use of our production environment. image The host is: ep-mzinkvxm.endpoint.windmill.com here is my route config:

- name: grpc
  priority: 1
  match:
    hosts:
      - {{ .Release.Name }}.endpoint.windmill.com
    paths:
      - "/*"
  backends:
    - serviceName: {{ .Release.Name }}
      servicePort: grpc

here is the config i copy from apisix-admin:

{
  "uris": ["/*"],
  "name": "default_ep-mzinkvxm_grpc",
  "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
  "priority": 1,
  "hosts": ["[ep-mzinkvxm.endpoint.windmill.com](http://ep-mzinkvxm.endpoint.windmill.com/)"],
  "upstream_id": "c54210dc",
  "labels": {
    "managed-by": "apisix-ingress-controller"
  },
  "status": 1
}

and there is some error I found from apisix-ingress-controller, i don't know if this error is related to this phenomenon.

2023-03-29T20:35:08+08:00   warn    ingress/apisix_route.go:440 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"default/ep-fuphptti","OldObject":null,"GroupVersion":"[apisix.apache.org/v2](http://apisix.apache.org/v2)"},"OldObject":null,"Tombstone":null}, "error": "6 errors occurred:\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 503; error message: {\"error_msg\":\"has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [cf3c7922]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [b83b49b4]: has no healthy etcd endpoint available\"}\n\n\t* unexpected status code 400; error message: {\"error_msg\":\"failed to fetch upstream info by upstream id [2132180e]: has no healthy etcd endpoint available\"}\n\n\n"}

Expected Behavior

I hope this mistake won't happen again.

Error Logs

No response

Steps to Reproduce

  1. Running APIsix on k8s using Helm Chart.
  2. Configure APIsix routes using YAML 
    
{{- if .Values.ingress.enabled -}}
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
name: {{ .Release.Name }}
spec:
http:
- name: http
  priority: 3
  match:
    paths:
      - /{{ .Release.Name }}/http/*
  backends:
    - serviceName: {{ .Release.Name }}
      servicePort: http
  plugins:
    - name: proxy-rewrite
      enable: true
      config:
        regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"]
    - name: api-breaker
      enable: true
      config:
        break_response_code: 502
        unhealthy:
          http_statuses: [500, 503]
          failure: 3
        healthy:
          http_statuses: [200]
          successes: 1
- name: grpc
  priority: 1
  match:
    hosts:
      - {{ .Release.Name }}.endpoint.windmill.com
    paths:
      - "/*"
  backends:
    - serviceName: {{ .Release.Name }}
      servicePort: grpc
- name: metrics
  priority: 2
  match:
    paths:
      - /{{ .Release.Name }}/metrics/*
  backends:
    - serviceName: {{ .Release.Name }}
      servicePort: metrics
  plugins:
    - name: proxy-rewrite
      enable: true
      config:
        regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"]

apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: name: {{ .Release.Name }} spec: loadbalancer: type: ewma retries: {{ .Values.ingress.retries }} timeout: connect: {{ .Values.ingress.timeout.connect }} send: {{ .Values.ingress.timeout.send }} read: {{ .Values.ingress.timeout.read }} portLevelSettings:

image

Environment

kingluo commented 1 year ago

rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.

And do you set a correct grpc port of the upstream?

FlyTOmeLight commented 1 year ago

rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.

And do you set a correct grpc port of the upstream?

yes, we will, and This error occurs intermittently and is not consistently reproducible.

FlyTOmeLight commented 1 year ago

rewriting the url of grpc request in proxy-rewrite pluing is not supported yet.

And do you set a correct grpc port of the upstream?

@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?

kingluo commented 1 year ago

rewriting the url of grpc request in proxy-rewrite pluing is not supported yet. And do you set a correct grpc port of the upstream?

@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?

Yes, we will handle it soon later.

But IMO, changing the URI is not a good practice for grpc requests and seems unnecessary. The format of grpc URI is /package.service/method, almost all official libraries from main programming langs, e.g. golang, will generate such URI automatically and disallow changing. So it's less possible to generate a non-standard URI like prefixing. So I am wondering why we need to change it.

FlyTOmeLight commented 1 year ago

rewriting the url of grpc request in proxy-rewrite pluing is not supported yet. And do you set a correct grpc port of the upstream?

@kingluo When will it be supported? Do you have any plan to support it? Also, do you have any recommended ways to proxy gRPC services?

Yes, we will handle it soon later.

But IMO, changing the URI is not a good practice for grpc requests and seems unnecessary. The format of grpc URI is /package.service/method, almost all official libraries from main programming langs, e.g. golang, will generate such URI automatically and disallow changing. So it's less possible to generate a non-standard URI like prefixing. So I am wondering why we need to change it.

I feel like you may have misunderstood my meaning. What we need is to use Apisix to proxy the gRPC service endpoint, for example, our original endpoint is 127.0.0.1:8100 and the http/2 port of Apisix is 127.0.0.1:8002. We want to use 127.0.0.1:8002/xxx/grpc to proxy out the endpoint 127.0.0.1:8100, and it is not necessary to rewrite the gRPC method. The above bug is a problem encountered during the proxy, Apisix often reports the upstream error, it is unstable, sometimes successful and sometimes failed. I guess it may be a bug, can you understand my meaning? @kingluo

image
kingluo commented 1 year ago

@FlyTOmeLight

  1. why do you configure proxy-rewrite to change uri? It's confusing and completely unnecessary for grpc.

  2. The IP and port are transport-level stuff, which is none of the business of the layer7 protocol, i.e. http2 and grpc based on http2. The downstream IP and port are completely decoupled from the upstream one. So, nothing special to do here, just ensure your upstream is correct.

  3. why you configure your upstream as three ports and different protocols?

apiVersion: apisix.apache.org/v2 kind: ApisixUpstream metadata: name: {{ .Release.Name }} spec: loadbalancer: type: ewma retries: {{ .Values.ingress.retries }} timeout: connect: {{ .Values.ingress.timeout.connect }} send: {{ .Values.ingress.timeout.send }} read: {{ .Values.ingress.timeout.read }} portLevelSettings:

  • port: 8000 scheme: http
  • port: 8001 scheme: grpc
  • port: 8002 scheme: http

Anyways, could you show the final configuration via admin API, instead of just template text? So that I could have a full view of your configuration and figure out what's wrong with your configuration.

Here is commands for reference:

# forward admin API port for access
# note that you need to specify the correct namespace you installed apisix
kubectl port-forward --namespace ingress-apisix svc/apisix-admin 9180
# show all routes
curl http://127.0.0.1:9180/apisix/admin/routes -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
# show all upstreams
curl http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
FlyTOmeLight commented 1 year ago

all routes:

{"node":{"dir":true,"nodes":[{"createdIndex":32,"value":{"uris":["\/*"],"priority":1,"hosts":["dongnan.endpoint.windmill.com"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"b9260e1d","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_grpc","id":"2c4a379a","labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238338,"key":"\/apisix\/routes\/2c4a379a"},{"createdIndex":228312,"value":{"uris":["\/*"],"priority":1,"hosts":["ep-mzinkvxm.endpoint.windmill.com"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"c54210dc","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_grpc","id":"502e295b","labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238347,"key":"\/apisix\/routes\/502e295b"},{"createdIndex":228313,"value":{"uris":["\/ep-mzinkvxm\/metrics\/*"],"priority":2,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"5c4b4166","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_metrics","id":"6064992d","plugins":{"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/ep-mzinkvxm\/metrics\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238348,"key":"\/apisix\/routes\/6064992d"},{"createdIndex":33,"value":{"uris":["\/dongnan\/metrics\/*"],"priority":2,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"202f5fa7","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_metrics","id":"6e97ffb8","plugins":{"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/dongnan\/metrics\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238340,"key":"\/apisix\/routes\/6e97ffb8"},{"createdIndex":31,"value":{"uris":["\/dongnan\/http\/*"],"priority":3,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"ce213e8b","update_time":1681020908,"create_time":1678936960,"name":"default_dongnan_http","id":"9076dfa4","plugins":{"api-breaker":{"break_response_code":502,"max_breaker_sec":300,"healthy":{"http_statuses":[200],"successes":1},"unhealthy":{"failure":3,"http_statuses":[500,503],"failures":3}},"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/dongnan\/http\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238337,"key":"\/apisix\/routes\/9076dfa4"},{"createdIndex":228311,"value":{"uris":["\/ep-mzinkvxm\/http\/*"],"priority":3,"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"b245204a","update_time":1681020909,"create_time":1680855294,"name":"default_ep-mzinkvxm_http","id":"ec12c165","plugins":{"api-breaker":{"break_response_code":502,"max_breaker_sec":300,"healthy":{"http_statuses":[200],"successes":1},"unhealthy":{"failure":3,"http_statuses":[500,503],"failures":3}},"proxy-rewrite":{"use_real_request_uri_unsafe":false,"regex_uri":["^\/ep-mzinkvxm\/http\/(.*)","\/$1"]}},"labels":{"managed-by":"apisix-ingress-controller"},"status":1},"modifiedIndex":238346,"key":"\/apisix\/routes\/ec12c165"}],"key":"\/apisix\/routes"},"count":6,"action":"get"}

all upstreams:

{"node":{"dir":true,"nodes":[{"createdIndex":68328,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-gdivkpvq_8001","id":"1b82663e","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.156","priority":0,"port":8001}],"scheme":"grpc","create_time":1679387520},"modifiedIndex":161948,"key":"\/apisix\/upstreams\/1b82663e"},{"createdIndex":30,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8002","id":"202f5fa7","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8002}],"scheme":"http","create_time":1678936960},"modifiedIndex":238354,"key":"\/apisix\/upstreams\/202f5fa7"},{"createdIndex":192271,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836108,"pass_host":"pass","name":"inference_ep-uxmeyitu_8000","id":"2040bfe5","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8000}],"scheme":"http","create_time":1680249445},"modifiedIndex":227721,"key":"\/apisix\/upstreams\/2040bfe5"},{"createdIndex":21,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154318,"pass_host":"pass","name":"default_ep-fuphptti_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"2132180e","scheme":"http","create_time":1678936959},"modifiedIndex":189383,"key":"\/apisix\/upstreams\/2132180e"},{"createdIndex":48,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903872,"pass_host":"pass","name":"inference_ep-hamvgnxm_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"26af8f93","scheme":"http","create_time":1678936961},"modifiedIndex":161961,"key":"\/apisix\/upstreams\/26af8f93"},{"createdIndex":38,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-disgsbce_8001","id":"38f5585c","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.141","priority":0,"port":8001}],"scheme":"grpc","create_time":1678936960},"modifiedIndex":161942,"key":"\/apisix\/upstreams\/38f5585c"},{"createdIndex":15,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154328,"pass_host":"pass","name":"default_ep-wyvpempc_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"41120bc4","scheme":"http","create_time":1678936959},"modifiedIndex":189391,"key":"\/apisix\/upstreams\/41120bc4"},{"createdIndex":37,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-disgsbce_8000","id":"4ff268ca","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.141","priority":0,"port":8000}],"scheme":"http","create_time":1678936960},"modifiedIndex":161941,"key":"\/apisix\/upstreams\/4ff268ca"},{"createdIndex":192272,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836108,"pass_host":"pass","name":"inference_ep-uxmeyitu_8001","id":"57478f73","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8001}],"scheme":"http","create_time":1680249445},"modifiedIndex":227722,"key":"\/apisix\/upstreams\/57478f73"},{"createdIndex":170503,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8002","id":"5c4b4166","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8002}],"scheme":"http","create_time":1679998256},"modifiedIndex":238363,"key":"\/apisix\/upstreams\/5c4b4166"},{"createdIndex":68327,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-gdivkpvq_8000","id":"6c8556a8","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.156","priority":0,"port":8000}],"scheme":"http","create_time":1679387520},"modifiedIndex":161947,"key":"\/apisix\/upstreams\/6c8556a8"},{"createdIndex":68330,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903830,"pass_host":"pass","name":"inference_ep-gdivkpvq_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"828b3784","scheme":"http","create_time":1679387520},"modifiedIndex":161957,"key":"\/apisix\/upstreams\/828b3784"},{"createdIndex":39,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903824,"pass_host":"pass","name":"inference_ep-disgsbce_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"a1fc09e6","scheme":"http","create_time":1678936960},"modifiedIndex":161953,"key":"\/apisix\/upstreams\/a1fc09e6"},{"createdIndex":13,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-wyvpempc_8000","id":"af1c6ae8","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8000}],"scheme":"http","create_time":1678936959},"modifiedIndex":189361,"key":"\/apisix\/upstreams\/af1c6ae8"},{"createdIndex":170500,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8000","id":"b245204a","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8000}],"scheme":"http","create_time":1679998256},"modifiedIndex":238361,"key":"\/apisix\/upstreams\/b245204a"},{"createdIndex":20,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-fuphptti_8001","id":"b83b49b4","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.158","priority":0,"port":8001}],"scheme":"http","create_time":1678936959},"modifiedIndex":189357,"key":"\/apisix\/upstreams\/b83b49b4"},{"createdIndex":29,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8001","id":"b9260e1d","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8001}],"scheme":"http","create_time":1678936960},"modifiedIndex":238352,"key":"\/apisix\/upstreams\/b9260e1d"},{"createdIndex":47,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-hamvgnxm_8001","id":"bfa6de29","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.120","priority":0,"port":8001}],"scheme":"http","create_time":1678936961},"modifiedIndex":161920,"key":"\/apisix\/upstreams\/bfa6de29"},{"createdIndex":170502,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_ep-mzinkvxm_8001","id":"c54210dc","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.171","priority":0,"port":8001}],"scheme":"grpc","create_time":1679998256},"modifiedIndex":238362,"key":"\/apisix\/upstreams\/c54210dc"},{"createdIndex":46,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1679903708,"pass_host":"pass","name":"inference_ep-hamvgnxm_8000","id":"c8a1eebf","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.120","priority":0,"port":8000}],"scheme":"http","create_time":1678936961},"modifiedIndex":161918,"key":"\/apisix\/upstreams\/c8a1eebf"},{"createdIndex":28,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1681021208,"pass_host":"pass","name":"default_dongnan_8000","id":"ce213e8b","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.129","priority":0,"port":8000}],"scheme":"http","create_time":1678936960},"modifiedIndex":238350,"key":"\/apisix\/upstreams\/ce213e8b"},{"createdIndex":192274,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680836403,"pass_host":"pass","name":"inference_ep-uxmeyitu_8002","nodes":[],"labels":{"managed-by":"apisix-ingress-controller"},"id":"ce4edec9","scheme":"http","create_time":1680249445},"modifiedIndex":227731,"key":"\/apisix\/upstreams\/ce4edec9"},{"createdIndex":19,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-fuphptti_8000","id":"cf3c7922","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.158","priority":0,"port":8000}],"scheme":"http","create_time":1678936959},"modifiedIndex":189356,"key":"\/apisix\/upstreams\/cf3c7922"},{"createdIndex":14,"value":{"type":"roundrobin","hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","update_time":1680154208,"pass_host":"pass","name":"default_ep-wyvpempc_8001","id":"d81b5a7e","labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"weight":100,"host":"10.233.64.148","priority":0,"port":8001}],"scheme":"http","create_time":1678936959},"modifiedIndex":189363,"key":"\/apisix\/upstreams\/d81b5a7e"}],"key":"\/apisix\/upstreams"},"count":24,"action":"get"}

@kingluo

FlyTOmeLight commented 1 year ago
  • why do you configure proxy-rewrite to change uri? It's confusing and completely unnecessary for grpc.

we has lots of same modelserver endpoint need to proxy, they have same port, http:8000,grpc:8001, metrics:8002, so we set it in upstreams like this.

kind: ApisixUpstream
metadata:
name: {{ .Release.Name }}
spec:
loadbalancer:
type: ewma
retries: {{ .Values.ingress.retries }}
timeout:
connect: {{ .Values.ingress.timeout.connect }}
send: {{ .Values.ingress.timeout.send }}
read: {{ .Values.ingress.timeout.read }}
portLevelSettings:
- port: 8000
scheme: http
- port: 8001
scheme: grpc
- port: 8002
scheme: http
image
kingluo commented 1 year ago

For grpc upstream server, the scheme must be grpc. If your service exposes different ports, choose the correct port for grpc only.

Some of your routes use http port but not grpc port.

For example:

this route should use grpc upstream according to what the name hints.

{
  "createdIndex":32,
  "value":{
    "uris":[
      "\/*"
    ],
    "priority":1,
    "hosts":[
      "dongnan.endpoint.windmill.com"
    ],
    "desc":"Created by apisix-ingress-controller, DO NOT modify it manually",
    "upstream_id":"b9260e1d",
    "update_time":1681020908,
    "create_time":1678936960,
    "name":"default_dongnan_grpc",
    "id":"2c4a379a",
    "labels":{
      "managed-by":"apisix-ingress-controller"
    },
    "status":1
  },
  "modifiedIndex":238338,
  "key":"\/apisix\/routes\/2c4a379a"
},

But actually, it uses http upstream:

{
  "createdIndex":29,
  "value":{
    "type":"roundrobin",
    "hash_on":"vars",
    "desc":"Created by apisix-ingress-controller, DO NOT modify it manually",
    "update_time":1681021208,
    "pass_host":"pass",
    "name":"default_dongnan_8001",
    "id":"b9260e1d",
    "labels":{
      "managed-by":"apisix-ingress-controller"
    },
    "nodes":[
      {
        "weight":100,
        "host":"10.233.64.129",
        "priority":0,
        "port":8001
      }
    ],
    "scheme":"http",
    "create_time":1678936960
  },
  "modifiedIndex":238352,
  "key":"\/apisix\/upstreams\/b9260e1d"
}

Please check your helm cfg according to ingress doc:

https://apisix.apache.org/zh/docs/ingress-controller/concepts/apisix_upstream/

FlyTOmeLight commented 1 year ago 
ingress

but i actually use grpc scheme @kingluo 

{{- if .Values.ingress.enabled -}}
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: {{ .Release.Name }}
spec:
  http:
    - name: http
      priority: 3
      match:
        paths:
          - /{{ .Release.Name }}/http/*
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: http
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"]
        - name: api-breaker
          enable: true
          config:
            break_response_code: 502
            unhealthy:
              http_statuses: [500, 503]
              failure: 3
            healthy:
              http_statuses: [200]
              successes: 1
    - name: grpc
      priority: 1
      match:
        hosts:
          - {{ .Release.Name }}.endpoint.windmill.com
        paths:
          - "/*"
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: grpc
    - name: metrics
      priority: 2
      match:
        paths:
          - /{{ .Release.Name }}/metrics/*
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: metrics
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"]

---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: {{ .Release.Name }}
spec:
  loadbalancer:
    type: ewma
  retries: {{ .Values.ingress.retries }}
  timeout:
    connect: {{ .Values.ingress.timeout.connect }}
    send: {{ .Values.ingress.timeout.send }}
    read: {{ .Values.ingress.timeout.read }}
  portLevelSettings:
    - port: 8000
      scheme: http
    - port: 8001
      scheme: grpc
    - port: 8002
      scheme: http
{{- end }}
kingluo commented 1 year ago

You should split the ApisixUpstream into multiple upstream specs, one for grpc.

FlyTOmeLight commented 1 year ago

You should split the ApisixUpstream into multiple upstream specs, one for grpc.

i follow by this example and set the apisixupstream spec, you mean i should split it into two specs? it's there any examples? @kingluo

image
kingluo commented 1 year ago

@FlyTOmeLight Could you confirm the route above used the upstream with the wrong scheme?

@tao12345666333, please help, thanks.

tao12345666333 commented 1 year ago

What's your APISIX Ingress version?

You can just use admin API to check the results

FlyTOmeLight commented 1 year ago

What's your APISIX Ingress version?

You can just use admin API to check the results

@tao12345666333

image
FlyTOmeLight commented 1 year ago 
ingress

but i actually use grpc scheme @kingluo

{{- if .Values.ingress.enabled -}}
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: {{ .Release.Name }}
spec:
  http:
    - name: http
      priority: 3
      match:
        paths:
          - /{{ .Release.Name }}/http/*
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: http
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: ["^/{{ .Release.Name }}/http/(.*)", "/$1"]
        - name: api-breaker
          enable: true
          config:
            break_response_code: 502
            unhealthy:
              http_statuses: [500, 503]
              failure: 3
            healthy:
              http_statuses: [200]
              successes: 1
    - name: grpc
      priority: 1
      match:
        hosts:
          - {{ .Release.Name }}.endpoint.windmill.com
        paths:
          - "/*"
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: grpc
    - name: metrics
      priority: 2
      match:
        paths:
          - /{{ .Release.Name }}/metrics/*
      backends:
        - serviceName: {{ .Release.Name }}
          servicePort: metrics
      plugins:
        - name: proxy-rewrite
          enable: true
          config:
            regex_uri: ["^/{{ .Release.Name }}/metrics/(.*)", "/$1"]

---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: {{ .Release.Name }}
spec:
  loadbalancer:
    type: ewma
  retries: {{ .Values.ingress.retries }}
  timeout:
    connect: {{ .Values.ingress.timeout.connect }}
    send: {{ .Values.ingress.timeout.send }}
    read: {{ .Values.ingress.timeout.read }}
  portLevelSettings:
    - port: 8000
      scheme: http
    - port: 8001
      scheme: grpc
    - port: 8002
      scheme: http
{{- end }}

yes, this is my whole setting @kingluo @tao12345666333 here is the apisixroute and apisixupstream description.

Name:         dongnan
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixRoute
Metadata:
  Creation Timestamp:  2023-02-27T02:50:52Z
  Generation:          1
  Managed Fields:
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2023-02-27T02:50:52Z
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:http:
      f:status:
        .:
        f:conditions:
    Manager:         apisix-ingress-controller
    Operation:       Update
    Time:            2023-02-27T02:50:52Z
  Resource Version:  54330836
  UID:               e58a7211-4153-45a9-b741-bca635bc2aa3
Spec:
  Http:
    Backends:
      Service Name:  dongnan
      Service Port:  http
    Match:
      Paths:
        /dongnan/http/*
    Name:  http
    Plugins:
      Config:
        regex_uri:
          ^/dongnan/http/(.*)
          /$1
      Enable:  true
      Name:    proxy-rewrite
      Config:
        break_response_code:  502
        Healthy:
          http_statuses:
            200
          Successes:  1
        Unhealthy:
          Failure:  3
          http_statuses:
            500
            503
      Enable:  true
      Name:    api-breaker
    Priority:  3
    Backends:
      Service Name:  dongnan
      Service Port:  grpc
    Match:
      Hosts:
        dongnan.endpoint.windmill.com
      Paths:
        /*
    Name:      grpc
    Priority:  1
    Backends:
      Service Name:  dongnan
      Service Port:  metrics
    Match:
      Paths:
        /dongnan/metrics/*
    Name:  metrics
    Plugins:
      Config:
        regex_uri:
          ^/dongnan/metrics/(.*)
          /$1
      Enable:  true
      Name:    proxy-rewrite
    Priority:  2
Status:
  Conditions:
    Message:              Sync Successfully
    Observed Generation:  1
    Reason:               ResourcesSynced
    Status:               True
    Type:                 ResourcesAvailable
Events:
  Type    Reason           Age                     From           Message
  ----    ------           ----                    ----           -------
  Normal  ResourcesSynced  2m58s (x3821 over 13d)  ApisixIngress  ApisixIngress synced successfully
Name:         dongnan
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixUpstream
Metadata:
  Creation Timestamp:  2023-02-27T02:50:52Z
  Generation:          1
  Managed Fields:
    API Version:  apisix.apache.org/v2
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:loadbalancer:
          .:
          f:type:
        f:portLevelSettings:
        f:retries:
        f:timeout:
          .:
          f:connect:
          f:read:
          f:send:
    Manager:         Go-http-client
    Operation:       Update
    Time:            2023-02-27T02:50:52Z
  Resource Version:  45712201
  UID:               c8dc04aa-7978-496e-827e-2cd22e475dbc
Spec:
  Loadbalancer:
    Type:  ewma
  Port Level Settings:
    Port:    8000
    Scheme:  http
    Port:    8001
    Scheme:  grpc
    Port:    8002
    Scheme:  http
  Retries:   3
  Timeout:
    Connect:  1s
    Read:     5s
    Send:     5s
Events:
  Type    Reason           Age                     From           Message
  ----    ------           ----                    ----           -------
  Normal  ResourcesSynced  3m30s (x3800 over 13d)  ApisixIngress  ApisixIngress synced successfully
FlyTOmeLight commented 1 year ago

@tao12345666333 @kingluo Hi, Is there any update?

kingluo commented 1 year ago

@FlyTOmeLight Could you confirm this route matched the wrong upstream scheme?

https://github.com/apache/apisix/issues/9240#issuecomment-1501064828

That is, you need to check the final configuration representation via admin API, if it is wrong and unexpected, then obviously the ApisixRoute and ApisixUpstream specs are wrong and you need to fix them.

FlyTOmeLight commented 1 year ago

@FlyTOmeLight Could you confirm this route matched the wrong upstream scheme?

I actually set grpc scheme, i give all the config and description in this issue, could you please tell me what's wrong with my setting? @kingluo

image image
kingluo commented 1 year ago

I mean, the name of this route contains "grpc", but it points to "http" upstream, is it expected or not?

1681209860508

1681209888968

If it's wrong, then you need to adjust your CRD, clear?

You could try to use one ApisixUpstream spec for each port instead, although they all point to the same k8s service.

FlyTOmeLight commented 1 year ago

I mean, the name of this route contains "grpc", but it points to "http" upstream, is it expected or not?

1681209860508

1681209888968

If it's wrong, then you need to adjust your CRD, clear?

You could try to use one ApisixUpstream spec for each port instead, although they all point to the same k8s service.

Of course, it's not what we expected. I repeatedly submitted our CRD in this issue and also asked you about the way to separate ApisixUpstream. When we submitted the CRD, the 8001 port was supposed to correspond to the gRPC scheme, but the Apisix Admin API returned HTTP instead. Isn't this an issue with Apisix? In addition, it is clearly stated in the Apisix documentation that arrays are supported, and there is no specific emphasis on the need to separate different schemes or services into different upstreams. image

kingluo commented 1 year ago

@FlyTOmeLight Try ApisixRoute without ApisixUpstream, specify the port directly.

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: foo-bar-route
spec:
  http:
  - name: foo
    match:
      hosts:
      - foo.com
      paths:
      - "/foo/*"
    backends:
    - serviceName: foo
      servicePort: 8001

@tao12345666333 any idea about ApisixUpstream for multi-ports service?

FlyTOmeLight commented 1 year ago

I conducted a lot of experiments and found that in apisixupstream, multiple schemes cannot be set. It always takes the scheme of the last element in the "portlevelsetting" array. I'm not sure if this is your design or a bug. According to the documentation, it seems like it should be possible to set it up this way.The setting shown in the following figure in the documentation is not effective. @kingluo @tao12345666333 image

shreemaan-abhishek commented 8 months ago

@FlyTOmeLight does your use case work correctly if you use the normal way to specifiy the scheme for the upstream?

sheharyaar commented 8 months ago

@FlyTOmeLight , any updates ? Can we close this ?