Http2Rpc配置两个Http2Rpc，路由转发失败

java4candy commented 4 months ago

Ⅰ. Issue Description

Ⅱ. Describe what happened

通过Higress，实现HTTP转同一个Dubbo下的两个Service时，配置了两个Ingress后，对应配置任意一个Http2Rpc，都可以正常访问。同时配置两个Http2Rpc，两个API接口都返回失败： NOT_FOUND: Could not resolve /account/name to a method NOT_FOUND: Could not resolve /echo/echo to a method

Ingress && Http2Rpc yaml

demo source dubbo-spring-boot-provider.zip

API

curl --location --request GET 'http://x.x.x.x:xxx/echo/echo?p=123'
curl --location --request GET 'http://x.x.x.x:xxx/account/name?p=123'


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
higress.io/destination: providers:com.alibaba.edas.boot.EchoService:1.0.0:DUBBO.zookeeper
higress.io/ignore-path-case: 'false'
higress.io/rpc-destination-name: echo
labels:
higress.io/resource-definer: higress
name: echo-higress
namespace: higress-system
spec:
ingressClassName: higress
rules:
- http:
    paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /echo
        pathType: Prefix
status:
loadBalancer: {}

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: higress.io/destination: providers:com.alibaba.edas.boot.AccountService:1.0.0:DUBBO.zookeeper higress.io/ignore-path-case: 'false' higress.io/rpc-destination-name: account labels: higress.io/resource-definer: higress name: account-higress namespace: higress-system spec: ingressClassName: higress rules:

http: paths:
- backend: resource: apiGroup: networking.higress.io kind: McpBridge name: default path: /account pathType: Prefix status: loadBalancer: {}

apiVersion: networking.higress.io/v1 kind: Http2Rpc metadata: name: echo namespace: higress-system spec: dubbo: group: DUBBO methods:

httpMethods:
- GET httpPath: /echo/echo params:
- paramKey: p paramSource: QUERY paramType: java.lang.String serviceMethod: echo service: com.alibaba.edas.boot.EchoService version: 1.0.0

apiVersion: networking.higress.io/v1 kind: Http2Rpc metadata: name: account namespace: higress-system spec: dubbo: group: DUBBO methods:

httpMethods:
- GET httpPath: /account/name params:
- paramKey: p paramSource: QUERY paramType: java.lang.String serviceMethod: name service: com.alibaba.edas.boot.AccountService version: 1.0.0

Environment:

Higress version: 1.3.3/1.3.6
OS : K8S v1.26.7

johnlanni commented 4 months ago

@java4candy 麻烦配置后，执行下hgctl gateway-config listener，发一下对应的配置

java4candy commented 4 months ago

@java4candy 麻烦配置后，执行下hgctl gateway-config listener，发一下对应的配置

配置如下：

{
  "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump",
  "dynamic_listeners": [
    {
      "active_state": {
        "last_updated": "2024-04-26T09:42:56.292Z",
        "listener": {
          "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
          "access_log": [
            {
              "filter": {
                "response_flag_filter": {
                  "flags": [
                    "NR"
                  ]
                }
              },
              "name": "envoy.access_loggers.file",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                "log_format": {
                  "text_format_source": {
                    "inline_string": "{\"authority\":\"%REQ(:AUTHORITY)%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"duration\":\"%DURATION%\",\"istio_policy_status\":\"%DYNAMIC_METADATA(istio.mixer:status)%\",\"method\":\"%REQ(:METHOD)%\",\"path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"request_id\":\"%REQ(X-REQUEST-ID)%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"route_name\":\"%ROUTE_NAME%\",\"start_time\":\"%START_TIME%\",\"trace_id\":\"%REQ(X-B3-TRACEID)%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"upstream_service_time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"user_agent\":\"%REQ(USER-AGENT)%\",\"x_forwarded_for\":\"%REQ(X-FORWARDED-FOR)%\"}\n"
                  }
                },
                "path": "/dev/stdout"
              }
            }
          ],
          "address": {
            "socket_address": {
              "address": "0.0.0.0",
              "port_value": 80
            }
          },
          "filter_chains": [
            {
              "filters": [
                {
                  "name": "envoy.filters.network.http_connection_manager",
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "access_log": [
                      {
                        "filter": {
                          "not_health_check_filter": {}
                        },
                        "name": "envoy.access_loggers.file",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
                          "log_format": {
                            "text_format_source": {
                              "inline_string": "{\"authority\":\"%REQ(:AUTHORITY)%\",\"bytes_received\":\"%BYTES_RECEIVED%\",\"bytes_sent\":\"%BYTES_SENT%\",\"downstream_local_address\":\"%DOWNSTREAM_LOCAL_ADDRESS%\",\"downstream_remote_address\":\"%DOWNSTREAM_REMOTE_ADDRESS%\",\"duration\":\"%DURATION%\",\"istio_policy_status\":\"%DYNAMIC_METADATA(istio.mixer:status)%\",\"method\":\"%REQ(:METHOD)%\",\"path\":\"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\",\"protocol\":\"%PROTOCOL%\",\"request_id\":\"%REQ(X-REQUEST-ID)%\",\"requested_server_name\":\"%REQUESTED_SERVER_NAME%\",\"response_code\":\"%RESPONSE_CODE%\",\"response_flags\":\"%RESPONSE_FLAGS%\",\"route_name\":\"%ROUTE_NAME%\",\"start_time\":\"%START_TIME%\",\"trace_id\":\"%REQ(X-B3-TRACEID)%\",\"upstream_cluster\":\"%UPSTREAM_CLUSTER%\",\"upstream_host\":\"%UPSTREAM_HOST%\",\"upstream_local_address\":\"%UPSTREAM_LOCAL_ADDRESS%\",\"upstream_service_time\":\"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%\",\"upstream_transport_failure_reason\":\"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"user_agent\":\"%REQ(USER-AGENT)%\",\"x_forwarded_for\":\"%REQ(X-FORWARDED-FOR)%\"}\n"
                            }
                          },
                          "path": "/dev/stdout"
                        }
                      }
                    ],
                    "common_http_protocol_options": {
                      "idle_timeout": "180s"
                    },
                    "forward_client_cert_details": "SANITIZE_SET",
                    "http2_protocol_options": {
                      "initial_connection_window_size": 1048576,
                      "initial_stream_window_size": 65535,
                      "max_concurrent_streams": 100
                    },
                    "http_filters": [
                      {
                        "name": "envoy.filters.http.cors",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                        }
                      },
                      {
                        "name": "envoy.filters.http.rbac",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC"
                        }
                      },
                      {
                        "name": "envoy.filters.http.local_ratelimit",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
                          "stat_prefix": "http_local_rate_limiter"
                        }
                      },
                      {
                        "name": "envoy.filters.http.fault",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                        }
                      },
                      {
                        "name": "envoy.filters.http.http_dubbo_transcoder",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.http_dubbo_transcoder.v3.HttpDubboTranscoder"
                        }
                      },
                      {
                        "name": "envoy.filters.http.http_dubbo_transcoder",
                        "typed_config": {
                          "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                          "type_url": "type.googleapis.com/envoy.extensions.filters.http.http_dubbo_transcoder.v3.HttpDubboTranscoder"
                        }
                      },
                      {
                        "name": "envoy.filters.http.router",
                        "typed_config": {
                          "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                        }
                      }
                    ],
                    "http_protocol_options": {
                      "accept_http_10": true
                    },
                    "max_request_headers_kb": 60,
                    "normalize_path": true,
                    "path_with_escaped_slashes_action": "KEEP_UNCHANGED",
                    "rds": {
                      "config_source": {
                        "ads": {},
                        "initial_fetch_timeout": "0s",
                        "resource_api_version": "V3"
                      },
                      "route_config_name": "http.80"
                    },
                    "server_name": "istio-envoy",
                    "set_current_client_cert_details": {
                      "cert": true,
                      "dns": true,
                      "subject": true,
                      "uri": true
                    },
                    "stat_prefix": "outbound_0.0.0.0_80",
                    "stream_idle_timeout": "180s",
                    "tracing": {
                      "client_sampling": {
                        "value": 100
                      },
                      "custom_tags": [
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_effective_policy_id"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.allow_policy.name"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_allow_shadow_engine_result"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.allow_policy.result"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_effective_policy_id"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.deny_policy.name"
                        },
                        {
                          "metadata": {
                            "kind": {
                              "request": {}
                            },
                            "metadata_key": {
                              "key": "envoy.filters.http.rbac",
                              "path": [
                                {
                                  "key": "istio_dry_run_deny_shadow_engine_result"
                                }
                              ]
                            }
                          },
                          "tag": "istio.authorization.dry_run.deny_policy.result"
                        },
                        {
                          "literal": {
                            "value": "latest"
                          },
                          "tag": "istio.canonical_revision"
                        },
                        {
                          "literal": {
                            "value": "unknown"
                          },
                          "tag": "istio.canonical_service"
                        },
                        {
                          "literal": {
                            "value": "unknown"
                          },
                          "tag": "istio.mesh_id"
                        },
                        {
                          "literal": {
                            "value": "higress-system"
                          },
                          "tag": "istio.namespace"
                        }
                      ],
                      "overall_sampling": {
                        "value": 100
                      },
                      "random_sampling": {
                        "value": 1
                      }
                    },
                    "upgrade_configs": [
                      {
                        "upgrade_type": "websocket"
                      }
                    ],
                    "use_remote_address": true
                  }
                }
              ]
            }
          ],
          "name": "0.0.0.0_80",
          "per_connection_buffer_limit_bytes": 32768,
          "traffic_direction": "OUTBOUND"
        },
        "version_info": "2024-04-26T09:13:52Z/21"
      },
      "name": "0.0.0.0_80"
    }
  ],
  "static_listeners": [
    {
      "last_updated": "2024-04-26T08:59:49.183Z",
      "listener": {
        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
        "address": {
          "socket_address": {
            "address": "0.0.0.0",
            "port_value": 15090
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.filters.network.http_connection_manager",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                  "http_filters": [
                    {
                      "name": "envoy.filters.http.router",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                      }
                    }
                  ],
                  "route_config": {
                    "virtual_hosts": [
                      {
                        "domains": [
                          "*"
                        ],
                        "name": "backend",
                        "routes": [
                          {
                            "match": {
                              "prefix": "/stats/prometheus"
                            },
                            "route": {
                              "cluster": "prometheus_stats"
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "stat_prefix": "stats"
                }
              }
            ]
          }
        ]
      }
    },
    {
      "last_updated": "2024-04-26T08:59:49.197Z",
      "listener": {
        "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
        "address": {
          "socket_address": {
            "address": "0.0.0.0",
            "port_value": 15021
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.filters.network.http_connection_manager",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                  "http_filters": [
                    {
                      "name": "envoy.filters.http.router",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                      }
                    }
                  ],
                  "route_config": {
                    "virtual_hosts": [
                      {
                        "domains": [
                          "*"
                        ],
                        "name": "backend",
                        "routes": [
                          {
                            "match": {
                              "prefix": "/healthz/ready"
                            },
                            "route": {
                              "cluster": "agent"
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "stat_prefix": "agent"
                }
              }
            ]
          }
        ]
      }
    }
  ],
  "version_info": "2024-04-26T09:42:56Z/30"
}

johnlanni commented 4 months ago

envoy.filters.http.http_dubbo_transcoder

这个filter在全局被插入了两次，多个http2rpc的cr，应该复用同一个filter cc @Hinsteny

hanxiantao commented 4 months ago

我这边来修复下，请帮忙分配下，感谢 @johnlanni

hanxiantao commented 4 months ago

https://github.com/alibaba/higress/blob/2031c659c2061293ef71f393ef2bf98a3881ae11/pkg/ingress/config/ingress_config.go#L512C1-L532C5 修复方案：在convertEnvoyFilter()调用constructHttp2RpcEnvoyFilter()时添加一个标志，只有第一次调用constructHttp2RpcEnvoyFilter()时才会添加envoy.filters.http.http_dubbo_transcoder的部分，后续转换的EnvoyFilter不再添加envoy.filters.http.http_dubbo_transcoder的部分我这边会在本地做下测试后，再提交PR

johnlanni commented 4 months ago

@hanxiantao 可以的

hanxiantao commented 4 months ago

@hanxiantao 可以的

@johnlanni pr已提交，有时间帮忙review下 https://github.com/alibaba/higress/pull/935

johnlanni commented 4 months ago

fixed

alibaba / higress