search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.12k stars 4.82k forks source link

Custom upstrem/backend response #12883

Open tony-liuliu opened 4 years ago

tony-liuliu commented 4 years ago

I want to rewrite the 4xx/5xx returned by the upstrem/backend service. I see that envoy 1.15.0 has been able to rewrite the error of envoy itself through local_reply_config, but it seems that it cannot rewrite the error returned by upstrem/backend. The function of local_reply_config is very good and can replace haproxy as the front end lb. It would be even better if could rewrite upstrem/backend.

I don’t know if my configuration is wrong, my configuration：

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: local-reply
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          local_reply_config:
            mappers:
            - filter:
                status_code_filter:
                  comparison:
                    op: EQ
                    value:
                      default_value: 503
                      runtime_key: key_b
              status_code: 200
              body:
                inline_string: "no health server"
              body_format_override:
                json_format:
                  status: "503"
                  message: "%LOCAL_REPLY_BODY%"
                  request_path: "%REQ(:path)%"
[root@k8s-20 ~]# curl -i http://test.domain.com/503/test/path                                                                                            
HTTP/1.1 503 Service Unavailable
server: nginx/1.15.6
date: Sat, 29 Aug 2020 03:51:36 GMT
content-type: text/html
content-length: 203
x-request-id: 3c4a04cd-c4d3-4819-af76-256b6c905a0b

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>
mattklein123 commented 4 years ago

We don't have generic rewrite functionality currently. This could be implemented by a custom filter or Lua potentially.

tony-liuliu commented 4 years ago

@mattklein123 Thank you very much, I will try to use custom filter. Is there a similar custom filter example.

dio commented 4 years ago

@tony-liuliu do you think you can make it available upstream? Thinking to have a way to modify the proxied response (with non-2xx status), do you think it makes sense to have an error_page "filter" similar to what NGINX has? (in NGINX, we need to turn on proxy_intercept_errors to let non-2xx proxied response to be handled by the proxy's error_page config).

I can see it can also be implemented in Lua (with some thinking on how to not complicate the "state management" there). Relevant: https://github.com/envoyproxy/envoy/issues/5998.

tony-liuliu commented 4 years ago

Yes, I think this feature is very useful because our back-end services are provided by different languages, such as nginx, tomcat, nodejs, go. If you do not manually compile and configure them, their server information will expose it when an error is encountered, which is prone to insecure factors. Using proxy_intercept_errors like nginx to unify redirect errors seems very suitable for this scenario. This way we don't have to modify error redirects such as http 404 for each backend language. I hope that envoy can consider such needs in future plans.

dio commented 4 years ago

@tony-liuliu if you can help us with a design doc, I think I can help you with the implementation.

cc. @mattklein123

mattklein123 commented 4 years ago

I think it would be great to have a custom response encoder filter that could do something like this.

Note also this would go really well with the work outlined here to allow for response matching: https://github.com/envoyproxy/envoy/issues/12968. cc @snowp

esnible commented 4 years ago

I tried to do something similar with an AssemblyScript WASM filter and failed.

My filter's onResponseHeaders() could see the 4xx/5xx returned by the backend. I tried calling send_local_response() from within onResponseHeaders() but by then it was too late.

I notice FilterHeadersStatus .ContinueAndDontEndStream was recently added which might solve my problem but that return isn't yet available in the Envoy version I am using nor in the Solo.io AssemblyScript runtime.

dio commented 4 years ago

Another ref: https://www.haproxy.com/blog/serve-dynamic-custom-error-pages-with-haproxy/

ceastman-ibm commented 4 years ago

@dio @mattklein123 Is this sufficient for a design doc? https://docs.google.com/document/d/1Bgi0agddua9LyBH3NnGBuDPcYJxoZwZLnL1rBcKFgZI

mattklein123 commented 4 years ago

This behavior should be implemented as a filter. It's not clear to me that is what is being proposed? Is it?

esmet commented 4 years ago

I ended up implementing this as an HTTP filter before I knew this issue (or any design documents/requirements) existed. For that reason, my approach may be different than what's being discussed here but I'm very much open to feedback. Please see https://github.com/envoyproxy/envoy/pull/14221 ( cc @mattklein123 / @tony-liuliu )

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 3 years ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

esmet commented 3 years ago

Let's move this to no stalebot. I'm tracking an implementation in https://github.com/envoyproxy/envoy/pull/14221 but it fell to the backburner during the holidays + other tasks. I'll be looking at it again soon, though.

joshpollara commented 3 years ago

Are there implementation plans for this issue? Looks like #14221 is closed.

esmet commented 3 years ago

@joshpollara it's on the back-burner right now, unfortunately, until I get cycles to come back to it. #14221 went stale since we were exploring alternative implementations like #14926

joshpollara commented 3 years ago

@esmet ok. thanks for the update!

esnible commented 3 years ago

I tried to do this with Wasm and succeeded. (I failed the first time, then finally got things exactly right).

See https://github.com/esnible/wasm-examples/tree/main/httpcall for an example of using EnvoyFilter to customize a response.

ghevge commented 3 years ago

Any plans to implement this functionality any time soon ? I'm looking for something similar too.

ghevge commented 3 years ago

Will it be possible to implement something that will allow to control the envoy response from the upstream server ? I'm think at using some custom envoy specific headers maybe, generated by the upstream server, which will be consumed by envoy and based on which the output will be formatted. I particular find this option appealing in the context of using envoy to do http to grpc transcoding, as out of the box, grpc doesn't offer much choices for response statuses.

feature-id commented 2 years ago

+1 vote for this issue.

14926 (and all related tickets, mentioned above) looks like closed by inactivity. :(