search

apache / apisix

The Cloud-Native API Gateway
https://apisix.apache.org/blog/
Apache License 2.0
14.45k stars 2.52k forks source link

allow proxy-rewrite plugin to rewrite x-forwarded-port request header #4942

Closed eastearth closed 1 year ago

eastearth commented 3 years ago

Issue description

If the upstream service through the APISIX proxy returns 302 redirect, the client redirected the port to the 9080 port of apisix,

For example, client access http://a.com:80/，

Normally, it would be redirected to the http://a.com:80/login

But it’s actually redirected to the http://a.com:9080/login

Environment

Steps to reproduce

1.client access http://a.com:80/，

2.Normally, upstream response 302 to the http://a.com:80/login

3.But it’s actually redirected to the http://a.com:9080/login

Actual result

3.But it’s actually redirected to the http://a.com:9080/login and the browser is waitting

Error log

the browser return: 无法访问此网站eks-jenkins-test.invik.com 的响应时间过长。

Expected result

No response

tokers commented 3 years ago

What's the result of requesting to the backend directly?

eastearth commented 3 years ago

What's the result of requesting to the backend directly?

it is normal，can redirect to the login api and logged in.

chloe88 commented 3 years ago

the response is not processed by the gateway (apisix).

See the nginx conf

the following x-forwarded-* headers is to send to upstream server

        set $var_x_forwarded_for        $remote_addr;
        set $var_x_forwarded_proto      $scheme;
        set $var_x_forwarded_host       $host;
        set $var_x_forwarded_port       $server_port;

        if ($http_x_forwarded_for != "") {
            set $var_x_forwarded_for "${http_x_forwarded_for}, ${realip_remote_addr}";
        }
        if ($http_x_forwarded_proto != "") {
            set $var_x_forwarded_proto $http_x_forwarded_proto;
        }
        if ($http_x_forwarded_host != "") {
            set $var_x_forwarded_host $http_x_forwarded_host;
        }
        if ($http_x_forwarded_port != "") {
            set $var_x_forwarded_port $http_x_forwarded_port;
        }

        proxy_set_header   X-Forwarded-For      $var_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $var_x_forwarded_proto;
        proxy_set_header   X-Forwarded-Host     $var_x_forwarded_host;
        proxy_set_header   X-Forwarded-Port     $var_x_forwarded_port;

so you can add some header to the previous proxy. set the X-Forwarded-Port to 80 to resolve it

tokers commented 3 years ago

the response is not processed by the gateway (apisix).

See the nginx conf

the following x-forwarded-* headers is to send to upstream server

        set $var_x_forwarded_for        $remote_addr;
        set $var_x_forwarded_proto      $scheme;
        set $var_x_forwarded_host       $host;
        set $var_x_forwarded_port       $server_port;

        if ($http_x_forwarded_for != "") {
            set $var_x_forwarded_for "${http_x_forwarded_for}, ${realip_remote_addr}";
        }
        if ($http_x_forwarded_proto != "") {
            set $var_x_forwarded_proto $http_x_forwarded_proto;
        }
        if ($http_x_forwarded_host != "") {
            set $var_x_forwarded_host $http_x_forwarded_host;
        }
        if ($http_x_forwarded_port != "") {
            set $var_x_forwarded_port $http_x_forwarded_port;
        }

        proxy_set_header   X-Forwarded-For      $var_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $var_x_forwarded_proto;
        proxy_set_header   X-Forwarded-Host     $var_x_forwarded_host;
        proxy_set_header   X-Forwarded-Port     $var_x_forwarded_port;

so you can add some header to the previous proxy. set the X-Forwarded-Port to 80 to resolve it

Good job, let see whether it can help @eastearth to solve his problems. If so, I think we need to add a FAQ entry to record this.

eastearth commented 3 years ago

the response is not processed by the gateway (apisix).

See the nginx conf

the following x-forwarded-* headers is to send to upstream server

        set $var_x_forwarded_for        $remote_addr;
        set $var_x_forwarded_proto      $scheme;
        set $var_x_forwarded_host       $host;
        set $var_x_forwarded_port       $server_port;

        if ($http_x_forwarded_for != "") {
            set $var_x_forwarded_for "${http_x_forwarded_for}, ${realip_remote_addr}";
        }
        if ($http_x_forwarded_proto != "") {
            set $var_x_forwarded_proto $http_x_forwarded_proto;
        }
        if ($http_x_forwarded_host != "") {
            set $var_x_forwarded_host $http_x_forwarded_host;
        }
        if ($http_x_forwarded_port != "") {
            set $var_x_forwarded_port $http_x_forwarded_port;
        }

        proxy_set_header   X-Forwarded-For      $var_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $var_x_forwarded_proto;
        proxy_set_header   X-Forwarded-Host     $var_x_forwarded_host;
        proxy_set_header   X-Forwarded-Port     $var_x_forwarded_port;

so you can add some header to the previous proxy. set the X-Forwarded-Port to 80 to resolve it

my previous proxy is Layer 4 proxy,can not add headers,and I think this is a bug,cant fix it?

eastearth commented 3 years ago

Is it possible to specify X-Forwarded-Port by configuring configmap? The corresponding configuration is not found. Where should I configure it? For example, which parameter should be set for this configuration?

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'
nic-chen commented 3 years ago

@eastearth

You could take a test for it.

Does your APISIX use port 9080? What is the relationship between port 80 and 9080?

chloe88 commented 3 years ago

@eastearth Can your layer 4 proxy be configured in a transparent manner?

chloe88 commented 3 years ago

@eastearth tell me wechat

eastearth commented 3 years ago

@eastearth tell me wechat

jxd1075943283

tokers commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX?

See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

eastearth commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX?

See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

actually,I install apisix by helm in k8s, now i configure apisix deployment like this,it is working now, _" lifecycle: postStart: exec: command:

but i think this still need a configurable item, because according to the introduction of k8s spec.containers.lifecycle.poststart,

The operation performed by poststart does not necessarily complete before the main program entry enterpoint starts

this is k8s official document introduction: "Kubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes.

"

tokers commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX? See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

actually,I install apisix by helm in k8s, now i configure apisix deployment like this,it is working now, _" lifecycle: postStart: exec: command: - /bin/sh - -c - sed -i 's/proxy_set_header X-Forwarded-Port $var_x_forwarded_port/#proxy_set_header X-Forwarded-Port $var_x_forwardedport/g' /usr/local/apisix/conf/nginx.conf "

but i think this still need a configurable item, because according to the introduction of k8s spec.containers.lifecycle.poststart,

The operation performed by poststart does not necessarily complete before the main program entry enterpoint starts

this is k8s official document introduction: "Kubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes.

"

That's right, the postStart hook is not a reliable way to do this. Would you like to submit a PR to let these X-Forwared-* items be configurable? Thanks!

kika-jiaxudong commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX? See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

actually,I install apisix by helm in k8s, now i configure apisix deployment like this,it is working now, _" lifecycle: postStart: exec: command: - /bin/sh - -c - sed -i 's/proxy_set_header X-Forwarded-Port $var_x_forwarded_port/#proxy_set_header X-Forwarded-Port $var_x_forwardedport/g' /usr/local/apisix/conf/nginx.conf " but i think this still need a configurable item, because according to the introduction of k8s spec.containers.lifecycle.poststart, The operation performed by poststart does not necessarily complete before the main program entry enterpoint starts this is k8s official document introduction: "Kubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes. "

That's right, the postStart hook is not a reliable way to do this. Would you like to submit a PR to let these X-Forwared-* items be configurable? Thanks!

Yes,I would very much want to let these items to be configurable,

What should I do?

tokers commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX? See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

actually,I install apisix by helm in k8s, now i configure apisix deployment like this,it is working now, _" lifecycle: postStart: exec: command: - /bin/sh - -c - sed -i 's/proxy_set_header X-Forwarded-Port $var_x_forwarded_port/#proxy_set_header X-Forwarded-Port $var_x_forwardedport/g' /usr/local/apisix/conf/nginx.conf " but i think this still need a configurable item, because according to the introduction of k8s spec.containers.lifecycle.poststart, The operation performed by poststart does not necessarily complete before the main program entry enterpoint starts this is k8s official document introduction: "Kubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes. "

That's right, the postStart hook is not a reliable way to do this. Would you like to submit a PR to let these X-Forwared-* items be configurable? Thanks!

Yes,I would very much want to let these items to be configurable,

What should I do?

The ngx_tpl.lua is a template to customize the nginx.conf file, so just let these items to be templated. You can see more in apisix/cli/ directory.

kika-jiaxudong commented 3 years ago

是否能通过配置configmap来指定X-Forwarded-Port呢，没有找到对应的配置，该在哪里配置呢，例如这个配置，应该设置哪一个参数呢？

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "/dev/stderr"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "/dev/stdout"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Forwarded-For"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    X-Forwarded-Port
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 192.168.0.0/16
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 'unix:'

How did you install APISIX? See https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua, currently, the related items are not configurable.

actually,I install apisix by helm in k8s, now i configure apisix deployment like this,it is working now, _" lifecycle: postStart: exec: command: - /bin/sh - -c - sed -i 's/proxy_set_header X-Forwarded-Port $var_x_forwarded_port/#proxy_set_header X-Forwarded-Port $var_x_forwardedport/g' /usr/local/apisix/conf/nginx.conf " but i think this still need a configurable item, because according to the introduction of k8s spec.containers.lifecycle.poststart, The operation performed by poststart does not necessarily complete before the main program entry enterpoint starts this is k8s official document introduction: "Kubernetes sends the postStart event immediately after the Container is created. There is no guarantee, however, that the postStart handler is called before the Container's entrypoint is called. The postStart handler runs asynchronously relative to the Container's code, but Kubernetes' management of the container blocks until the postStart handler completes. The Container's status is not set to RUNNING until the postStart handler completes. "

That's right, the postStart hook is not a reliable way to do this. Would you like to submit a PR to let these X-Forwared-* items be configurable? Thanks!

Yes,I would very much want to let these items to be configurable, What should I do?

The ngx_tpl.lua is a template to customize the nginx.conf file, so just let these items to be templated. You can see more in apisix/cli/ directory.

Yes,I see,you mean that I modify ngx_tpl.lua to be configurable myself and change the X-Forwarded-Port in k8s configmap?

Actually, my lua is not good,I can't do it myself.

Can you help?

kika-jiaxudong commented 3 years ago

I have a request, can someone help me to change the x-forward-* in the https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua to configurable?

kika-jiaxudong commented 3 years ago

Especially the X-Forwarded-Port

kika-jiaxudong commented 3 years ago

@eastearth

You could take a test for it.

Does your APISIX use port 9080? What is the relationship between port 80 and 9080?

eg. I request this url http://eks-jenkins-test.invik.com:80

when jenkins response 302 to browser,

and the browser request http://eks-jenkins-test.invik.com:9080/login

The port of apisix is 9080,

and 80 is the port of aws LB layer 4

tokers commented 3 years ago

I have a request, can someone help me to change the x-forward-* in the https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua to configurable?

As an example, you can see how the keepalive_* was set to be configurable, see https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua#L328-L330 for more details.

nic-chen commented 3 years ago

@eastearth

you can use proxy-rewrite plugin to set the header. see:

https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/proxy-rewrite.md

z-yuxie commented 2 years ago

@eastearth @nic-chen @tokers Hello everyone, has this issue been resolved? I encountered the same problem with image

I checked the instructions about reverse proxy in jenkins’ own documentation. ↓Screenshot of the description of this content in the above document image According to the description, Jenkins will redirect itself through the 302 status code in some steps, and return the address to be redirected in the Location of the http response header. However, currently, the port of the Localtion returned by Jenkins every time is apisix. There are two ways to handle the port, one of which requires us to set the following parameters in the request header: X-Forwarded-Host: jenkins.xxx.xxx (the domain name of its own jenkins) X-Forwarded-Port: 80 X-Forwarded-Proto: https

Regarding the way to set the request header, this is the method I have tried, but it has no effect. Jenkins still returns the redirection address with port 9080:

  1. Set the request header through the "Request Header Rewrite" function in the "Routes" of the interface image ↓This is how the routing configuration formed in this way looks like in the "Data Editor" image
  2. Set the request header through the "proxy-rewrite" plug-in in the "service" of the interface image ↓This is how the service configuration formed in this way looks like in the "Data Editor" image

Then because I am not sure whether this is a problem with Jenkins itself or a problem with the apisix plug-in, I now want to try another method of modifying the Location in the response header. Is there any way that I can modify the Location in the response header? Do you perform regular matching and modify it? The desired effect is roughly this: response header: Location: https://jenkins.xxx.xxx:9080/ ↓ response header: Location: https://jenkins.xxx.xxx/

I know that there is a "response-rewrite" plug-in that can modify the Location mentioned above, but because the plug-in provides The instructions of How should I configure it? Thank you.

In addition, one method that I think is probably effective is to open port 9080, and then reconfigure a route to identify requests from port 9080, and redirect them to port 80, but To be honest, I don't really want to open a new port unless I have to, and this way of handling it will add one more http request, which doesn't always feel good.

nic-chen commented 2 years ago

hi @z-yuxie Are there other routes configured in your APISIX? Need to confirm whether the requests hit other routes.

z-yuxie commented 2 years ago

@nic-chen 您好，我有配置别的路由，但是jenkins能命中的路由仅有这一个，因为我的每个路由都是针对独立的域名配置的

My routing strategy is based on the domain name. Although there are multiple routes configured, Jenkins will only hit its own route and not the others.

tokers commented 2 years ago

@z-yuxie Try to use the serverless-functions plugins to rewrite headers in the header_filter phase, you can inject your own logics there.

nic-chen commented 2 years ago

@z-yuxie After testing it, I found that X-Forwarded-Port cannot be rewritten directly. It is recommended that jenkins and APISIX use the same port number, and there is no such problem.

Or use a temporary solution, comment out these lines https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua#L612-L613, and restart APISIX.

haozileung commented 2 years ago

I have encountered exactly the same problem here, using k8s + apisix ingress + jenkins. Changing the port is not a good approach and modified the source code could work but annoying.

zsjinwei commented 2 years ago

@z-yuxie Try to use the serverless-functions plugins to rewrite headers in the header_filter phase, you can inject your own logics there.

I have this issue too, could you show me how to rewrite using serverless-functions?

return200 commented 2 years ago

@eastearth @nic-chen @tokers 您们好，请问这个问题有被解决么？我遇到了与之相同的问题 image

我去查看了jenkins自己的文档中关于反向代理的说明， ↓上述文档中关于此内容描述的截图 image

根据其中的描述，是jenkins在某些步骤会自己通过302状态码进行重定向，并在http响应头的Location返回要重定向到的地址，但是目前jenkins方面每次返回的Localtion的端口都是apisix的端口，有两种处理方式，其中一种是需要我们在请求头中设置以下参数： X-Forwarded-Host: jenkins.xxx.xxx（自身jenkins的域名） X-Forwarded-Port: 80 X-Forwarded-Proto: https

关于设置请求头的这种方式，这是我尝试过的方法，但都没有效果，jenkins依然返回了带有9080端口的重定向地址： 1、通过界面的“路由”(routes)中的“请求头重写”功能设置请求头 image ↓这是通过该方式形成的路由配置在“数据编辑器”中展示的样子 image 2、通过界面的“服务”(service)中的“proxy-rewrite”插件设置请求头 image ↓这是通过该方式形成的服务配置在“数据编辑器”中展示的样子 image

然后因为我不太确定这是jenkins自身的问题还是apisix插件的问题，于是我现在想要试试另一种修改响应头中的Location的方法，请问有什么方式能让我对响应头中的Location进行正则匹配，并进行修改的么？想要的效果大概是这样的： response header: Location：https://jenkins.xxx.xxx:9080/ ↓ response header: Location：https://jenkins.xxx.xxx/

我知道有个“response-rewrite”插件可以对上面所说的Location进行修改，但因为该插件提供的说明中并没有对其进行较多的描述，也没有提供类似的改写范例，我并不确定他是否能做到我上面说的这种基于正则的改写，如果可以的话，能否给我说下我应该怎么去配置，谢谢。

除此之外，我目前觉得大概能有效的一种方法是，我将9080端口也开放出去，然后再重新配置一个路由去识别来自9080端口的请求，并将其重新重定向到80端口，但老实说不到万不得已，我并不是太想新开放一个端口，而且这样的处理方式会让多增加一次http的请求，总感觉不太美好。

你字多，就回复你吧。你域名解析到哪里了，是 nginx 还是 slb？如果是 Nginx 加一个 proxy_set_header X-Forwarded-Port 443; slb 应该有单独的选项可以设置。

return200 commented 2 years ago

Issue description

If the upstream service through the APISIX proxy returns 302 redirect, the client redirected the port to the 9080 port of apisix,

For example, client access http://a.com:80/，

Normally, it would be redirected to the http://a.com:80/login

But it’s actually redirected to the http://a.com:9080/login

Environment

  • apisix version: 2.7.0(helm chart installed,and chart version is apisix-0.3,6)
  • apisixroute:

apiVersion: apisix.apache.org/v2beta1 kind: ApisixRoute metadata: name: jenkins namespace: jenkins spec: http:

  • name: jenkins match: hosts:

    • eks-jenkins-test.invik.com paths:

    • /* backend: serviceName: jenkins2 servicePort: 8080 plugins:

    • name: ip-restriction enable: true config: whitelist:

    • 39.106.1.281/32

    • 192.168.0.0/16

Service:

apiVersion: v1 kind: Service metadata: name: jenkins2 namespace: jenkins spec: ports:

  • name: web port: 8080 protocol: TCP targetPort: web
  • name: agent port: 50000 protocol: TCP targetPort: agent selector: app: jenkins2 sessionAffinity: None type: ClusterIP

Steps to reproduce

1.client access http://a.com:80/，

2.Normally, upstream response 302 to the http://a.com:80/login

3.But it’s actually redirected to the http://a.com:9080/login

Actual result

3.But it’s actually redirected to the http://a.com:9080/login and the browser is waitting

Error log

the browser return: 无法访问此网站eks-jenkins-test.invik.com 的响应时间过长。

Expected result

No response

where does your domain a.com resolve to? Add proxy_set_header X-Forwarded-Port $server_port; if nginx; if it's a slb/alb, there should have a configuration to add the header X-Forwarded-Port.

syokensyo commented 2 years ago

@eastearth @nic-chen @tokers 您们好，请问这个问题有被解决么？我遇到了与之相同的问题 image 我去查看了jenkins自己的文档中关于反向代理的说明， ↓上述文档中关于此内容描述的截图 image 根据其中的描述，是jenkins在某些步骤会自己通过302状态码进行重定向，并在http响应头的Location返回要重定向到的地址，但是目前jenkins方面每次返回的Localtion的端口都是apisix的端口，有两种处理方式，其中一种是需要我们在请求头中设置以下参数： X-Forwarded-Host: jenkins.xxx.xxx（自身jenkins的域名） X-Forwarded-Port: 80 X-Forwarded-Proto: https 关于设置请求头的这种方式，这是我尝试过的方法，但都没有效果，jenkins依然返回了带有9080端口的重定向地址： 1、通过界面的“路由”(routes)中的“请求头重写”功能设置请求头 image ↓这是通过该方式形成的路由配置在“数据编辑器”中展示的样子 image 2、通过界面的“服务”(service)中的“proxy-rewrite”插件设置请求头 image ↓这是通过该方式形成的服务配置在“数据编辑器”中展示的样子 image 然后因为我不太确定这是jenkins自身的问题还是apisix插件的问题，于是我现在想要试试另一种修改响应头中的Location的方法，请问有什么方式能让我对响应头中的Location进行正则匹配，并进行修改的么？想要的效果大概是这样的： response header: Location：https://jenkins.xxx.xxx:9080/ ↓ response header: Location：https://jenkins.xxx.xxx/ 我知道有个“response-rewrite”插件可以对上面所说的Location进行修改，但因为该插件提供的说明中并没有对其进行较多的描述，也没有提供类似的改写范例，我并不确定他是否能做到我上面说的这种基于正则的改写，如果可以的话，能否给我说下我应该怎么去配置，谢谢。 除此之外，我目前觉得大概能有效的一种方法是，我将9080端口也开放出去，然后再重新配置一个路由去识别来自9080端口的请求，并将其重新重定向到80端口，但老实说不到万不得已，我并不是太想新开放一个端口，而且这样的处理方式会让多增加一次http的请求，总感觉不太美好。

你字多，就回复你吧。你域名解析到哪里了，是 nginx 还是 slb？如果是 Nginx 加一个 proxy_set_header X-Forwarded-Port 443; slb 应该有单独的选项可以设置。

This problem has existed for a long time, #2777, test nginx on the upper layer of apisix and add proxy_set_header X-Forwarded-Port 443; available. It feels like the simplest workaround at present.

bingoku commented 2 years ago

Brothers, is this problem solved?

tokers commented 2 years ago

兄弟们，这个问题解决吗？

We have a PR https://github.com/apache/apisix/pull/6686 to track this.

By the way, please use English in public channel.

kwanhur commented 2 years ago

One temporary solution, enable redirect plugin and set uri like https://$host:your-server-port$request_uri. It'll return new url on response's header field Location.

zhoutaoo commented 2 years ago

I have the same problem, using k8s + apisix ingress + jenkins.

zhoutaoo commented 2 years ago

I have the same problem, using k8s + apisix ingress + jenkins.

{
  "uris": [
    "/",
    "/*"
  ],
  "name": "ingress_devops-jenkins-base.xxxxx.com_/",
  "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
  "host": "devops-jenkins-base.xxxxx.com",
  "plugins": {
    "response-rewrite": {
      "headers": {
        "Location": "//devops-jenkins-base.xxxxx.com"
      },
      "vars": [
        [
          "status",
          "==",
          302
        ]
      ]
    }
  },
  "upstream_id": "767e1afc",
  "labels": {
    "managed-by": "apisix-ingress-controller"
  },
  "status": 1
}
haozileung commented 2 years ago

For anyone needs workaround: Apply plugin serverless-pre-function to the route config Sample code:


  "plugins": {
        "serverless-pre-function": {
            "phase": "rewrite",
            "functions": [
                "return function(conf, ctx) ngx.var.var_x_forwarded_port = 443 end"
            ]
        }
    },
spacewander commented 2 years ago

Maybe we can allow proxy-rewrite plugin to rewrite x_forwarded_port. What about everyone's opinion?

soulbird commented 2 years ago

Maybe we can allow proxy-rewrite plugin to rewrite x_forwarded_port. What about everyone's opinion?

LGTM

tzssangglass commented 2 years ago

Maybe we can allow proxy-rewrite plugin to rewrite x_forwarded_port. What about everyone's opinion?

agree +1

kwanhur commented 2 years ago

Maybe we can allow proxy-rewrite plugin to rewrite x_forwarded_port. What about everyone's opinion?

What's the action rewrite meaning? Like add update delete ? It seems proxy-rewrite plugin can do it now, see the doc.

spacewander commented 2 years ago

Currently this header is generated via var_x_forwarded_port: https://github.com/apache/apisix/blob/d7e49c91ea5c9371e704d488c6164e416fb63587/apisix/cli/ngx_tpl.lua#L683

spacewander commented 2 years ago

We can apply the same handling like: https://github.com/apache/apisix/blob/d7e49c91ea5c9371e704d488c6164e416fb63587/apisix/init.lua#L228

kwanhur commented 2 years ago

Maybe I still don't get the accurate meanings, I guess that take plugin proxy-rewrite to rewrite attributes x_forwarded_for x_forwarded_proto x_forwarded_host x_forwarded_port like the host scheme attributes.

If these extensible headers supported one by one, it seems not a good idea. Why not reuse the existed attribute headers?

https://github.com/apache/apisix/blob/2800a06f00ff9010d893eb8c4b9c94359cc3a96b/docs/en/latest/plugins/proxy-rewrite.md?plain=1#L43

tzssangglass commented 2 years ago

If these extensible headers supported one by one, it seems not a good idea. Why not reuse the existed attribute headers?

We don't need to extend the schema with new attributes just that proxy-rewrite currently can't rewrite the x_forwarded_port in request headers and we need to support this feature.

tokers commented 2 years ago

Once we let proxy-rewrite plugin support modifying these headers, also take care the precedence between it and proxy_set_header.

tzssangglass commented 2 years ago

If proxy-rewrite can modify x-forward-port, then the following scenario occurs (to be confirmed)

  1. client access http://a.com:80/

  2. upstream response 302 to the http://a.com:80/login

  3. it redirected to the http://a.com:80/login

My questions:

  1. What port is 80? The port exposed by APISIX in the container, like -p 80:9080?
  2. Does the 80 port have anything to do with Jenkins?
hotbaby commented 2 years ago

If Jenkins is deployed in kubernetes, you can add another Nginx container to the pod, forward the traffic to Jenkins through Nginx, and set X-Forwarded-Port to 80. The test can avoid this problem.

server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port 80;
        proxy_pass http://localhost:8080;
    }
}
niuguy commented 1 year ago

For anyone needs workaround: Apply plugin serverless-pre-function to the route config Sample code:

  "plugins": {
        "serverless-pre-function": {
            "phase": "rewrite",
            "functions": [
                "return function(conf, ctx) ngx.var.var_x_forwarded_port = 443 end"
            ]
        }
    },

Thank you so much bro. btw for guys who use ingress controller 

    plugins:
      - name: serverless-pre-function
        enable: true
        config:
          phase: "rewrite"
          functions: ["return function(conf, ctx) ngx.var.var_x_forwarded_port = 443 end"]
brentmjohnson commented 1 year ago

Another alternative is to modify the listen ports for apisix to the following:

apisix:    # universal configurations
  node_listen: 80    # APISIX listening port
  ssl:
    enable: true
      listen:
        - port: 443

Since these are privileged ports, in kubernetes if you don't want to run as root, this will require you to use the following securityContext on the pod (not container):

securityContext:
    sysctls:
      - name: net.ipv4.ip_unprivileged_port_start
        value: "1"