nginxinc / kubernetes-ingress

NGINX and NGINX Plus Ingress Controllers for Kubernetes
https://docs.nginx.com/nginx-ingress-controller
Apache License 2.0
4.64k stars 1.96k forks source link

Location Snippets ignored when using Action.Return #5746

Closed jasonwilliams14 closed 1 month ago

jasonwilliams14 commented 3 months ago

Discussed in https://github.com/nginxinc/kubernetes-ingress/discussions/5733

Originally posted by **privateVoit** June 12, 2024 I want to expose a robots.txt endpoint. This can be done quiet easily with the Action.Return. Additionally I want to add CORS Headers. As there doesn't seem to be an option for this I am using location-snippets. The issue is that the location snippet is ignored. Heres the config: ``` apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: example.com namespace: default spec: host: example.com routes: - path: /robots.txt action: return: body: |- User-agent: * Disallow: / code: 200 type: text/plain location-snippets: | add_header 'Cache-Control' 'private, max-age=0' always; add_header 'Cross-Origin-Resource-Policy' 'cross-origin' always; add_header 'X-Content-Type-Options' 'nosniff' always; add_header 'X-Xss-Protection' '1; mode=block' always; add_header 'X-Frame-Options' 'SAMEORIGIN' always; add_header 'Frame-Options' 'SAMEORIGIN' always; ``` The result using curl is: ``` > GET /robots.txt HTTP/2 > Host: example.com > User-Agent: curl/8.6.0 > Accept: */* > < HTTP/2 200 < server: nginx < date: Wed, 12 Jun 2024 07:46:11 GMT < content-type: text/plain < content-length: 25 < User-agent: * * Connection #0 to host example.com left intact Disallow: /% ``` It's simply missing within the rendered nginx.conf: ``` location @return_0 { default_type "text/plain"; # status code is ignored here, using 0 return 0 "User-agent: * Disallow: /"; } location /robots.txt { set $service ""; status_zone ""; error_page 418 =200 "@return_0"; proxy_intercept_errors on; proxy_pass http://unix:/var/lib/nginx/nginx-418-server.sock; set $default_connection_header close; } ``` I did enable the location-snippets flag and it works when using the proxy option: ``` apiVersion: k8s.nginx.org/v1 kind: VirtualServer metadata: name: example.com namespace: default spec: host: example.com routes: - action: proxy: upstream: nginx-app location-snippets: | add_header 'Cache-Control' 'private, max-age=0' always; add_header 'Cross-Origin-Resource-Policy' 'cross-origin' always; add_header 'X-Content-Type-Options' 'nosniff' always; add_header 'X-Xss-Protection' '1; mode=block' always; add_header 'X-Frame-Options' 'SAMEORIGIN' always; add_header 'Frame-Options' 'SAMEORIGIN' always; path: / upstreams: - name: nginx-app port: 8080 service: nginx-app ``` The location snippet is added to the nginx.conf ``` location / { set $service "nginx-app"; status_zone "nginx-app"; set $resource_type "virtualserver"; set $resource_name "example.com"; set $resource_namespace "default"; add_header 'Cache-Control' 'private, max-age=0' always; add_header 'Cross-Origin-Resource-Policy' 'cross-origin' always; add_header 'X-Content-Type-Options' 'nosniff' always; add_header 'X-Xss-Protection' '1; mode=block' always; add_header 'X-Frame-Options' 'SAMEORIGIN' always; add_header 'Frame-Options' 'SAMEORIGIN' always; ... ``` I couldn't find any corresponding log entries to this. And I tried version 3.1 and 3.5 and they both do not work. Can anybody help?
github-actions[bot] commented 3 months ago

Hi @jasonwilliams14 thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:

Cheers!

brianehlert commented 2 months ago

There is a perception with Action:Proxy that it functions similar to an If or Switch statement. "When this, do this."

The customer in this case expects both the custom body and the path snippets to be returned. The path (location) snippets would be expected to be written at the path/location with the body return nested under.

"for this path, add these headers. And return this body" - if I were to write this as conversation.

jjngx commented 2 months ago

@shaun-nx currently location snippets for action.Return are not supported: link. So, the observed behavior is correct. Are we considering this as a bug or as functionality that needs to be scoped and added to the backlog?

jjngx commented 2 months ago

@jasonwilliams14 @brianehlert @shaun-nx

vs.yaml :

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: example.com
  namespace: default
spec:
  host: example.com
  routes:
  - path: /robots.txt
    action:
      return:
        body: |-
          User-agent: *
          Disallow: /
        code: 200
        type: text/plain
    location-snippets: |
      add_header 'Cache-Control' 'private, max-age=0' always;
      add_header 'Cross-Origin-Resource-Policy' 'cross-origin' always;
      add_header 'X-Content-Type-Options' 'nosniff' always;
      add_header 'X-Xss-Protection' '1; mode=block' always;
      add_header 'X-Frame-Options' 'SAMEORIGIN' always;
      add_header 'Frame-Options' 'SAMEORIGIN' always;

would this output be the correct expected behavior for the use case?

nginx@nginx-ingress-7d778d5dbf-jmtcf:/etc/nginx/conf.d$ cat vs_default_example-bug.com.conf

server {
    listen 80;
    listen [::]:80;

    server_name example-bug.com;
    status_zone example-bug.com;
    set $resource_type "virtualserver";
    set $resource_name "example-bug.com";
    set $resource_namespace "default";

    server_tokens "on";

    location @return_0 {
        default_type "text/plain";

        # status code is ignored here, using 0
        return 0 "User-agent: *
Disallow: /";
    }

    location /robots.txt {
        set $service "";
        status_zone "";
        add_header 'Cache-Control' 'private, max-age=0' always;
        add_header 'Cross-Origin-Resource-Policy' 'cross-origin' always;
        add_header 'X-Content-Type-Options' 'nosniff' always;
        add_header 'X-Xss-Protection' '1; mode=block' always;
        add_header 'X-Frame-Options' 'SAMEORIGIN' always;
        add_header 'Frame-Options' 'SAMEORIGIN' always;

        error_page 418 =200 "@return_0";
        proxy_intercept_errors on;
        proxy_pass http://unix:/var/lib/nginx/nginx-418-server.sock;
        set $default_connection_header close;
    }
}
vepatel commented 2 months ago

@privateVoit https://github.com/nginxinc/kubernetes-ingress/discussions/5733 is being addressed in this issue so closing the original discussion, please feel free to provide any feedback and further details here. Thanks!

jjngx commented 2 months ago

@privateVoit after closer examination of your use case we propose a simpler solution. There is no need to use location snippets. The functionality you are asking about can be achieved using standard Virtual Server configuration options for the action.Return.

The example workflow and configuration:

cafe.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: coffee
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tea
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: tea

cafe-virtual-server.yaml:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  host: cafe.example.com
  tls:
    secret: cafe-secret
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee
  - path: /robots.txt
    action:
      return:
        body: |
          User-Agent: *
          Disallow: /
        code: 200
        type: text/plain
        headers:
        - name: Cache-Control
          value: private, max-age=0 always
        - name: Cross-Origin-Resource-Policy
          value: cross-origin always
        - name: X-Content-Type-Options
          value: nosniff always
        - name: X-Xss-Protection
          value: 1; mode=block always
        - name: X-Frame-Options
          value: SAMEORIGIN always
        - name: Frame-Options
          value: SAMEORIGIN always

Testing:

/tea

root@469a216b7b8f:/# curl -v --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP http://cafe.example.com:$IC_HTTP_PORT/tea
* Added cafe.example.com:30436:172.18.0.2 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 172.18.0.2:30436...
* Connected to cafe.example.com (172.18.0.2) port 30436 (#0)
> GET /tea HTTP/1.1
> Host: cafe.example.com:30436
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.25.5
< Date: Tue, 16 Jul 2024 09:32:57 GMT
< Content-Type: text/plain
< Content-Length: 154
< Connection: keep-alive
< Expires: Tue, 16 Jul 2024 09:32:56 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.8:8080
Server name: tea-596697966f-xgndm
Date: 16/Jul/2024:09:32:57 +0000
URI: /tea
Request ID: df350a8a0ca9f6ff923e6b05a01d13cb

/coffee

root@469a216b7b8f:/# curl -v --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP http://cafe.example.com:$IC_HTTP_PORT/coffee
* Added cafe.example.com:30436:172.18.0.2 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 172.18.0.2:30436...
* Connected to cafe.example.com (172.18.0.2) port 30436 (#0)
> GET /coffee HTTP/1.1
> Host: cafe.example.com:30436
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.25.5
< Date: Tue, 16 Jul 2024 09:33:02 GMT
< Content-Type: text/plain
< Content-Length: 160
< Connection: keep-alive
< Expires: Tue, 16 Jul 2024 09:33:01 GMT
< Cache-Control: no-cache
<
Server address: 10.244.0.9:8080
Server name: coffee-56b44d4c55-jthfn
Date: 16/Jul/2024:09:33:02 +0000
URI: /coffee
Request ID: 7b840eccd2d0669a5dddb33858554ee1
root@469a216b7b8f:/# curl -v --resolve cafe.example.com:$IC_HTTP_PORT:$IC_IP http://cafe.example.com:$IC_HTTP_PORT/robots.txt
* Added cafe.example.com:30436:172.18.0.2 to DNS cache
* Hostname cafe.example.com was found in DNS cache
*   Trying 172.18.0.2:30436...
* Connected to cafe.example.com (172.18.0.2) port 30436 (#0)
> GET /robots.txt HTTP/1.1
> Host: cafe.example.com:30436
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.25.5
< Date: Tue, 16 Jul 2024 09:35:15 GMT
< Content-Type: text/plain
< Content-Length: 26
< Connection: keep-alive
< Cache-Control: private, max-age=0 always
< Cross-Origin-Resource-Policy: cross-origin always
< X-Content-Type-Options: nosniff always
< X-Xss-Protection: 1; mode=block always
< X-Frame-Options: SAMEORIGIN always
< Frame-Options: SAMEORIGIN always
<
User-Agent: *
Disallow: /

Note the generated config file. The required headers are added in the location @return_0:

nginx@nginx-ingress-7cbd85446c-pzdnj:/etc/nginx/conf.d$ cat vs_default_cafe.conf

upstream vs_default_cafe_coffee {
    zone vs_default_cafe_coffee 512k;
    random two least_conn;
    server 10.244.0.7:8080 max_fails=1 fail_timeout=10s max_conns=0;
    server 10.244.0.9:8080 max_fails=1 fail_timeout=10s max_conns=0;

}

upstream vs_default_cafe_tea {
    zone vs_default_cafe_tea 512k;
    random two least_conn;
    server 10.244.0.8:8080 max_fails=1 fail_timeout=10s max_conns=0;

}

server {
    listen 80;
    listen [::]:80;

    server_name cafe.example.com;
    status_zone cafe.example.com;
    set $resource_type "virtualserver";
    set $resource_name "cafe";
    set $resource_namespace "default";
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate $secret_dir_path/default-cafe-secret;
    ssl_certificate_key $secret_dir_path/default-cafe-secret;

    server_tokens "on";

    location @return_0 {
        default_type "text/plain";

        add_header Cache-Control "private, max-age=0 always" always;

        add_header Cross-Origin-Resource-Policy "cross-origin always" always;

        add_header X-Content-Type-Options "nosniff always" always;

        add_header X-Xss-Protection "1; mode=block always" always;

        add_header X-Frame-Options "SAMEORIGIN always" always;

        add_header Frame-Options "SAMEORIGIN always" always;

        # status code is ignored here, using 0
        return 0 "User-Agent: *
Disallow: /
";
    }

    location /tea {
        set $service "tea-svc";
        status_zone "tea-svc";

        set $default_connection_header close;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;

        proxy_buffering on;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $vs_connection_header;
        proxy_pass_request_headers on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host "$host";
        proxy_pass http://vs_default_cafe_tea;
        proxy_next_upstream error timeout;
        proxy_next_upstream_timeout 0s;
        proxy_next_upstream_tries 0;
    }
    location /coffee {
        set $service "coffee-svc";
        status_zone "coffee-svc";

        set $default_connection_header close;
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
        client_max_body_size 1m;

        proxy_buffering on;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $vs_connection_header;
        proxy_pass_request_headers on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host "$host";
        proxy_pass http://vs_default_cafe_coffee;
        proxy_next_upstream error timeout;
        proxy_next_upstream_timeout 0s;
        proxy_next_upstream_tries 0;
    }
    location /robots.txt {
        set $service "";
        status_zone "";

        error_page 418 =200 "@return_0";
        proxy_intercept_errors on;
        proxy_pass http://unix:/var/lib/nginx/nginx-418-server.sock;
        set $default_connection_header close;
    }

}

Could you please re-test your use case using suggested configuration (action.Return) and let us know if it works for you.

cc / @shaun-nx @vepatel

danielnginx commented 2 months ago

@privateVoit did you have a chance to look at the proposed solution?

privateVoit commented 2 months ago

@danielnginx Sorry for the delay. I'm currently not able to test this out. I'll forward this to my colleagues asap.

AlexFenlon commented 1 month ago

@privateVoit We are just wondering if you or your colleagues have had a chance to try the solutions proposed in the replies?

privateVoit commented 1 month ago

@AlexFenlon I am very sorry for the delay. I was able to test this and it worked as expected. Thank you very much and sorry for the delays.

AlexFenlon commented 1 month ago

@privateVoit No worries, thank you for getting back, I am glad that the issue got resolved 👍