Unobvious behavior of canary ingress

zvlb commented 7 months ago

What happened:

If I create an Ingress for a service that is being monitored by another Ingress with canary settings, the newly created Ingress does not work.

Prepare envoronment:

Install kind for testing:

❯ kind create cluster --name k8s-playground --config ./kind.yaml
Creating cluster "k8s-playground" ...
 ✓ Ensuring node image (kindest/node:v1.27.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-k8s-playground"
You can now use your cluster with:

kubectl cluster-info --context kind-k8s-playground

Have a nice day! 👋

❯ cat ./kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Install ingress nginx:

❯ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

Ingress Nginx version:

❯ k exec -it -n ingress-nginx ingress-nginx-controller-d9b6d7bf5-wbcfj -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.4
  Build:         846d251814a09d8a5d8d28e2e604bfc7749bcb49
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Deploy all manifests from THIS documentation:

❯ k get all
NAME                             READY   STATUS    RESTARTS   AGE
pod/canary-546f4b94f6-nphdn      1/1     Running   0          64s
pod/production-94549b7b6-rrvqj   1/1     Running   0          71s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/canary       ClusterIP   10.96.217.3    <none>        80/TCP    64s
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   26m
service/production   ClusterIP   10.96.131.69   <none>        80/TCP    71s

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/canary       1/1     1            1           64s
deployment.apps/production   1/1     1            1           71s

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/canary-546f4b94f6      1         1         1       64s
replicaset.apps/production-94549b7b6   1         1         1       71s

❯ k get ing
NAME         CLASS   HOSTS                    ADDRESS     PORTS   AGE
canary       nginx   echo.prod.mydomain.com   localhost   80      72s
production   nginx   echo.prod.mydomain.com   localhost   80      76s

Test it:

❯ for i in $(seq 1 10); do curl -s --resolve echo.prod.mydomain.com:80:127.0.0.1 echo.prod.mydomain.com  | grep "Hostname"; done

Hostname: canary-546f4b94f6-nphdn
Hostname: canary-546f4b94f6-nphdn
Hostname: canary-546f4b94f6-nphdn
Hostname: canary-546f4b94f6-nphdn
Hostname: canary-546f4b94f6-nphdn
Hostname: canary-546f4b94f6-nphdn
Hostname: production-94549b7b6-rrvqj
Hostname: production-94549b7b6-rrvqj
Hostname: production-94549b7b6-rrvqj
Hostname: canary-546f4b94f6-nphdn

Show what work wrong:

Then I create NEW ingress for NEW domain, but with canary service:

❯ k apply -f ingress.yaml
ingress.networking.k8s.io/test created

❯ cat ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: test.prod.mydomain.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: canary
            port:
              number: 80

But this ingress not work:

❯ curl -s --resolve test.prod.mydomain.com:80:127.0.0.1 test.prod.mydomain.com
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

In nginx-ingress-controller logs:

172.18.0.1 - - [06/Dec/2023:13:02:04 +0000] "GET / HTTP/1.1" 404 146 "-" "curl/7.88.1" 86 0.000 [upstream-default-backend] [] 127.0.0.1:8181 146 0.001 404 46afc979d0173e850948ab84150f170a

I don’t understand why my new ingress and new domain (test.prod.mydomain.com) doesn’t work(

zvlb commented 7 months ago

Nginx configuration

k exec -it -n ingress-nginx ingress-nginx-controller-d9b6d7bf5-wbcfj -- nginx -T

``` nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # configuration file /etc/nginx/nginx.conf: # Configuration checksum: 8632516323550193052 # setup custom paths that do not require root access pid /tmp/nginx/nginx.pid; daemon off; worker_processes 6; worker_rlimit_nofile 1047552; worker_shutdown_timeout 240s ; events { multi_accept on; worker_connections 16384; use epoll; } http { lua_package_path "/etc/nginx/lua/?.lua;;"; lua_shared_dict balancer_ewma 10M; lua_shared_dict balancer_ewma_last_touched_at 10M; lua_shared_dict balancer_ewma_locks 1M; lua_shared_dict certificate_data 20M; lua_shared_dict certificate_servers 5M; lua_shared_dict configuration_data 20M; lua_shared_dict global_throttle_cache 10M; lua_shared_dict ocsp_response_cache 5M; init_by_lua_block { collectgarbage("collect") -- init modules local ok, res ok, res = pcall(require, "lua_ingress") if not ok then error("require failed: " .. tostring(res)) else lua_ingress = res lua_ingress.set_config({ use_forwarded_headers = false, use_proxy_protocol = false, is_ssl_passthrough_enabled = false, http_redirect_code = 308, listen_ports = { ssl_proxy = "442", https = "443" }, hsts = true, hsts_max_age = 15724800, hsts_include_subdomains = true, hsts_preload = false, global_throttle = { memcached = { host = "", port = 11211, connect_timeout = 50, max_idle_timeout = 10000, pool_size = 50, }, status_code = 429, } }) end ok, res = pcall(require, "configuration") if not ok then error("require failed: " .. tostring(res)) else configuration = res configuration.prohibited_localhost_port = '10246' end ok, res = pcall(require, "balancer") if not ok then error("require failed: " .. tostring(res)) else balancer = res end ok, res = pcall(require, "monitor") if not ok then error("require failed: " .. tostring(res)) else monitor = res end ok, res = pcall(require, "certificate") if not ok then error("require failed: " .. tostring(res)) else certificate = res certificate.is_ocsp_stapling_enabled = false end ok, res = pcall(require, "plugins") if not ok then error("require failed: " .. tostring(res)) else plugins = res end -- load all plugins that'll be used here plugins.init({ }) } init_worker_by_lua_block { lua_ingress.init_worker() balancer.init_worker() monitor.init_worker(10000) plugins.run() } geoip_country /etc/nginx/geoip/GeoIP.dat; geoip_city /etc/nginx/geoip/GeoLiteCity.dat; geoip_org /etc/nginx/geoip/GeoIPASNum.dat; geoip_proxy_recursive on; aio threads; aio_write on; tcp_nopush on; tcp_nodelay on; log_subrequest on; reset_timedout_connection on; keepalive_timeout 75s; keepalive_requests 1000; client_body_temp_path /tmp/nginx/client-body; fastcgi_temp_path /tmp/nginx/fastcgi-temp; proxy_temp_path /tmp/nginx/proxy-temp; client_header_buffer_size 1k; client_header_timeout 60s; large_client_header_buffers 4 8k; client_body_buffer_size 8k; client_body_timeout 60s; http2_max_concurrent_streams 128; types_hash_max_size 2048; server_names_hash_max_size 1024; server_names_hash_bucket_size 64; map_hash_bucket_size 64; proxy_headers_hash_max_size 512; proxy_headers_hash_bucket_size 64; variables_hash_bucket_size 256; variables_hash_max_size 2048; underscores_in_headers off; ignore_invalid_headers on; limit_req_status 503; limit_conn_status 503; include /etc/nginx/mime.types; default_type text/html; # Custom headers for response server_tokens off; more_clear_headers Server; # disable warnings uninitialized_variable_warn off; # Additional available variables: # $namespace # $ingress_name # $service_name # $service_port log_format upstreaminfo '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id'; map $request_uri $loggable { default 1; } access_log /var/log/nginx/access.log upstreaminfo if=$loggable; error_log /var/log/nginx/error.log notice; resolver 10.96.0.10 valid=30s; # See https://www.nginx.com/blog/websocket-nginx map $http_upgrade $connection_upgrade { default upgrade; # See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive '' ''; } # Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server. # If no such header is provided, it can provide a random value. map $http_x_request_id $req_id { default $http_x_request_id; "" $request_id; } # Create a variable that contains the literal $ character. # This works because the geo module will not resolve variables. geo $literal_dollar { default "$"; } server_name_in_redirect off; port_in_redirect off; ssl_protocols TLSv1.2 TLSv1.3; ssl_early_data off; # turn on session caching to drastically improve performance ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; # allow configuring ssl session tickets ssl_session_tickets off; # slightly reduce the time-to-first-byte ssl_buffer_size 4k; # allow configuring custom ssl ciphers ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384'; ssl_prefer_server_ciphers on; ssl_ecdh_curve auto; # PEM sha: 741488fc39707bfaa3dbddc85dfc22242e9a8e0f ssl_certificate /etc/ingress-controller/ssl/default-fake-certificate.pem; ssl_certificate_key /etc/ingress-controller/ssl/default-fake-certificate.pem; proxy_ssl_session_reuse on; upstream upstream_balancer { ### Attention!!! # # We no longer create "upstream" section for every backend. # Backends are handled dynamically using Lua. If you would like to debug # and see what backends ingress-nginx has in its memory you can # install our kubectl plugin https://kubernetes.github.io/ingress-nginx/kubectl-plugin. # Once you have the plugin you can use "kubectl ingress-nginx backends" command to # inspect current backends. # ### server 0.0.0.1; # placeholder balancer_by_lua_block { balancer.balance() } keepalive 320; keepalive_time 1h; keepalive_timeout 60s; keepalive_requests 10000; } # Cache for internal auth checks proxy_cache_path /tmp/nginx/nginx-cache-auth levels=1:2 keys_zone=auth_cache:10m max_size=128m inactive=30m use_temp_path=off; # Global filters ## start server _ server { server_name _ ; listen 80 default_server reuseport backlog=4096 ; listen [::]:80 default_server reuseport backlog=4096 ; listen 443 default_server reuseport backlog=4096 ssl http2 ; listen [::]:443 default_server reuseport backlog=4096 ssl http2 ; set $proxy_upstream_name "-"; ssl_reject_handshake off; ssl_certificate_by_lua_block { certificate.call() } location / { set $namespace ""; set $ingress_name ""; set $service_name ""; set $service_port ""; set $location_path ""; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = false, force_no_ssl_redirect = false, preserve_trailing_slash = false, use_port_in_redirects = false, global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } }, }) balancer.rewrite() plugins.run() } # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)` # other authentication method such as basic auth or external auth useless - all requests will be allowed. #access_by_lua_block { #} header_filter_by_lua_block { lua_ingress.header() plugins.run() } body_filter_by_lua_block { plugins.run() } log_by_lua_block { balancer.log() monitor.call() plugins.run() } access_log off; port_in_redirect off; set $balancer_ewma_score -1; set $proxy_upstream_name "upstream-default-backend"; set $proxy_host $proxy_upstream_name; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; set $proxy_alternative_upstream_name ""; client_max_body_size 1m; proxy_set_header Host $best_http_host; # Pass the extracted client certificate to the backend # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Request-ID $req_id; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Host $best_http_host; proxy_set_header X-Forwarded-Port $pass_port; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_set_header X-Forwarded-Scheme $pass_access_scheme; proxy_set_header X-Scheme $pass_access_scheme; # Pass the original X-Forwarded-For proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for; # mitigate HTTPoxy Vulnerability # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ proxy_set_header Proxy ""; # Custom headers to proxied server proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_buffering off; proxy_buffer_size 4k; proxy_buffers 4 4k; proxy_max_temp_file_size 1024m; proxy_request_buffering on; proxy_http_version 1.1; proxy_cookie_domain off; proxy_cookie_path off; # In case of errors try the next upstream server before returning an error proxy_next_upstream error timeout; proxy_next_upstream_timeout 0; proxy_next_upstream_tries 3; proxy_pass http://upstream_balancer; proxy_redirect off; } # health checks in cloud providers require the use of port 80 location /healthz { access_log off; return 200; } # this is required to avoid error if nginx is being monitored # with an external software (like sysdig) location /nginx_status { allow 127.0.0.1; allow ::1; deny all; access_log off; stub_status on; } } ## end server _ ## start server echo.prod.mydomain.com server { server_name echo.prod.mydomain.com ; listen 80 ; listen [::]:80 ; listen 443 ssl http2 ; listen [::]:443 ssl http2 ; set $proxy_upstream_name "-"; ssl_certificate_by_lua_block { certificate.call() } location / { set $namespace "default"; set $ingress_name "production"; set $service_name "production"; set $service_port "80"; set $location_path "/"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, preserve_trailing_slash = false, use_port_in_redirects = false, global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } }, }) balancer.rewrite() plugins.run() } # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)` # other authentication method such as basic auth or external auth useless - all requests will be allowed. #access_by_lua_block { #} header_filter_by_lua_block { lua_ingress.header() plugins.run() } body_filter_by_lua_block { plugins.run() } log_by_lua_block { balancer.log() monitor.call() plugins.run() } port_in_redirect off; set $balancer_ewma_score -1; set $proxy_upstream_name "default-production-80"; set $proxy_host $proxy_upstream_name; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; set $proxy_alternative_upstream_name ""; client_max_body_size 1m; proxy_set_header Host $best_http_host; # Pass the extracted client certificate to the backend # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Request-ID $req_id; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Host $best_http_host; proxy_set_header X-Forwarded-Port $pass_port; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_set_header X-Forwarded-Scheme $pass_access_scheme; proxy_set_header X-Scheme $pass_access_scheme; # Pass the original X-Forwarded-For proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for; # mitigate HTTPoxy Vulnerability # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ proxy_set_header Proxy ""; # Custom headers to proxied server proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_buffering off; proxy_buffer_size 4k; proxy_buffers 4 4k; proxy_max_temp_file_size 1024m; proxy_request_buffering on; proxy_http_version 1.1; proxy_cookie_domain off; proxy_cookie_path off; # In case of errors try the next upstream server before returning an error proxy_next_upstream error timeout; proxy_next_upstream_timeout 0; proxy_next_upstream_tries 3; proxy_pass http://upstream_balancer; proxy_redirect off; } } ## end server echo.prod.mydomain.com ## start server test.prod.mydomain.com server { server_name test.prod.mydomain.com ; listen 80 ; listen [::]:80 ; listen 443 ssl http2 ; listen [::]:443 ssl http2 ; set $proxy_upstream_name "-"; ssl_certificate_by_lua_block { certificate.call() } location / { set $namespace "default"; set $ingress_name "test"; set $service_name "canary"; set $service_port "80"; set $location_path "/"; set $global_rate_limit_exceeding n; rewrite_by_lua_block { lua_ingress.rewrite({ force_ssl_redirect = false, ssl_redirect = true, force_no_ssl_redirect = false, preserve_trailing_slash = false, use_port_in_redirects = false, global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } }, }) balancer.rewrite() plugins.run() } # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)` # other authentication method such as basic auth or external auth useless - all requests will be allowed. #access_by_lua_block { #} header_filter_by_lua_block { lua_ingress.header() plugins.run() } body_filter_by_lua_block { plugins.run() } log_by_lua_block { balancer.log() monitor.call() plugins.run() } port_in_redirect off; set $balancer_ewma_score -1; set $proxy_upstream_name "upstream-default-backend"; set $proxy_host $proxy_upstream_name; set $pass_access_scheme $scheme; set $pass_server_port $server_port; set $best_http_host $http_host; set $pass_port $pass_server_port; set $proxy_alternative_upstream_name ""; client_max_body_size 1m; proxy_set_header Host $best_http_host; # Pass the extracted client certificate to the backend # Allow websocket connections proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_set_header X-Request-ID $req_id; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Host $best_http_host; proxy_set_header X-Forwarded-Port $pass_port; proxy_set_header X-Forwarded-Proto $pass_access_scheme; proxy_set_header X-Forwarded-Scheme $pass_access_scheme; proxy_set_header X-Scheme $pass_access_scheme; # Pass the original X-Forwarded-For proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for; # mitigate HTTPoxy Vulnerability # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ proxy_set_header Proxy ""; # Custom headers to proxied server proxy_connect_timeout 5s; proxy_send_timeout 60s; proxy_read_timeout 60s; proxy_buffering off; proxy_buffer_size 4k; proxy_buffers 4 4k; proxy_max_temp_file_size 1024m; proxy_request_buffering on; proxy_http_version 1.1; proxy_cookie_domain off; proxy_cookie_path off; # In case of errors try the next upstream server before returning an error proxy_next_upstream error timeout; proxy_next_upstream_timeout 0; proxy_next_upstream_tries 3; proxy_pass http://upstream_balancer; proxy_redirect off; } } ## end server test.prod.mydomain.com # backend for when default-backend-service is not configured or it does not have endpoints server { listen 8181 default_server reuseport backlog=4096; listen [::]:8181 default_server reuseport backlog=4096; set $proxy_upstream_name "internal"; access_log off; location / { return 404; } } # default server, used for NGINX healthcheck and access to nginx stats server { # Ensure that modsecurity will not run on an internal location as this is not accessible from outside listen 127.0.0.1:10246; set $proxy_upstream_name "internal"; keepalive_timeout 0; gzip off; access_log off; location /healthz { return 200; } location /is-dynamic-lb-initialized { content_by_lua_block { local configuration = require("configuration") local backend_data = configuration.get_backends_data() if not backend_data then ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR) return end ngx.say("OK") ngx.exit(ngx.HTTP_OK) } } location /nginx_status { stub_status on; } location /configuration { client_max_body_size 21M; client_body_buffer_size 21M; proxy_buffering off; content_by_lua_block { configuration.call() } } location / { content_by_lua_block { ngx.exit(ngx.HTTP_NOT_FOUND) } } } } stream { lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;;"; lua_shared_dict tcp_udp_configuration_data 5M; resolver 10.96.0.10 valid=30s; init_by_lua_block { collectgarbage("collect") -- init modules local ok, res ok, res = pcall(require, "configuration") if not ok then error("require failed: " .. tostring(res)) else configuration = res end ok, res = pcall(require, "tcp_udp_configuration") if not ok then error("require failed: " .. tostring(res)) else tcp_udp_configuration = res tcp_udp_configuration.prohibited_localhost_port = '10246' end ok, res = pcall(require, "tcp_udp_balancer") if not ok then error("require failed: " .. tostring(res)) else tcp_udp_balancer = res end } init_worker_by_lua_block { tcp_udp_balancer.init_worker() } lua_add_variable $proxy_upstream_name; log_format log_stream '[$remote_addr] [$time_local] $protocol $status $bytes_sent $bytes_received $session_time'; access_log /var/log/nginx/access.log log_stream ; error_log /var/log/nginx/error.log notice; upstream upstream_balancer { server 0.0.0.1:1234; # placeholder balancer_by_lua_block { tcp_udp_balancer.balance() } } server { listen 127.0.0.1:10247; access_log off; content_by_lua_block { tcp_udp_configuration.call() } } # TCP services # UDP services # Stream Snippets } # configuration file /etc/nginx/mime.types: types { text/html html htm shtml; text/css css; text/xml xml; image/gif gif; image/jpeg jpeg jpg; application/javascript js; application/atom+xml atom; application/rss+xml rss; text/mathml mml; text/plain txt; text/vnd.sun.j2me.app-descriptor jad; text/vnd.wap.wml wml; text/x-component htc; image/avif avif; image/png png; image/svg+xml svg svgz; image/tiff tif tiff; image/vnd.wap.wbmp wbmp; image/webp webp; image/x-icon ico; image/x-jng jng; image/x-ms-bmp bmp; font/woff woff; font/woff2 woff2; application/java-archive jar war ear; application/json json; application/mac-binhex40 hqx; application/msword doc; application/pdf pdf; application/postscript ps eps ai; application/rtf rtf; application/vnd.apple.mpegurl m3u8; application/vnd.google-earth.kml+xml kml; application/vnd.google-earth.kmz kmz; application/vnd.ms-excel xls; application/vnd.ms-fontobject eot; application/vnd.ms-powerpoint ppt; application/vnd.oasis.opendocument.graphics odg; application/vnd.oasis.opendocument.presentation odp; application/vnd.oasis.opendocument.spreadsheet ods; application/vnd.oasis.opendocument.text odt; application/vnd.openxmlformats-officedocument.presentationml.presentation pptx; application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx; application/vnd.openxmlformats-officedocument.wordprocessingml.document docx; application/vnd.wap.wmlc wmlc; application/wasm wasm; application/x-7z-compressed 7z; application/x-cocoa cco; application/x-java-archive-diff jardiff; application/x-java-jnlp-file jnlp; application/x-makeself run; application/x-perl pl pm; application/x-pilot prc pdb; application/x-rar-compressed rar; application/x-redhat-package-manager rpm; application/x-sea sea; application/x-shockwave-flash swf; application/x-stuffit sit; application/x-tcl tcl tk; application/x-x509-ca-cert der pem crt; application/x-xpinstall xpi; application/xhtml+xml xhtml; application/xspf+xml xspf; application/zip zip; application/octet-stream bin exe dll; application/octet-stream deb; application/octet-stream dmg; application/octet-stream iso img; application/octet-stream msi msp msm; audio/midi mid midi kar; audio/mpeg mp3; audio/ogg ogg; audio/x-m4a m4a; audio/x-realaudio ra; video/3gpp 3gpp 3gp; video/mp2t ts; video/mp4 mp4; video/mpeg mpeg mpg; video/quicktime mov; video/webm webm; video/x-flv flv; video/x-m4v m4v; video/x-mng mng; video/x-ms-asf asx asf; video/x-ms-wmv wmv; video/x-msvideo avi; } ```

longwuyuan commented 7 months ago

/remove-kind bug @zvlb , While we have to look at the implementation on why this did not work for you, it will help a lot if you can explain the real world use case here. We discussed in the community meeting just now and could not understand why a user would create a ingress for a backend service, that is already the configured backend canary service for another service.

longwuyuan commented 7 months ago

/triage needs-information

Gacko commented 7 months ago

/assign

zvlb commented 7 months ago

it will help a lot if you can explain the real world use case here

Oh. I have a problem with english, but I will try)

In my case I have production application, and I want test new release with 1% requests. (To gather user experience). Then I have 2 deployment, 2 services and 2 ingresses. (For main and new application version)

In this time my QA team say: "We want access to new release on production". Then I create new ingress for service with new application version with new internal-domain, but it's not work.

(I find fast fix - I create new service for new release (with new name and without ingresses), and create Ingress for internal domain with new service and it work good)

It's my real case)

Gacko commented 7 months ago

Hey @zvlb! That's also the use case we were assuming when we went through the issues recently.

From a users expectation of how the Ingress API and this feature in particular work, one should not need to create a third service for this use case.

I'll try to reproduce and debug this issue as soon as I got time for it. Just to be sure I'm getting you right: You just followed the instructions of the guide and added on extra Ingress pointing to the canary / second service so you can access it with a 100% chance for testing purposes.

zvlb commented 7 months ago

Just to be sure I'm getting you right: You just followed the instructions of the guide and added on extra Ingress pointing to the canary / second service so you can access it with a 100% chance for testing purposes.

Correct

Gacko commented 7 months ago

/triage accepted

Gacko commented 7 months ago

So I put some effort into testing this on my own cluster and actually it's far from perfect:

If you first create ...

Deployment A
Service A pointing to Deployment A
Ingress A pointing to Service A

... and then create ...

Deployment B
Service B pointing to Deployment B
Ingress B pointing to Service B

... you can access both Ingresses without any problems - as expected.

When you now create the canary Ingress A pointing to Service B, it's not getting taken into account at all.

If you create ...

Deployment A
Service A pointing to Deployment A
Ingress A pointing to Service A
Canary Ingress A pointing to Service B

... and then create ...

Deployment B
Service B pointing to Deployment B
Ingress B pointing to Service B

... the canary feature for Ingress A works as expected, but you can not access Service B via Ingress B at all. Same goes for the first scenario and deleting Ingress B.

I feel like it's a question of order and that the backend for Service B is somehow getting "consumed" and therefore not available for an extra Ingress anymore. In the nginx.conf you can see that the backend for Ingress B is the default 404 backend when you follow the guide.

I'll now try to have a look into the code and check what's happening there.

Gacko commented 7 months ago

I continued working on this. As your logs already tell, the upstream upstream-default-backend is getting set for this Ingress while it should be your service instead. Unfortunately I haven't managed to dig down this rabbit hole, yet.

Gacko commented 7 months ago

Here are my manifests and the resulting nginx.conf. Storing them here so they don't get lost.

canary.tar.gz

kubernetes / ingress-nginx

Unobvious behavior of canary ingress #10729