Closed vadlakiran closed 8 months ago
Hi @vadlakiran thanks for reporting!
Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:
I've parsed the text of your issue and it looks like you might be mixing up the two Ingress Controllers, please take a look at this page to see the differences between nginxinc/kubernetes-ingress
(this repo) and kubernetes/ingress-nginx
.
Cheers!
Hi @vadlakiran looking at annotation nginx.ingress.kubernetes.io/whitelist-source-range: "ip address"
and version v1.10.1,
seems like you're using different project kubernetes/ingress-nginx. I recommend creating an issue there instead.
Thanks!
@vepatel we have refered this page and installed long back https://docs.nginx.com/nginx-ingress-controller/technical-specifications/ as i checked it we are using this nginxinc/kubernetes-ingress repo only, could you please suggest it, we are deploying as daemonset as below
`apiVersion: apps/v1 kind: DaemonSet metadata: annotations: deprecated.daemonset.template.generation: "1" meta.helm.sh/release-name: nginx-controller meta.helm.sh/release-namespace: default creationTimestamp: "2023-07-10T10:09:06Z" generation: 1 labels: app.kubernetes.io/managed-by: Helm name: nginx-ingress namespace: default resourceVersion: "10717027" uid: a5eff5e1-2eaa-433d-88c4-6b754b8af0d3 spec: revisionHistoryLimit: 10 selector: matchLabels: app: nginx-ingress template: metadata: creationTimestamp: null labels: app: nginx-ingress spec: containers:
`
sorry, yeah this daemon-set belongs to this project but the build you're using is more than 2 years old as the latest release is 3.2.0
.
The annotation nginx.ingress.kubernetes.io/whitelist-source-range: "ip address"
does not belong to this project but the one mentioned in my earlier comment.
https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/#accesscontrol might fit your use-case. eg: https://github.com/nginxinc/kubernetes-ingress/tree/v3.2.0/examples/custom-resources/access-control
yeah, i agree that annotation nginx.ingress.kubernetes.io/whitelist-source-range: "ip address" mentioned is not belongs to this project, i just tried likewise do we have any annotations ?
what is the latest version which supports k8s version v1.21.5 with ip witelisting ? any suggestion please
https://docs.nginx.com/nginx-ingress-controller/releases/#nginx-ingress-controller-302 is the last release with 1.21 support and please use the example https://github.com/nginxinc/kubernetes-ingress/tree/v3.0.2/examples/custom-resources/access-control/
Thank you @vepatel for the example of the access-control i tried it but its supports only virtualsroute and virtualserver. how to make it work with nginx ingress controller ?
Unfortunately k8s Ingress resource does not natively support access-control. virtualserverroute
and virtualserver
are custom resources used by nginx ingress controller and this functionality is natively supported by them through policy.
You can use location-snippets to insert nginx config required to enable this:
@vepatel i have tried with location-snippets but did not work, i have given like below for location-snippets and on daemonset mentioned args
`apiVersion: v1 data: client-max-body-size: "0" http-snippets: | add_header X-Frame-Options SAMEORIGIN always; nginx.status.tmpl: "server {\n listen 8080;\n{{- $cidr := .Values.nginx_status_allow_cidrs | default \"\" }}\n{{- range (splitList \",\" $cidr) }}\n allow {{.}};\n{{- end }}\n deny all;\n\n location /stub_status {\n stub_status;\n } \n }\n" nginx.virtualserver.tmpl: |2
{{ range $u := .Upstreams }}
upstream {{ $u.Name }} {
{{ if ne $u.UpstreamZoneSize "0" }}zone {{ $u.Name }} {{ $u.UpstreamZoneSize }};{{ end }}
{{ if $u.LBMethod }}{{ $u.LBMethod }};{{ end }}
{{ range $s := $u.Servers }}
server {{with $x := $s.Address | len}}{{with $y := slice $s.Address 3 | len}}[{{slice $s.Address 0 $y}}]{{slice $s.Ad dress $y $x}}{{end}}{{end}} max_fails={{ $u.MaxFails }} fail_timeout={{ $u.FailTimeout }} max_conns={{ $u.MaxConns }};
{{ end }}
{{ if $u.Keepalive }}
keepalive {{ $u.Keepalive }};
{{ end }}
}
{{ end }}
{{ range $sc := .SplitClients }}
split_clients {{ $sc.Source }} {{ $sc.Variable }} {
{{ range $d := $sc.Distributions }}
{{ $d.Weight }} {{ $d.Value }};
{{ end }}
}
{{ end }}
{{ range $m := .Maps }}
map {{ $m.Source }} {{ $m.Variable }} {
{{ range $p := $m.Parameters }}
{{ $p.Value }} {{ $p.Result }};
{{ end }}
}
{{ end }}
{{ range $snippet := .HTTPSnippets }}
{{- $snippet }}
{{ end }}
{{ range $z := .LimitReqZones }}
limit_req_zone {{ $z.Key }} zone={{ $z.ZoneName }}:{{ $z.ZoneSize }} rate={{ $z.Rate }};
{{ end }}
{{ $s := .Server }}
server {
listen [::]:80{{ if $s.ProxyProtocol }} proxy_protocol{{ end }};
server_name {{ $s.ServerName }};
set $resource_type "virtualserver";
set $resource_name "{{$s.VSName}}";
set $resource_namespace "{{$s.VSNamespace}}";
{{ with $ssl := $s.SSL }}
{{ if $s.TLSPassthrough }}
listen unix:/var/lib/nginx/passthrough-https.sock{{ if $ssl.HTTP2 }} http2{{ end }} proxy_protocol;
set_real_ip_from unix:;
real_ip_header proxy_protocol;
{{ else }}
listen [::]:443 ssl{{ if $ssl.HTTP2 }} http2{{ end }}{{ if $s.ProxyProtocol }} proxy_protocol{{ end }};
{{ end }}
ssl_certificate {{ $ssl.Certificate }};
ssl_certificate_key {{ $ssl.CertificateKey }};
{{ if $ssl.Ciphers }}
ssl_ciphers {{ $ssl.Ciphers }};
{{ end }}
{{ end }}
{{ with $s.IngressMTLS }}
ssl_client_certificate {{ .ClientCert }};
ssl_verify_client {{ .VerifyClient }};
ssl_verify_depth {{ .VerifyDepth }};
{{ end }}
{{ with $s.TLSRedirect }}
if ({{ .BasedOn }} = 'http') {
return {{ .Code }} https://$host$request_uri;
}
{{ end }}
server_tokens "{{ $s.ServerTokens }}";
{{ range $setRealIPFrom := $s.SetRealIPFrom }}
set_real_ip_from {{ $setRealIPFrom }};
{{ end }}
{{ if $s.RealIPHeader }}
real_ip_header {{ $s.RealIPHeader }};
{{ end }}
{{ if $s.RealIPRecursive }}
real_ip_recursive on;
{{ end }}
{{ with $s.PoliciesErrorReturn }}
return {{ .Code }};
{{ end }}
{{ range $allow := $s.Allow }}
allow {{ $allow }};
{{ end }}
{{ if gt (len $s.Allow) 0 }}
deny all;
{{ end }}
{{ range $deny := $s.Deny }}
deny {{ $deny }};
{{ end }}
{{ if gt (len $s.Deny) 0 }}
allow all;
{{ end }}
{{ if $s.LimitReqOptions.DryRun }}
limit_req_dry_run on;
{{ end }}
{{ with $level := $s.LimitReqOptions.LogLevel }}
limit_req_log_level {{ $level }};
{{ end }}
{{ with $code := $s.LimitReqOptions.RejectCode }}
limit_req_status {{ $code }};
{{ end }}
{{ range $rl := $s.LimitReqs }}
limit_req zone={{ $rl.ZoneName }}{{ if $rl.Burst }} burst={{ $rl.Burst }}{{ end }}
{{ if $rl.Delay }} delay={{ $rl.Delay }}{{ end }}{{ if $rl.NoDelay }} nodelay{{ end }};
{{ end }}
{{ with $s.EgressMTLS }}
{{ if .Certificate }}
proxy_ssl_certificate {{ .Certificate }};
proxy_ssl_certificate_key {{ .CertificateKey }};
{{ end }}
{{ if .TrustedCert }}
proxy_ssl_trusted_certificate {{ .TrustedCert }};
{{ end }}
proxy_ssl_verify {{ if .VerifyServer }}on{{else}}off{{end}};
proxy_ssl_verify_depth {{ .VerifyDepth }};
proxy_ssl_protocols {{ .Protocols }};
proxy_ssl_ciphers {{ .Ciphers }};
proxy_ssl_session_reuse {{ if .SessionReuse }}on{{else}}off{{end}};
proxy_ssl_server_name {{ if .ServerName }}on{{else}}off{{end}};
proxy_ssl_name {{ .SSLName }};
{{ end }}
{{ range $snippet := $s.Snippets }}
{{- $snippet }}
{{ end }}
{{ range $l := $s.InternalRedirectLocations }}
location {{ $l.Path }} {
rewrite ^ {{ $l.Destination }} last;
}
{{ end }}
{{ range $e := $s.ErrorPageLocations }}
location {{ $e.Name }} {
{{ if $e.DefaultType }}
default_type "{{ $e.DefaultType }}";
{{ end }}
{{ range $h := $e.Headers }}
add_header {{ $h.Name }} "{{ $h.Value }}" always;
{{ end }}
return 0 "{{ $e.Return.Text }}";
}
{{ end }}
{{ range $l := $s.ReturnLocations }}
location {{ $l.Name }} {
default_type "{{ $l.DefaultType }}";
return 0 "{{ $l.Return.Text }}";
}
{{ end }}
{{ range $l := $s.Locations }}
location {{ $l.Path }} {
set $service "{{ $l.ServiceName }}";
{{ if $l.IsVSR }}
set $resource_type "virtualserverroute";
set $resource_name "{{ $l.VSRName }}";
set $resource_namespace "{{ $l.VSRNamespace }}";
{{ end }}
{{ if $l.Internal }}
internal;
{{ end }}
{{ range $snippet := $l.Snippets }}
{{- $snippet }}
{{ end }}
{{ with $l.PoliciesErrorReturn }}
return {{ .Code }};
{{ end }}
{{ range $allow := $l.Allow }}
allow {{ $allow }};
{{ end }}
{{ if gt (len $l.Allow) 0 }}
deny all;
{{ end }}
{{ range $deny := $l.Deny }}
deny {{ $deny }};
{{ end }}
{{ if gt (len $l.Deny) 0 }}
allow all;
{{ end }}
{{ if $l.LimitReqOptions.DryRun }}
limit_req_dry_run on;
{{ end }}
{{ with $level := $l.LimitReqOptions.LogLevel }}
limit_req_log_level {{ $level }};
{{ end }}
{{ with $code := $l.LimitReqOptions.RejectCode }}
limit_req_status {{ $code }};
{{ end }}
{{ range $rl := $l.LimitReqs }}
limit_req zone={{ $rl.ZoneName }}{{ if $rl.Burst }} burst={{ $rl.Burst }}{{ end }}
{{ if $rl.Delay }} delay={{ $rl.Delay }}{{ end }}{{ if $rl.NoDelay }} nodelay{{ end }};
{{ end }}
{{ with $l.EgressMTLS }}
{{ if .Certificate }}
proxy_ssl_certificate {{ .Certificate }};
proxy_ssl_certificate_key {{ .CertificateKey }};
{{ end }}
{{ if .TrustedCert }}
proxy_ssl_trusted_certificate {{ .TrustedCert }};
{{ end }}
proxy_ssl_verify {{ if .VerifyServer }}on{{else}}off{{end}};
proxy_ssl_verify_depth {{ .VerifyDepth }};
proxy_ssl_protocols {{ .Protocols }};
proxy_ssl_ciphers {{ .Ciphers }};
proxy_ssl_session_reuse {{ if .SessionReuse }}on{{else}}off{{end}};
proxy_ssl_server_name {{ if .ServerName }}on{{else}}off{{end}};
proxy_ssl_name {{ .SSLName }};
{{ end }}
{{ range $e := $l.ErrorPages }}
error_page {{ $e.Codes }} {{ if ne 0 $e.ResponseCode }}={{ $e.ResponseCode }}{{ end }} "{{ $e.Name }}";
{{ end }}
{{ if $l.ProxyInterceptErrors }}
proxy_intercept_errors on;
{{ end }}
{{ if $l.InternalProxyPass }}
proxy_pass {{ $l.InternalProxyPass }};
{{ end }}
{{ if $l.ProxyPass }}
set $default_connection_header {{ if $l.HasKeepalive }}""{{ else }}close{{ end }};
{{ range $r := $l.Rewrites }}
rewrite {{ $r }};
{{ end }}
proxy_connect_timeout {{ $l.ProxyConnectTimeout }};
proxy_read_timeout {{ $l.ProxyReadTimeout }};
proxy_send_timeout {{ $l.ProxySendTimeout }};
client_max_body_size {{ $l.ClientMaxBodySize }};
{{ if $l.ProxyMaxTempFileSize }}
proxy_max_temp_file_size {{ $l.ProxyMaxTempFileSize }};
{{ end }}
proxy_buffering {{ if $l.ProxyBuffering }}on{{ else }}off{{ end }};
{{ if $l.ProxyBuffers }}
proxy_buffers {{ $l.ProxyBuffers }};
{{ end }}
{{ if $l.ProxyBufferSize }}
proxy_buffer_size {{ $l.ProxyBufferSize }};
{{ end }}
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto {{ with $s.TLSRedirect }}{{ .BasedOn }}{{ else }}$scheme{{ end }};
{{ range $h := $l.ProxySetHeaders }}
proxy_set_header {{ $h.Name }} "{{ $h.Value }}";
{{ end }}
{{ range $h := $l.ProxyHideHeaders }}
proxy_hide_header {{ $h }};
{{ end }}
{{ range $h := $l.ProxyPassHeaders }}
proxy_pass_header {{ $h }};
{{ end }}
{{ with $l.ProxyIgnoreHeaders }}
proxy_ignore_headers {{ $l.ProxyIgnoreHeaders }};
{{ end }}
{{ range $h := $l.AddHeaders }}
add_header {{ $h.Name }} "{{ $h.Value }}" {{ if $h.Always }}always{{ end }};
{{ end }}
proxy_pass {{ $l.ProxyPass }}{{ $l.ProxyPassRewrite }};
proxy_next_upstream {{ $l.ProxyNextUpstream }};
proxy_next_upstream_timeout {{ $l.ProxyNextUpstreamTimeout }};
proxy_next_upstream_tries {{ $l.ProxyNextUpstreamTries }};
proxy_pass_request_headers {{ if $l.ProxyPassRequestHeaders }}on{{ else }}off{{ end }};
{{ end }}
}
{{ end }}
}
proxy-body-size: "0" proxy-read-timeout: "600" proxy-send-timeout: "600" server-tokens: "false" worker-connections: "30000" kind: ConfigMap metadata: annotations: meta.helm.sh/release-name: nginx-controller meta.helm.sh/release-namespace: default creationTimestamp: "2023-07-10T09:41:17Z" labels: app.kubernetes.io/managed-by: Helm name: nginx-vs-config namespace: default resourceVersion: "31487204" uid: 38d3cbb5-bcb4-4c88-b97b-b33d48bb2722 `
below i have given for args on nginx-ingress controller daemonset
`apiVersion: apps/v1 kind: DaemonSet metadata: annotations: deprecated.daemonset.template.generation: "19" meta.helm.sh/release-name: nginx-controller meta.helm.sh/release-namespace: default creationTimestamp: "2023-07-10T09:41:17Z" generation: 19 labels: app.kubernetes.io/managed-by: Helm name: nginx-ingress namespace: default resourceVersion: "31592131" uid: 05a2a428-0a13-4052-b2ec-3883f94e485f spec: revisionHistoryLimit: 10 selector: matchLabels: app: nginx-ingress template: metadata: annotations: nginx.ingress.kubernetes.io/proxy-ssl-protocols: TLSv1.2 nginx.ingress.kubernetes.io/service-upstream: "true" nginx.org/server-tokens: "false" creationTimestamp: null labels: app: nginx-ingress spec: containers:
are location-snippets being added in ingress yaml? I see mentions of nginx.virtualserver.tmpl: in the code above. Also is controller.enableSnippets
set correctly in helm command?
something like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cafe-ingress-with-snippets
annotations:
nginx.org/location-snippets: |
deny 192.168.1.1;
allow 192.168.1.0/24;
allow 10.1.1.0/16;
allow 2001:0db8::/32;
deny all;
}
The key is that enable-snippets is set on the deployment. It is not uncommon for snippets to be defined but snippets not being enabled.
@brianehlert we have given - enable-snippets=true its not working, is there any way we can enable that, and we are not using helm chart to install, we just doing the deployment of daemonset.
@vadlakiran can you please post kubectl describe output of your daemonset (with snippet flag enabled) and ingress (with required snippets in it)? please use github code-blocks to enclose both.
below are the ingress entries `Name: webapp-ingress-ipv4 Namespace: default Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) TLS: apps-tls-secret terminates webapp.test.example.com Rules: Host Path Backends
webapp.test.example.com / web-app-svc:80 (10.233.108.216:80,10.233.84.40:80) /ws web-socket-none-service:80 (10.233.84.151:80) /socket.io node-ssh-svc:80 (10.233.86.57:80) Annotations: ingress.kubernetes.io/ssl-redirect: false meta.helm.sh/release-name: web-app meta.helm.sh/release-namespace: default nginx.ingress.kubernetes.io/configuration-snippet: proxy_set_header Upgrade "websocket"; proxy_set_header Connection "Upgrade"; nginx.ingress.kubernetes.io/enable-cors: false nginx.ingress.kubernetes.io/force-ssl-redirect: false nginx.ingress.kubernetes.io/rewrite-target: /$1 nginx.ingress.kubernetes.io/use-regex: true nginx.org/lb-method: round_robin nginx.org/location-snippets: allow 172.24.40.239/24; deny all; nginx.org/websocket-services: node-ssh-svc Events: Type Reason Age From Message
`
below is the daemonset describe which i have enabled snippet
Name: nginx-ingress
Selector: app=nginx-ingress
Node-Selector:
Normal SuccessfulDelete 23m daemonset-controller Deleted pod: nginx-ingress-h67ds Normal SuccessfulCreate 23m daemonset-controller Created pod: nginx-ingress-gn7cp Normal SuccessfulDelete 23m daemonset-controller Deleted pod: nginx-ingress-nj9zg Normal SuccessfulCreate 23m daemonset-controller Created pod: nginx-ingress-xz7r7 Normal SuccessfulDelete 23m daemonset-controller Deleted pod: nginx-ingress-btdxl Normal SuccessfulCreate 22m daemonset-controller Created pod: nginx-ingress-57vkv Normal SuccessfulDelete 32s daemonset-controller Deleted pod: nginx-ingress-xz7r7 Normal SuccessfulCreate 27s daemonset-controller Created pod: nginx-ingress-85sxn Normal SuccessfulDelete 24s daemonset-controller Deleted pod: nginx-ingress-gn7cp Normal SuccessfulCreate 21s daemonset-controller Created pod: nginx-ingress-k2w7j Normal SuccessfulDelete 19s daemonset-controller Deleted pod: nginx-ingress-57vkv Normal SuccessfulCreate 16s daemonset-controller Created pod: nginx-ingress-glh42 draj@master3:~$
You are saying "it is not working" and I have to apologize but I am going to ask: what is not working?
Do any of your Ingress objects have errors? Is the snippet not being written to the nginx.conf in the ingress controller pod?
I am not following how you are determining that it is not working.
executing nginx -T
in the pod will quickly show us the resolved configuration.
@vadlakiran any update on this?
@brianehlert, I am trying to whitelist the specific IP in ingress, I have tried with snippet but did not worked Is there any solution to whitelist the ip
I take it that what @vepatel suggested here is not working? Are there errors? Did you enable snippets with the deployment option?
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: cafe-ingress-with-snippets annotations: nginx.org/location-snippets: | deny 192.168.1.1; allow 192.168.1.0/24; allow 10.1.1.0/16; allow 2001:0db8::/32; deny all; }
I am guessing this is not an option because you are trying to exclusively use the Ingress resource: https://docs.nginx.com/nginx-ingress-controller/configuration/policy-resource/#accesscontrol
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.
This issue was closed because it has been stalled for 10 days with no activity.
Describe the bug Hi team,
we have deployed the daemonset a nginx-ingress controller and i want to witelisting the sepcific ip address how we can try and i have tried with below annotations but did not work.
nginx.ingress.kubernetes.io/whitelist-source-range: "ip address"
To Reproduce Steps to reproduce the behavior: we can reproduce it if needed
Expected behavior A clear and concise description of what you expected to happen.
Your environment : baremetal kubernetes cluster
Additional context Add any other context about the problem here. Any log files you want to share.