github-actions[bot]
commented
3 years ago Hi @llyons thanks for reporting!
Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:
Cheers!
Closed llyons closed 3 years ago
github-actions[bot]
commented
3 years ago Hi @llyons thanks for reporting!
Be sure to check out the docs while you wait for a human to take a look at this :slightly_smiling_face:
Cheers!
llyons
commented
3 years ago A couple of additional notes. I was trying to check to see if the config for the ingress was generated and I am not able to see anything in /etc/nginx. There is no folder named /etc/nginx
running kubectl exec
returns
`
worker_processes auto; daemon off;
error_log stderr notice; pid /var/lib/nginx/nginx.pid;
events { worker_connections 1024; }
http { include /etc/nginx/mime.types; default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /dev/stdout main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65s;
keepalive_requests 100;
#gzip on;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 256;
variables_hash_bucket_size 256;
variables_hash_max_size 1024;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
map $http_upgrade $vs_connection_header {
default upgrade;
'' $default_connection_header;
}
server {
# required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
set $default_connection_header "";
set $resource_type "";
set $resource_name "";
set $resource_namespace "";
set $service "";
listen 80 default_server;
listen 443 ssl default_server;
ssl_certificate /etc/nginx/secrets/default;
ssl_certificate_key /etc/nginx/secrets/default;
server_name _;
server_tokens "on";
location / {
return 404;
}
}
# stub_status
server {
listen 8080;
allow 127.0.0.1;
deny all;
location /stub_status {
stub_status;
}
}
include /etc/nginx/config-version.conf;
include /etc/nginx/conf.d/*.conf;
server {
listen unix:/var/lib/nginx/nginx-502-server.sock;
access_log off;
return 502;
}
server {
listen unix:/var/lib/nginx/nginx-418-server.sock;
access_log off;
return 418;
}}
stream { log_format stream-main '$remote_addr [$time_local] ' '$protocol $status $bytes_sent $bytes_received ' '$session_time "$ssl_preread_server_name"';
access_log /dev/stdout stream-main;
include /etc/nginx/stream-conf.d/*.conf;}`
jasonwilliams14
commented
3 years ago Hi @llyons Can you tell me about the coffee/tea applications that you deployed? What namespace did you deploy them into? For the NGINX Ingress controller, what namespace did you deploy NGINX Ingress into?
could you provide the output of:
kubectl get po,svc -AWe will take a look at your results. Thank you.
llyons
commented
3 years ago Thanks Jason, Much appreciated.
NGINX is in the nginx-ingress namespace kubectl get po,svc -A
I suspect there is something small. I am not getting anything generated in the /etc/nginx/conf.d folder. There is actually no /etc/nginx folder
NAMESPACE NAME READY STATUS RESTARTS AGE default pod/aks-helloworld-one-75c9fcd88-sx48t 1/1 Running 0 8h default pod/aks-helloworld-two-6dcdb7f788-w7nx2 1/1 Running 0 8h default pod/coffee-6d68b8ff4f-8jdkf 1/1 Running 0 7h42m default pod/coffee-6d68b8ff4f-g5rpk 1/1 Running 0 7h42m default pod/tea-766cdd6757-gdj57 1/1 Running 0 7h42m default pod/tea-766cdd6757-kswcj 1/1 Running 0 7h42m default pod/tea-766cdd6757-smqjx 1/1 Running 0 7h42m kube-system pod/coredns-558bd4d5db-2zwrp 1/1 Running 1 20d kube-system pod/coredns-558bd4d5db-tsbb8 1/1 Running 1 20d kube-system pod/etcd-iabrl-k8inp01.olh.local 1/1 Running 1 20d kube-system pod/kube-apiserver-iabrl-k8inp01.olh.local 1/1 Running 1 20d kube-system pod/kube-controller-manager-iabrl-k8inp01.olh.local 1/1 Running 1 20d kube-system pod/kube-flannel-ds-4npfv 1/1 Running 1 20d kube-system pod/kube-flannel-ds-fxqlw 1/1 Running 1 20d kube-system pod/kube-flannel-ds-qpvpn 1/1 Running 3 20d kube-system pod/kube-flannel-ds-windows-amd64-fd6pk 1/1 Running 3 12h kube-system pod/kube-flannel-ds-windows-amd64-kxkvz 1/1 Running 6 9d kube-system pod/kube-flannel-ds-windows-amd64-ppxd7 1/1 Running 9 9d kube-system pod/kube-proxy-9q8f7 1/1 Running 1 20d kube-system pod/kube-proxy-dtv2g 1/1 Running 1 20d kube-system pod/kube-proxy-windows-d7wmq 1/1 Running 1 12h kube-system pod/kube-proxy-windows-f5w95 1/1 Running 6 7d4h kube-system pod/kube-proxy-windows-xx59w 1/1 Running 6 7d4h kube-system pod/kube-proxy-xfdk5 1/1 Running 1 20d kube-system pod/kube-scheduler-iabrl-k8inp01.olh.local 1/1 Running 1 20d metallb-system pod/controller-6b78bff7d9-pqjsq 1/1 Running 0 11h metallb-system pod/speaker-c4k5v 1/1 Running 0 11h metallb-system pod/speaker-gdzpv 1/1 Running 0 11h metallb-system pod/speaker-s85mj 1/1 Running 0 11h nginx-ingress pod/nginx-ingress-55f77659df-nrsqp 1/1 Running 0 2d9h nginx-ingress pod/nginx-ingress-5bw4f 1/1 Running 0 9d nginx-ingress pod/nginx-ingress-m6btc 1/1 Running 0 9d
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/aks-helloworld-one ClusterIP 10.96.236.14
``
llyons
commented
3 years ago Some additional results from some of the troubleshooting commands.
kubectl describe pol webapp-policy Error from server (NotFound): policies.k8s.nginx.org "webapp-policy" not found
kubectl describe vsr coffee Error from server (NotFound): virtualserverroutes.k8s.nginx.org "coffee" not found
kubectl describe vs cafe Error from server (NotFound): virtualservers.k8s.nginx.org "cafe" not found
kubectl describe ing cafe-ingress
Name: cafe-ingress Namespace: default Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) TLS: cafe-secret terminates cafe.example.com Rules: Host Path Backends
cafe.example.com
/tea tea-svc:80 (192.168.1.26:8080,192.168.2.33:8080,192.168.2.34:8080)
/coffee coffee-svc:80 (192.168.2.31:8080,192.168.2.32:8080)
Annotations:
llyons
commented
3 years ago I have tried to set the cafe.example.com host to the external IP of the ingress controller served up by metallb and I have tried to set cafe.example.com to the ip of one of the linux nodes where ingress controllers are running. In both cases I get this.
$ curl http://cafe.example.com/
$ curl http://cafe.example.com/tea
$ curl http://cafe.example.com/coffee
$ curl 10.109.253.105 Server address: 192.168.2.32:8080 Server name: coffee-6d68b8ff4f-8jdkf Date: 18/Jun/2021:13:58:28 +0000 URI: / Request ID: 7914329ab48caa9fd5ffc35f2234c894
$ curl 192.168.2.32:8080 Server address: 192.168.2.32:8080 Server name: coffee-6d68b8ff4f-8jdkf Date: 18/Jun/2021:13:58:57 +0000 URI: / Request ID: 589d81df45de33ddff1df34b2f572899
jasonwilliams14
commented
3 years ago Hi @llyons
Can you verify if your ingress has a class set?
Here is an example from my lab:
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default cafe-ingress nginx cafe.example.com 172.18.0.2 80, 443 5m4sit is hard to tell from your output if a Class was set for the Ingress resource.
If it is not set, you can set a class in the Ingress manifest like so:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
spec:
ingressClassName: nginx # change this to your Ingressclass you have created in your cluster
tls:
- hosts:
- cafe.example.com
secretName: cafe-secret
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80Try that and let us know.
llyons
commented
3 years ago I did add the ingressClassname or uncommented it in the example and it seems to work. I was able set the cafe.example.com to the IP of the ingress (that was served up by metallb) and go to http://cafe.example.com/tea and http://cafe.example.com/coffee and get this.
Server address: 192.168.2.31:8080 Server name: coffee-6d68b8ff4f-g5rpk Date: 19/Jun/2021:00:52:53 +0000 URI: /coffee Request ID: 4767e173143b06a517639133b8f34a6b
does that mean its working from your perspective?
I still dont see files generated in /etc/nginx/conf.d
jasonwilliams14
commented
3 years ago Hello @llyons That looks correct. That is our sample coffee/tea app that responds back with some info on the app/pod, also verifying that the Ingress controller is accepting and routing traffic correctly to your applications in the cluster.
I still dont see files generated in /etc/nginx/conf.d
What you can do to check the config, logon to the NGINX ingress pod and execute:
nginx -T
You see at the top that main file being nginx.conf and any additional includes will also be listed.
Let us know if you have any other questions. Cheers!
llyons
commented
3 years ago when you say logon to the NGINX ingress pod, how is that done? I have never done that. Are you meaning create a shell into the ingress container and run that command?
Thanks
llyons
commented
3 years ago i seem to be missing something that might help understand how the routing should be setup to make these work.
I have another helloworld example that I pulled over from an AKS deploy. it is tied to host helloworld.onlifehealth.com which points to the IP of the ingress controller like the cafe.example.com prior.
`apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
ingressClassName: nginx
rules:
- host: helloworld.onlifehealth.com
http:
paths:
- path: /hello-world-one
pathType: Prefix
backend:
service:
name: aks-helloworld-one
port:
number: 80
- path: /hello-world-two
pathType: Prefix
backend:
service:
name: aks-helloworld-two
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: aks-helloworld-one
port:
number: 80`and the two deployments /services
`apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-helloworld-one
spec:
replicas: 1
selector:
matchLabels:
app: aks-helloworld-one
template:
metadata:
labels:
app: aks-helloworld-one
spec:
containers:
- name: aks-helloworld-one
image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
ports:
- containerPort: 80
env:
- name: TITLE
value: "Welcome to Azure Kubernetes Service (AKS)"
nodeSelector:
kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
name: aks-helloworld-one
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: aks-helloworld-one``apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-helloworld-two
spec:
replicas: 1
selector:
matchLabels:
app: aks-helloworld-two
template:
metadata:
labels:
app: aks-helloworld-two
spec:
containers:
- name: aks-helloworld-two
image: mcr.microsoft.com/azuredocs/aks-helloworld:v1
ports:
- containerPort: 80
env:
- name: TITLE
value: "AKS Ingress Demo"
nodeSelector:
kubernetes.io/os: linux
---
apiVersion: v1
kind: Service
metadata:
name: aks-helloworld-two
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: aks-helloworld-two`
with the above example I can see a response on http://helloworld.onlifehealth.com but i am seeing
Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
with these URLs http://helloworld.onlifehealth.com/hello-world-one and http://helloworld.onlifehealth.com/hello-world-two
jasonwilliams14
commented
3 years ago Hi @llyons Looking at your settings:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1Those specific annotations are for a different Ingress controller project and will not work with this particular NGINX Ingress controller. Those above are for the Community Edition of NGINX Ingress Controller maintained by the community. This particular Ingress Controller on this repo, is maintained by NGINX inc.
You will want to look at these annotations to use with your Ingress:
You will need to convert your annotations to the ones supported by this Ingress product.
Keep in mind that for regex, that is supported in our CRD using VirtualServer and VirtualServerRoute, which provides more advanced functionality and routing capabilities:
Here is a quick write I did of converting your Ingress resource to a VirtualServer resource:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: aks-helloworld
spec:
host: helloworld.onlifehealth.com
tls:
secret: helloworld-secret
upstreams:
- name: aks-helloworld-one
service: aks-helloworld-one-svc
port: 80
- name: aks-helloworld-two
service: aks-helloworld-two-svc
port: 80
routes:
- path: /
action:
pass: aks-helloworld-one
- path: /hello-world-two
action:
pass: aks-helloworld-twoNOTE: I added svc to the end of the service just for clarity.
In the above, you can remove the tls/secret spec. That does TLS termination. I put it there for an example.
Any request that comes into helloworld.onelifehealth.com routes to aks-helloworld-one app.
Requests coming into helloworld.onelifehealth.com/hello-world-two get routed to aks-helloworld-two app.
Pardon any typos. Let us know if that is helpful.
llyons
commented
3 years ago thanks alot. i appreciate that. I was able to get a few other of our apps working behind the ingress.
I will spend time this afternoon trying to digest the info you just provided like the virtualserver kind and the different annotations
again thanks
jasonwilliams14
commented
3 years ago @llyons You are very welcome. Happy to help out anytime. Cheers
llyons
commented
3 years ago one more question..
I change the hello world to use the virtual server and change the service definitions to have -svc as you did.
I am able to curl the IP of the pod and the IP of the service
NAMESPACE NAME READY STATUS RESTARTS AGE IP
default pod/aks-helloworld-one-75c9fcd88-bv8mk 1/1 Running 0 5m20s 192.168.1.28
default pod/aks-helloworld-two-6dcdb7f788-qgxh2 1/1 Running 0 5m15s 192.168.1.29
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/aks-helloworld-one-svc ClusterIP 10.110.232.106
so curl 192.168.1.28 or 10.110.232.106 returns data from app hello-world-one so curl 192.168.1.29 or 10.103.104.138 returns data from app hello-world-two
however i have my ingress controller set to IP 10.244.0.30
nginx-ingress service/nginx-ingress LoadBalancer 10.111.32.192 10.244.0.30 80:32132/TCP,443:32544/TCP 18d app=nginx-ingress
and have helloworld.onlifehealth.com set to this ip in my hosts file.
http://helloworld.onlifehealth.com works http://helloword.onlifehealth.com/hello-world-one or two give me this error.
should the virtualserver have an IP? I am guessing I cant get to the new virtualserver through the ingress controller IP.
kubectl get vs
NAME STATE HOST IP PORTS AGE aks-helloworld Valid helloworld.onlifehealth.com 113m
also
kubectl describe vs aks-helloworld
Name: aks-helloworld
Namespace: default
Labels:
llyons
commented
3 years ago It looks like the path / works with the above virtualserver configuration but the other paths /hello-world-one and /hello-world-two do not.
I did change the path / to point to the other service as well just to confirm the two pods/services are working. For some reason the /something paths are not routhing. No errors in the logs that I can see.
im guessing this is a simple issue that I have set wrong.
kubectl describe vs aks-helloworld
Name: aks-helloworld
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8s.nginx.org/v1
Kind: VirtualServer
Metadata:
Creation Timestamp: 2021-06-21T18:58:08Z
Generation: 9
Managed Fields:
API Version: k8s.nginx.org/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:host:
f:routes:
f:upstreams:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-06-21T18:58:08Z
API Version: k8s.nginx.org/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:externalEndpoints:
f:message:
f:reason:
f:state:
Manager: nginx-ingress
Operation: Update
Time: 2021-06-21T18:58:08Z
Resource Version: 3658255
UID: 00cc13d8-6869-4c29-a450-2dd2b865509f
Spec:
Host: helloworld.onlifehealth.com
Routes:
Action:
Pass: aks-helloworld-one
Path: /
Action:
Pass: aks-helloworld-one
Path: /hello-world-one
Action:
Pass: aks-helloworld-two
Path: /hello-world-two
Upstreams:
Name: aks-helloworld-one
Port: 80
Service: aks-helloworld-one-svc
Name: aks-helloworld-two
Port: 80
Service: aks-helloworld-two-svc
Status:
External Endpoints:
Ip:
Ports:
Message: Configuration for default/aks-helloworld was added or updated
Reason: AddedOrUpdated
State: Valid
Events: <none>@llyons Can you provide us with the output of your nginx.conf? Here is a doc on how to do so. When you have that, and paste it here, we can review. Thank you.
https://docs.nginx.com/nginx-ingress-controller/troubleshooting/#checking-the-generated-config
llyons
commented
3 years ago Hi Jason, Thanks. We dont seem to get any auto generated nginx.conf in the /etc/nginx/conf.d folder so I am a bit confused there.
Here the the complete nginx.conf from the command kubectl exec
sorry for how busy this is.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
worker_processes auto;
daemon off;
error_log stderr notice;
pid /var/lib/nginx/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /dev/stdout main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65s;
keepalive_requests 100;
#gzip on;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 256;
variables_hash_bucket_size 256;
variables_hash_max_size 1024;
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
map $http_upgrade $vs_connection_header {
default upgrade;
'' $default_connection_header;
}
server {
# required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
set $default_connection_header "";
set $resource_type "";
set $resource_name "";
set $resource_namespace "";
set $service "";
listen 80 default_server;
listen 443 ssl default_server;
ssl_certificate /etc/nginx/secrets/default;
ssl_certificate_key /etc/nginx/secrets/default;
server_name _;
server_tokens "on";
location / {
return 404;
}
}
# stub_status
server {
listen 8080;
allow 127.0.0.1;
deny all;
location /stub_status {
stub_status;
}
}
include /etc/nginx/config-version.conf;
include /etc/nginx/conf.d/*.conf;
server {
listen unix:/var/lib/nginx/nginx-502-server.sock;
access_log off;
return 502;
}
server {
listen unix:/var/lib/nginx/nginx-418-server.sock;
access_log off;
return 418;
}
}
stream {
log_format stream-main '$remote_addr [$time_local] '
'$protocol $status $bytes_sent $bytes_received '
'$session_time "$ssl_preread_server_name"';
access_log /dev/stdout stream-main;
include /etc/nginx/stream-conf.d/*.conf;
}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/wasm wasm;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/config-version.conf:
server {
listen unix:/var/lib/nginx/nginx-config-version.sock;
access_log off;
location /configVersion {
return 200 134;
}
}
map $http_x_expected_config_version $config_version_mismatch {
"134" "";
default "mismatch";
}
# configuration file /etc/nginx/conf.d/default-cafe-ingress.conf:
# configuration for default/cafe-ingress
upstream default-cafe-ingress-cafe.example.com-coffee-svc-80 {
zone default-cafe-ingress-cafe.example.com-coffee-svc-80 256k;
random two least_conn;
server 192.168.2.31:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.32:8080 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-cafe-ingress-cafe.example.com-tea-svc-80 {
zone default-cafe-ingress-cafe.example.com-tea-svc-80 256k;
random two least_conn;
server 192.168.1.26:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.33:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.34:8080 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/default-cafe-secret;
ssl_certificate_key /etc/nginx/secrets/default-cafe-secret;
server_tokens on;
server_name cafe.example.com;
set $resource_type "ingress";
set $resource_name "cafe-ingress";
set $resource_namespace "default";
if ($scheme = http) {
return 301 https://$host:443$request_uri;
}
location /tea {
set $service "tea-svc";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-cafe-ingress-cafe.example.com-tea-svc-80;
}
location /coffee {
set $service "coffee-svc";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-cafe-ingress-cafe.example.com-coffee-svc-80;
}
}
# configuration file /etc/nginx/conf.d/default-clientportal-ingress.conf:
# configuration for default/clientportal-ingress
upstream default-clientportal-ingress-aks-clientportal.onlifehealth.com-clientportal-80 {
zone default-clientportal-ingress-aks-clientportal.onlifehealth.com-clientportal-80 256k;
random two least_conn;
server 192.168.4.21:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name aks-clientportal.onlifehealth.com;
set $resource_type "ingress";
set $resource_name "clientportal-ingress";
set $resource_namespace "default";
location / {
set $service "clientportal";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-clientportal-ingress-aks-clientportal.onlifehealth.com-clientportal-80;
}
}
# configuration file /etc/nginx/conf.d/default-ecforecasting-ingress-ci.conf:
# configuration for default/ecforecasting-ingress-ci
upstream default-ecforecasting-ingress-ci-ci-ecforecasting.onlifehealth.com-ecforecasting-ci-80 {
zone default-ecforecasting-ingress-ci-ci-ecforecasting.onlifehealth.com-ecforecasting-ci-80 256k;
random two least_conn;
server 192.168.1.27:8866 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name ci-ecforecasting.onlifehealth.com;
set $resource_type "ingress";
set $resource_name "ecforecasting-ingress-ci";
set $resource_namespace "default";
location / {
set $service "ecforecasting-ci";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-ecforecasting-ingress-ci-ci-ecforecasting.onlifehealth.com-ecforecasting-ci-80;
}
}
# configuration file /etc/nginx/conf.d/default-tsdistributor-ingress-tsdev.conf:
# configuration for default/tsdistributor-ingress-tsdev
upstream default-tsdistributor-ingress-tsdev-tsdev-apitesting.onlifehealth.com-tsdistributor-tsdev-80 {
zone default-tsdistributor-ingress-tsdev-tsdev-apitesting.onlifehealth.com-tsdistributor-tsdev-80 256k;
random two least_conn;
server 192.168.4.24:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name tsdev-apitesting.onlifehealth.com;
set $resource_type "ingress";
set $resource_name "tsdistributor-ingress-tsdev";
set $resource_namespace "default";
location / {
set $service "tsdistributor-tsdev";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-tsdistributor-ingress-tsdev-tsdev-apitesting.onlifehealth.com-tsdistributor-tsdev-80;
}
}
# configuration file /etc/nginx/conf.d/default-validicservice-ingress-ci.conf:
# configuration for default/validicservice-ingress-ci
upstream default-validicservice-ingress-ci-ci-validicservice.onlifehealth.com-validicservice-ci-80 {
zone default-validicservice-ingress-ci-ci-validicservice.onlifehealth.com-validicservice-ci-80 256k;
random two least_conn;
server 192.168.5.24:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name ci-validicservice.onlifehealth.com;
set $resource_type "ingress";
set $resource_name "validicservice-ingress-ci";
set $resource_namespace "default";
location / {
set $service "validicservice-ci";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-validicservice-ingress-ci-ci-validicservice.onlifehealth.com-validicservice-ci-80;
}
}
# configuration file /etc/nginx/conf.d/default-validicservice-ingress-qa.conf:
# configuration for default/validicservice-ingress-qa
upstream default-validicservice-ingress-qa-qa-validicservice.onlifehealth.com-validicservice-qa-80 {
zone default-validicservice-ingress-qa-qa-validicservice.onlifehealth.com-validicservice-qa-80 256k;
random two least_conn;
server 192.168.3.18:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name qa-validicservice.onlifehealth.com;
set $resource_type "ingress";
set $resource_name "validicservice-ingress-qa";
set $resource_namespace "default";
location / {
set $service "validicservice-qa";
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-validicservice-ingress-qa-qa-validicservice.onlifehealth.com-validicservice-qa-80;
}
}
# configuration file /etc/nginx/conf.d/vs_default_aks-helloworld.conf:
upstream vs_default_aks-helloworld_aks-helloworld-one {
zone vs_default_aks-helloworld_aks-helloworld-one 256k;
random two least_conn;
server 192.168.1.28:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream vs_default_aks-helloworld_aks-helloworld-two {
zone vs_default_aks-helloworld_aks-helloworld-two 256k;
random two least_conn;
server 192.168.1.29:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_name helloworld.onlifehealth.com;
set $resource_type "virtualserver";
set $resource_name "aks-helloworld";
set $resource_namespace "default";
server_tokens "on";
location / {
set $service "aks-helloworld-one-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_aks-helloworld_aks-helloworld-one;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
location /hello-world-one {
set $service "aks-helloworld-one-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_aks-helloworld_aks-helloworld-one;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
location /hello-world-two {
set $service "aks-helloworld-two-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_aks-helloworld_aks-helloworld-two;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
}
Can you provide the output of:
kubectl get po,svc -Aalso, are you curling the hostname or the IP? Where are you curling/accessing the app from? Outside the cluster? inside the cluster? Have you modified the coffee/tea sample at all?
llyons
commented
3 years ago Hi Jason,
I am able to curl the pods and the svc using the IP of the pod and svc from the master.
I was able to change the VirtualServer definition to have / go to hello-world-one and hello-world-two and both work.
path: / action: pass: aks-helloworld-one
and
both bring up the different pods/svc under http://helloworld.onlifehealth.com
However they dont work when behind the path /hello-world-one or /hello-world-two
http://helloworld.onlifehealth.com/hello-world-one (or two) do not work and give the error
Not Found
The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.Have not modified the coffee/tea example yet although i really am hoping to get to the point where we can use virtualservers and virtualserverroutes since this lines up a little more with our F5 approach used in the past.
kubectl get po,svc -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/aks-helloworld-one-75c9fcd88-bv8mk 1/1 Running 0 45h
default pod/aks-helloworld-two-6dcdb7f788-qgxh2 1/1 Running 0 45h
default pod/clientportal 1/1 Running 0 2d
default pod/coffee-6d68b8ff4f-8jdkf 1/1 Running 0 5d21h
default pod/coffee-6d68b8ff4f-g5rpk 1/1 Running 0 5d21h
default pod/ecforecasting-ci 1/1 Running 1 46h
default pod/tea-766cdd6757-gdj57 1/1 Running 0 5d21h
default pod/tea-766cdd6757-kswcj 1/1 Running 0 5d21h
default pod/tea-766cdd6757-smqjx 1/1 Running 0 5d21h
default pod/tsdistributor-tsdev 1/1 Running 0 109m
default pod/tsengagementhub-tsdev 1/1 Running 0 45m
default pod/validicservice-ci 1/1 Running 5 42h
default pod/validicservice-qa 1/1 Running 11 23h
kube-system pod/coredns-558bd4d5db-2zwrp 1/1 Running 1 26d
kube-system pod/coredns-558bd4d5db-tsbb8 1/1 Running 1 26d
kube-system pod/etcd-iabrl-k8inp01.olh.local 1/1 Running 1 26d
kube-system pod/kube-apiserver-iabrl-k8inp01.olh.local 1/1 Running 1 26d
kube-system pod/kube-controller-manager-iabrl-k8inp01.olh.local 1/1 Running 1 26d
kube-system pod/kube-flannel-ds-4npfv 1/1 Running 1 26d
kube-system pod/kube-flannel-ds-fxqlw 1/1 Running 1 26d
kube-system pod/kube-flannel-ds-qpvpn 1/1 Running 3 26d
kube-system pod/kube-flannel-ds-windows-amd64-fd6pk 1/1 Running 3 6d2h
kube-system pod/kube-flannel-ds-windows-amd64-kxkvz 1/1 Running 6 15d
kube-system pod/kube-flannel-ds-windows-amd64-ppxd7 1/1 Running 9 14d
kube-system pod/kube-proxy-9q8f7 1/1 Running 1 26d
kube-system pod/kube-proxy-dtv2g 1/1 Running 1 26d
kube-system pod/kube-proxy-windows-d7wmq 1/1 Running 1 6d2h
kube-system pod/kube-proxy-windows-f5w95 1/1 Running 6 12d
kube-system pod/kube-proxy-windows-xx59w 1/1 Running 6 12d
kube-system pod/kube-proxy-xfdk5 1/1 Running 1 26d
kube-system pod/kube-scheduler-iabrl-k8inp01.olh.local 1/1 Running 1 26d
metallb-system pod/controller-6b78bff7d9-pqjsq 1/1 Running 0 6d1h
metallb-system pod/speaker-c4k5v 1/1 Running 0 6d1h
metallb-system pod/speaker-gdzpv 1/1 Running 0 6d1h
metallb-system pod/speaker-s85mj 1/1 Running 0 6d1h
nginx-ingress pod/nginx-ingress-55f77659df-nrsqp 1/1 Running 0 7d23h
nginx-ingress pod/nginx-ingress-5bw4f 1/1 Running 0 14d
nginx-ingress pod/nginx-ingress-m6btc 1/1 Running 0 14d
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/aks-helloworld-one-svc ClusterIP 10.110.232.106 <none> 80/TCP 45h
default service/aks-helloworld-two-svc ClusterIP 10.103.104.138 <none> 80/TCP 45h
default service/clientportal ClusterIP 10.108.222.13 <none> 80/TCP 2d
default service/coffee-svc ClusterIP 10.109.253.105 <none> 80/TCP 5d21h
default service/ecforecasting-ci ClusterIP 10.111.26.143 <none> 80/TCP 46h
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 26d
default service/tea-svc ClusterIP 10.102.6.16 <none> 80/TCP 5d21h
default service/tsdistributor-tsdev ClusterIP 10.107.145.197 <none> 80/TCP 109m
default service/tsengagementhub-tsdev ClusterIP 10.106.233.245 <none> 80/TCP 45m
default service/validicservice-ci ClusterIP 10.105.238.162 <none> 80/TCP 42h
default service/validicservice-qa ClusterIP 10.98.59.54 <none> 80/TCP 23h
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 26d
nginx-ingress service/nginx-ingress LoadBalancer 10.111.32.192 10.244.0.30 80:32132/TCP,443:32544/TCP 20dSo I did convert the cafe.example.com from ingress to virtualserver.
NVM what I added below. it does work. Without the
tls: secret: cafe-secret
it was behaving the same way relative to the /tea /coffee paths but with that in, it works fine**
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: cafe-virtualserver
spec:
host: cafe.example.com
upstreams:
- name: tea
service: tea-svc
port: 80
- name: coffee
service: coffee-svc
port: 80
routes:
- path: /
action:
pass: tea
- path: /tea
action:
pass: tea
- path: /coffee
action:
pass: coffee
the behavior is the same... If I go to http://cafe.example.com i get.
Server address: 192.168.2.34:8080 Server name: tea-766cdd6757-gdj57 Date: 23/Jun/2021:19:52:51 +0000 URI: / Request ID: 49fab0269687e30eda4a976054f2b508
if I go to http://cafe.example.com/tea or /coffee it throws the 404 not found.
here is the unique nginx.conf
# configuration file /etc/nginx/conf.d/vs_default_cafe-virtualserver.conf:
upstream vs_default_cafe-virtualserver_tea {
zone vs_default_cafe-virtualserver_tea 256k;
random two least_conn;
server 192.168.1.26:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.33:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.34:8080 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream vs_default_cafe-virtualserver_coffee {
zone vs_default_cafe-virtualserver_coffee 256k;
random two least_conn;
server 192.168.2.31:8080 max_fails=1 fail_timeout=10s max_conns=0;
server 192.168.2.32:8080 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_name cafe.example.com;
set $resource_type "virtualserver";
set $resource_name "cafe-virtualserver";
set $resource_namespace "default";
server_tokens "on";
location / {
set $service "tea-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_cafe-virtualserver_tea;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
location /tea {
set $service "tea-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_cafe-virtualserver_tea;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
location /coffee {
set $service "coffee-svc";
set $default_connection_header close;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_buffering on;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_default_cafe-virtualserver_coffee;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
proxy_pass_request_headers on;
}
}
@llyons
Did you by chance delete your original Ingress resource?
Can you run:
kubectl get ing,vs -AAlso, can you do a:
kubectl describe vs <name_of_your_vs>Post the above output in here as well.
Can you also run a few curl commands with -v for the three paths?
/
/tea
/coffee
Your nginx config looks correct Can you post your NGINX deployment file? Your above output has three NGINX ingress pods running. Is that correct?
llyons
commented
3 years ago Hi Jason, the coffee tea conversion to virtualserver does work. The simple helloworld app doesnt.
I did remove the ingress for each after creating the virtual server.
kubectl get ing,vs -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default ingress.networking.k8s.io/clientportal-ingress nginx aks-clientportal.onlifehealth.com 80 3d2h
default ingress.networking.k8s.io/ecforecasting-ingress-ci nginx ci-ecforecasting.onlifehealth.com 80 3d
default ingress.networking.k8s.io/tsdistributor-ingress-tsdev nginx tsdev-apitesting.onlifehealth.com 80 28h
default ingress.networking.k8s.io/tsengagementhub-ingress-tsdev nginx tsdev-apitesting.onlifehealth.com 80 27h
default ingress.networking.k8s.io/validicservice-ingress-ci nginx ci-validicservice.onlifehealth.com 80 2d21h
default ingress.networking.k8s.io/validicservice-ingress-qa nginx qa-validicservice.onlifehealth.com 80 2d2h
NAMESPACE NAME STATE HOST IP PORTS AGE
default virtualserver.k8s.nginx.org/aks-helloworld Valid helloworld.example.com 3d
default virtualserver.k8s.nginx.org/cafe-virtualserver Valid cafe.example.com 23hkubectl get vs aks-helloworld
NAME STATE HOST IP PORTS AGE
aks-helloworld Valid helloworld.example.com 3dOn the curl -v
curl -v helloworld.onlifehealth.com
* Rebuilt URL to: helloworld.onlifehealth.com/
* Trying 10.244.0.30...
* TCP_NODELAY set
* Connected to helloworld.onlifehealth.com (10.244.0.30) port 80 (#0)
> GET / HTTP/1.1
> Host: helloworld.onlifehealth.com
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.21.0
< Date: Thu, 24 Jun 2021 19:08:10 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 629
< Connection: keep-alive
<
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<link rel="stylesheet" type="text/css" href="/static/default.css">
<title>Welcome to Azure Kubernetes Service (AKS)</title>
<script language="JavaScript">
function send(form){
}
</script>
</head>
<body>
<div id="container">
<form id="form" name="form" action="/"" method="post"><center>
<div id="logo">Welcome to Azure Kubernetes Service (AKS)</div>
<div id="space"></div>
<img src="/static/acs.png" als="acs logo">
<div id="form">
</div>
</div>
</body>
</html>* Connection #0 to host helloworld.onlifehealth.com left intact
curl -v helloworld.onlifehealth.com/hello-world-one
* Trying 10.244.0.30...
* TCP_NODELAY set
* Connected to helloworld.onlifehealth.com (10.244.0.30) port 80 (#0)
> GET /hello-world-one HTTP/1.1
> Host: helloworld.onlifehealth.com
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 404 NOT FOUND
< Server: nginx/1.21.0
< Date: Thu, 24 Jun 2021 19:09:01 GMT
< Content-Type: text/html
< Content-Length: 233
< Connection: keep-alive
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
* Connection #0 to host helloworld.onlifehealth.com left intact
Here are my deployment yaml files. I had used these during some work in AKS although we are deploying into an on prem k8 cluster.
hello-world-virtualserver.yml.txt hello-world-one.yml.txt hello-world-two.yml.txt
llyons
commented
3 years ago I do want to express my thanks for the help on this. Thanks! I know this was not a bug with the product.
That leads me to the next general question, what is the best way to ask these questions related to the nginx ingress controller that are not bugs,etc. I know there are some slack channels for the ingress-nginx product.
jasonwilliams14
commented
3 years ago @llyons I downloaded your above files to test and was getting the same results as you were. I believe it has something to do with the app itself. I did some digging on the application itself and found this link:
https://docs.microsoft.com/en-us/azure/aks/ingress-basic
I have not broken down the application itself (yet, but will because I am curious now), but it looks like that particular setup is looking for a regex setup based upon my review (based on the annotations I see in the Ingress resource). The example is using Community Edition Ingress which is different from this projects Ingress controller, so those settings will not apply to ours.
I wanted to test out a few things for you, so I downloaded a simple hello world app, just to make sure all the routing that was setup on your VS was correct. Here is what I did.
Essentially, all I did was swap out your particular image with a 'hello-world' response image to verify that KIC would route properly based on the request, to the proper backend :
/
/aks-hello-world-one
/aks-hello-world-twoI used your exisiting VS setup to test with the new image. Everything worked as expected when I would send curl requests.
Question: Is there something specific you need to test out with your given setup with that particular image? Do you have a particular use case that you are trying to solve with that example, or something else? If there is something specific you want or need from that example, let us know and we can dig in further on it. Would it be possible to test with a different image, purely for testing purposes to see what your results would be?
My testing for this: Here are my files that you can reference, and testing output if it is helpful:
Deployment and services for simple 'hello response'
apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-helloworld-one
spec:
replicas: 1
selector:
matchLabels:
app: aks-helloworld-one
template:
metadata:
labels:
app: aks-helloworld-one
spec:
containers:
- name: aks-helloworld-one
image: gcr.io/google-samples/hello-app:1.0
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: aks-helloworld-one-svc
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: aks-helloworld-one
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: aks-helloworld-two
spec:
replicas: 1
selector:
matchLabels:
app: aks-helloworld-two
template:
metadata:
labels:
app: aks-helloworld-two
spec:
containers:
- name: aks-helloworld-two
image: gcr.io/google-samples/hello-app:2.0
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: aks-helloworld-two-svc
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: aks-helloworld-twoAnd my NGINX VS setup:
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: aks-helloworld
spec:
host: helloworld.onlifehealth.com
upstreams:
- name: aks-helloworld-one
service: aks-helloworld-one-svc
port: 80
- name: aks-helloworld-two
service: aks-helloworld-two-svc
port: 80
routes:
- path: /
action:
pass: aks-helloworld-one
- path: /hello-world-one
action:
pass: aks-helloworld-one
- path: /hello-world-two
action:
pass: aks-helloworld-twoAnd my curl responses, for the specific paths:
╰─$ curl http://helloworld.onlifehealth.com/ 130 ↵
Hello, world!
Version: 1.0.0
Hostname: aks-helloworld-one-69654d6d67-jljg6
╰─$ curl http://helloworld.onlifehealth.com/hello-world-one
Hello, world!
Version: 1.0.0
Hostname: aks-helloworld-one-69654d6d67-jljg6
╰─$ curl http://helloworld.onlifehealth.com/hello-world-two
Hello, world!
Version: 2.0.0
Hostname: aks-helloworld-two-69bc765fd-dw2rmAlso, results of my po,svc output (NOTE: All in default namespace:
╰─$ k get po,svc
NAME READY STATUS RESTARTS AGE
pod/aks-helloworld-one-69654d6d67-jljg6 1/1 Running 0 12m
pod/aks-helloworld-two-69bc765fd-dw2rm 1/1 Running 0 12m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 11h
service/aks-helloworld-one-svc ClusterIP 10.43.198.104 <none> 80/TCP 12m
service/aks-helloworld-two-svc ClusterIP 10.43.58.21 <none> 80/TCP 8m36sHope this is helpful.
github-actions[bot]
commented
3 years ago This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.
github-actions[bot]
commented
3 years ago This issue was closed because it has been stalled for 7 days with no activity.
Describe the bug with kubernete 1.21.1 cluster (with 3 linux and 3 windows nodes) and nginx-ingress (kubernetes-ingress v1.11.3 ) we are constantly getting 404 not found
To Reproduce Steps to reproduce the behavior:
kubectl get svc -o wide -n nginx-ingress
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE cafe-ingress cafe.example.com 80, 443 39m
kubectl get pods -A showing only cafe related
default coffee-6d68b8ff4f-8jdkf 1/1 Running 0 31m default coffee-6d68b8ff4f-g5rpk 1/1 Running 0 31m default tea-766cdd6757-gdj57 1/1 Running 0 31m default tea-766cdd6757-kswcj 1/1 Running 0 31m default tea-766cdd6757-smqjx 1/1 Running 0 31m
doing curl --resolve cafe.example.com:80:10.244.0.30 -v http://cafe.example.com returns 404
Expected behavior expected the nginx-ingress to route to the cafe or tea deployments
Your environment
kubectl get ingress cafe-ingress
NAME CLASS HOSTS ADDRESS PORTS AGE cafe-ingress cafe.example.com 80, 443 43m
kubectl describe ingress cafe-ingress
Name: cafe-ingress Namespace: default Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) TLS: cafe-secret terminates cafe.example.com Rules: Host Path Backends
cafe.example.com /tea tea-svc:80 (192.168.1.26:8080,192.168.2.33:8080,192.168.2.34:8080) /coffee coffee-svc:80 (192.168.2.31:8080,192.168.2.32:8080) Annotations:
Events:
curl 192.168.1.26:8080
Server address: 192.168.1.26:8080 Server name: tea-766cdd6757-kswcj Date: 17/Jun/2021:19:26:10 +0000 URI: / Request ID: 60d611f651c50b9186804c420882fc8e
curl cafe.example.com
404 Not Found