Can not visit my service running on kubernetes cluster by ingress

[pangwawa]

Describe

I try to visit my service running on kubernetes cluster by ingress，but faill

tell me why please, thanks

To Reproduce cat faskapp-deployment.yaml `

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flaskapp-1
spec:
  selector:
    matchLabels:
      run: flaskapp-1
  replicas: 1
  template:
    metadata:
      labels:
        run: flaskapp-1
    spec:
      containers:
      - name: flaskapp-1
        image: jcdemo/flaskapp
        ports:
        - containerPort: 5000

`

cat faskapp-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: flaskapp-1
  labels:
    run: flaskapp-1
spec:
  ports:
  - port: 5000
    name: web
    targetPort: 5000 
  selector:
    run: flaskapp-1

cat faskapp-ingress.yaml

`

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: faskapp-ingress
  #kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: mydemo.com     
    http:
      paths:
      - path: /
        backend:
          serviceName: flaskapp-1  
          servicePort: 5000 

kubectl describe ingress

Name:             faskapp-ingress
Namespace:        default
Address:          192.168.161.121
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  mydemo.com  
              /   flaskapp-1:5000   10.244.97.193:5000)
Annotations:  <none>
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    2m14s (x2 over 2m25s)  nginx-ingress-controller  Scheduled for sync

My environment kubernetes version : v1.19.0 ingress-nginx version : v0.41.0

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.41.0
  Build:         f3a6b809bd4bb6608266b35cf5b8423bf107d7bc
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.4
-------------------------------------------------------------------------------

Expected behavior I want to visit my service through domain name ' mydemo.com' ，domain name resolution is ok

Additional context the envirnoment is fine, and I have tried to visit my service by "NodePort Type" , it work

[AfrouzMashayekhi]

can you curl mydemo.com curl -vvv -I http://mydemo.com @pangwawa

[Krishna-teja]

I am also having the same issue.

@afrouzMashaykhi When I do the curl command I am getting connection timed out errors

*   Trying 120.0.0.1:80...
* TCP_NODELAY set
* connect to 120.0.0.1 port 80 failed: Connection timed out
* Failed to connect to seq.local.com port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to seq.local.com port 80: Connection timed out

[jrareas]

I am having similar issue

jareas@Jareass-Mac-mini droplet %  kubectl describe ingress echo-ingress
Name:             echo-ingress
Namespace:        default
Address:          138.197.135.165
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  thats4free.com  
                  /   echo2:80   10.244.0.47:80)
Annotations:      kubernetes.io/ingress.class: nginx
                  nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                  nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    105s (x2 over 2m37s)  nginx-ingress-controller  Scheduled for sync

jareas@Jareass-Mac-mini droplet % curl -vvv -I http://thats4free.com 
*   Trying 138.197.135.165...
* TCP_NODELAY set
* Connection failed
* connect to 138.197.135.165 port 80 failed: Connection refused
* Failed to connect to thats4free.com port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to thats4free.com port 80: Connection refused

but, if I go to the machine and access the endpoint it works.

root@ubuntu-s-2vcpu-2gb-tor1-01:~# curl 10.244.0.47
<html><body><h1>It works!</h1></body></html>

I have a feeling is because the sync is not going through

[AfrouzMashayekhi]

I am also having the same issue.

@afrouzMashaykhi When I do the curl command I am getting connection timed out errors

*   Trying 120.0.0.1:80...
* TCP_NODELAY set
* connect to 120.0.0.1 port 80 failed: Connection timed out
* Failed to connect to seq.local.com port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to seq.local.com port 80: Connection timed out

@Krishna-teja you are trying localhost so you got timeout who did you setup your cluster in local(kind?) on cloud ..?

[AfrouzMashayekhi]

I am having similar issue

jareas@Jareass-Mac-mini droplet %  kubectl describe ingress echo-ingress
Name:             echo-ingress
Namespace:        default
Address:          138.197.135.165
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  thats4free.com  
                  /   echo2:80   10.244.0.47:80)
Annotations:      kubernetes.io/ingress.class: nginx
                  nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                  nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  Sync    105s (x2 over 2m37s)  nginx-ingress-controller  Scheduled for sync

jareas@Jareass-Mac-mini droplet % curl -vvv -I http://thats4free.com 
*   Trying 138.197.135.165...
* TCP_NODELAY set
* Connection failed
* connect to 138.197.135.165 port 80 failed: Connection refused
* Failed to connect to thats4free.com port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to thats4free.com port 80: Connection refused

but, if I go to the machine and access the endpoint it works.

root@ubuntu-s-2vcpu-2gb-tor1-01:~# curl 10.244.0.47
<html><body><h1>It works!</h1></body></html>

I have a feeling is because the sync is not going through

@jrareas You got connection refused so app is not listening on that IP or port so just netstat in app and comment

[tianhe-oi]

We are having similar issue and it seems to be cause by different ingresses using same host name. Maybe check whether you have a duplicate ingress resource defined in other namespace.

[jcbagtas]

Same experience here.

Somehow I can fix the issue whenever I install curl and exec a curl localhost command inside the pod

here is my simple deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: hello-world
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hello-world
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        name: hello-app
        ports:
          - containerPort: 8080
        resources: {}
status: {}
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: hello-world
  name: hello-world
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello-world
status:
  loadBalancer: {}

ran kubectl apply -f deployment.yaml

Then

curl -vvv -I http://$SERVICE_IP:80

Got

* Rebuilt URL to: http://$SERVICE_IP:80/
*   Trying 10.104.110.93...
* TCP_NODELAY set

Then I tried to curl localhost from the inside:

(installed curl first) kubectl exec hello-world-pod -- apk add --no-cache curl

(curl localhost from inside the single pod)

kubectl exec hello-world-5f664ff56d-lxfxh -- curl localhost:8080

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100Hello, world!   0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Version: 1.0.0
Hostname: hello-world-5f664ff56d-lxfxh
    68  100    68    0     0  17000      0 --:--:-- --:--:-- --:--:-- 17000 

Then I tried to curl it from the outside:

curl -vvv -I http://$SERVICE_IP:80

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[iamNoah1]

Hi @pangwawa @AfrouzMashayekhi @Krishna-teja @jrareas @tianhe-oi @jcbagtas, do you still have issues with ingress-nginx? I think that the ingress-nginx-users slack channel might a better place to aks for help. Please consider closing this issue.

[robincher]

I am facing the same issue right, for some reasons it keep stuck on schedule for sync. Is there anyway to further debug this?

Updated: Managed to fix my issues, it seems like the upstream service wasn't stable enough.

[bharathreddy-lab]

I am facing the same issue. Any solutions?

Name:             default-backend-ingress-example
Namespace:        default
Address:          ****************.elb.ap-south-1.amazonaws.com
Default backend:  sample-service:80 (10.32.0.4:80,10.32.0.5:80,10.32.0.6:80)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     sample-service:80 (10.32.0.4:80,10.32.0.5:80,10.32.0.6:80)
Annotations:  kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    17m (x2 over 17m)  nginx-ingress-controller  Scheduled for sync```

[longwuyuan]

hi, Accessing your app through a ingress resource that is managed by ingress-nginx-controller is documented here https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/ and is discussed daily at kubernetes.slack.com in the ingress-nginx-users channel. The usual problems range from firewalled port or ipaddress or misconfigurations.

Creating a issue here just for not being able to connect is a slow and less useful way to resolve this problem.

I suggest you close this issue and come to kubernetes.slack.com in the channel ingress-nginx-users. There are a lot of developers and engineers there so there is many chances of more people looking at your messages and helping you out.

[bharathreddy-lab]

@longwuyuan I wanted to join the workspace. How do I join? Do someone need to send the invite for the workspace?

[longwuyuan]

Slack.k8s.ioThanks,  -- ; Long

12 Jul 2021, 13:21 by @.***:

@longwuyuan https://github.com/longwuyuan> I wanted to join the workspace. How do I join? Do someone need to send the invite for the workspace?

— You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub https://github.com/kubernetes/ingress-nginx/issues/6506#issuecomment-878055067> , or > unsubscribe https://github.com/notifications/unsubscribe-auth/ABGZVWUYMUQU3H56IIUHJYDTXKNJFANCNFSM4UAIXHSQ> .

[andrewrotherham]

@tianhe-oi you were spot on ! I had an ingress with the same name in a different name space...once I'd got rid of it, I could browse to my app just fine. .just do kubectl get ingress --all-namespaces to check

[longwuyuan]

/lifecycle rotten

[karezza]

Experiencing this also. Clean kubernetes install with baremetal style install of ingress. I have not setup a loadbalancer such as metallb, maybe that will help? Interesting how it is saying "Scheduled for sync" and just hanging there.

Update: Installing metallb & then there was a controller service that had to be updated that needed a loadbalancer ip, then redid my deployment/service/ingress and it worked.

[dharmendrakariya]

same issue!! can anybody please help here?

[iamNoah1]

/remove lifecycle-rotten

[iamNoah1]

/remove-lifecycle rotten

[anurag4516]

Any Update on issue .. We are also facing same

[iamNoah1]

/help

[k8s-ci-robot]

@iamNoah1: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/6506): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[adiii717]

Any Updates on the issue? We are also facing the same, no error but keep sync in progress

[nauco]

I had a similar problem, and I solved it. In my case, such an event had nothing to do with it (I also saw message "Scheduled for sync"). I just kept modifying my code and test curl commands in local over and over. The problem was that the loadbalancer was in a private-subnet (I was developing with aws).

If you develop in cloud, turn your eyes to the infrastructure. Test in different environments. (on eks, some pods, bastion (if you have), in&out of vpc, subnet and so on, not only local)

I hope it helps, I write my experience. Good luck.

[karezza]

1.22.* be sure you have:

spec:
  ingressClassName: nginx

[rikatz]

/assign

[rikatz]

Hi folks, please:

• Make sure you are running the latest version
• Send a way for us to reproduce this

That said, can you please let me know if this is still a problem?

Thanks

[iamNoah1]

/triage needs-information

[Choko256]

Still a problem, it worked fine during 4 hours, and adding a new ingress for another app made everything down. So I deleted the other app but my main app is still in timeout status.

I have two ingresses in the same namespace with different names and no other ingresses :

# kubectl get ing -A
NAMESPACE   NAME                             CLASS   HOSTS             ADDRESS       PORTS     AGE
mynamespace    ingress-api                  nginx   api.myapp.com   10.71.54.27   80, 443   6d12h
mynamespace    ingress-web                 nginx   www.myapp.com   10.71.54.27   80, 443   6d12h

The attached services are working well, I can curl them from the server itself and the response is immediate. But curl the ingress and nothing happens, it times out. Same thing from the domain name (and the DNS configuration is OK, it worked on Monday).

The endpoints in the ingresses description are OK, they are correctly pointing to the services. But when I get the logs of the ingress controller :

# kubectl ingress-nginx logs -n ingress-nginx
I1116 17:30:27.733861       9 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"mynamespace", 
Name:"ingress-web", UID:"79fa3926-c63e-41fd-9b57-394156d577a5", APIVersion:"networking.k8s.io/v1", 
ResourceVersion:"1104336", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

And that's it. Nothing happens whatever I do. Is a load balancer needed ? I don't have one. I spent 4 hours yesterday trying to understand what happens, I even updated the controller to 1.0.5 but nothing changed.

I'm on microk8s, installed nginx controller with the baremetal method (the embedded microk8s addon is far outdated), on Ubuntu Server 20.04.

[iamNoah1]

@Choko256 have you configured your ingressClass correctly?

[Choko256]

yes I have

spec:
  ingressClassName: nginx

in my ingress yaml descriptors and as you can see in the list of ingresses, the class is registered correctly.

[iamNoah1]

Is it configured also in the ingress resources correctly?

[Choko256]

# kubectl describe ing -n mynamespace ingress-api
Name:             ingress-api
Namespace:        mynamespace
Address:          10.71.54.27
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
    secret-cert-api terminates api.myapp.com
Rules:
    Host             Path  Backends
    ----             ----  --------
    api.myapp.com
                       /   service-api:64003 (10.1.196.222:64003)
Annotations:       cert-manager.io/cluster-issuer: letsencrypt-prod
                   kubernetes.io/tls-acme: true
                   nginx.ingress.kubernetes.io/backend-protocol: HTTP
Events:            <none>

I think so, what do you mean by ingress resources ?

[longwuyuan]

Hi, The questions you are posting are more support related and less of a possible bug or problem in the controller, based on the data provided.

A better solution is to come talk about this on kubernetes.slack.com. You can register at slack.k8s.io if required a new login. There are more engineers, developers and experienced professionals on slack and talking in the ingress-nginx-users channel on slack will have a wider reach.

For the actual info, you can look at kubectl get events as well the logs of the ingress-controller pod.

Thanks.

[Choko256]

Ok I'll try that thanks!

[longwuyuan]

If there is a bug or a problem with the controller, then we can add that data here, change the title and track the bug/problem. For now let us close this issue and reopen it after we find some bug/problem related data. Others facing the same problem are advised to come talk on slack. Regards.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/6506#issuecomment-971463600): >If there is a bug or a problem with the controller, then we can add that data here, change the title and track the bug/problem. For now let us close this issue and reopen it after we find some bug/problem related data. Others facing the same problem are advised to come talk on slack. Regards. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[akshat-crypto]

I am still facing this issue. Did anyone find the solution to this minikube version: v1.23.2

[Erokos]

Running the latest version of the chart 4.0.17 on the latest kubernetes version 1.23.4 on AWS EC2, configured by kubeadm yields the same results. Most of the time requests to my apps pass, but often timeout. I see this "Scheduled for sync" on every ingress resource in different namespaces. After I downgraded my chart version to 4.0.6, it worked as it previously did with that version. So in my experience this has something to do with the nginx controller code.

[longwuyuan]

Come discuss this in the ingress-nginx-users channel of the Kubernetes slack please. There is less resources to track support type use-cases in GitHub.

[dinobagio]

Hi, facing the same issue. How do we get invited to the slack channel ?

[longwuyuan]

@dinobagio slack.k8s.io

[dinobagio]

@dinobagio slack.k8s.io

Thanks...I'm in ! And asked for help there !

[linmuqin]

@longwuyuan I have the same problem, it has been bothering me for a long time, how can I solve it?

[dinobagio]

@linmuqin After I've deleted my other ingress/ingress controller the issue disappeared.

[linmuqin]

@dinobagio Why delete other ingress? I need to do a canary to make the specified traffic go into the new service

[dinobagio]

@linmuqin I wasn't using that second ingress on same service, it was for an old purpose never confirmed eventually.

[Nowaker]

Facing the same issue too. No changes to the cluster or ingresses - just started affecting one of our ingress resources.

[chatagiri]

I faced the same problem, I removed all resources of nginx-ingress-controller and installed nginx-ingress-controller again and now I can access it. I do not know the root cause.