Canary ingress with multiple paths not working.

[Dandush03]

When adding paths to k8s ingress with canary I get the following error

"successfully validated configuration, accepting" ingress="app/app-canary"
unable to find real backend for alternative backend app-service-80. Deleting.
Event(v1.ObjectReference{Kind:"Ingress", Namespace:"app", Name:"app-canary", UID:"f97c03f7-4a72-4d89-8381-60ce40b6700d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1663220", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for syn

and when doing a request I get the following

GET //_next/static/Lh4bPvu6zP4HnWLP4aQu-/_ssgManifest.js HTTP/1.1" 301 152 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:27.0) Gecko/20121011 Firefox/27.0" 211 1.442 [app1-app1-service-80] []

and the expected request should look like this (compare the end of both requests)

GET //_next/static/Lh4bPvu6zP4HnWLP4aQu-/_ssgManifest.js HTTP/1.1" 301 152 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:27.0) Gecko/20121011 Firefox/27.0" 211 1.442 [app1-app1-service-80] [app-app-service-80]

This issue only happens when I add a path to nginx canary. It does work when using root path, but it's not what is intended to do

Working Config

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-canary
  namespace: app
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "100"
    nginx.ingress.kubernetes.io/canary-by-header: "canary"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

Not Working Config

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-canary
  namespace: app
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "100"
    nginx.ingress.kubernetes.io/canary-by-header: "canary"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /assets
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

Desired Config, which is working without canary so there is no issue with the services.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-canary
  namespace: app
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "100"
    nginx.ingress.kubernetes.io/canary-by-header: "canary"
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /assets
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80
      - path: /gcp-assets
        pathType: Prefix
        backend:
          service:
            name: app1-service
            port:
              number: 80
        - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

Looking forward to any assistance.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[longwuyuan]

The template of a new issue asks questions that help understand the context, the envirnment, and other details, which readers use to understand the issue being reported. And without all that information, its harder to work on the issue or to try and reproduce it.

Please click new issue button and see the questions asked. Please answer those questions by editing your original post.

Canary or even otherwise specific data requires some simple info related to the event and the processing. Yaml files posted here cause a guess to be the basis of comments. For example, in this canary problem, you could help by posting ;

• kubectl get po,svc,ing -A -o wide

• kubectl -n $appnamespace describe ing

• Complete curl command with -vv and its response

• kubectl -n $controllernamespace logs $controllerpodname

• If you don't have a non-canary rule for the host+path match but then create a ingress with canary-annotation for the same host+path, I don't currently understand how it qualifies to be a canary

• If your backend pod can not process the path passed to it in the request, then the controller-pod will just return whatever response comes from the backend pod

• But above 2 remarks are guesses and not based on any data posted here

[longwuyuan]

/remove-kind bug

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[hemuvemula]

Hi we are facing the exact same issue behavior but when having multiple Hosts.

should one expect a canary to work between the below two ingress objects? were both have one common host+path combo and one new Host + Path combo that is not in the non-canary ingress object.

`apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app-canary-v1 namespace: app annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "false" nginx.ingress.kubernetes.io/canary-weight: "0" spec: rules:

• host: www.example.com http: paths:
  • path: / pathType: Prefix backend: service: name: app-service-v1 port: number: 80`

and another object

`apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app-canary-v2 namespace: app annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50" spec: rules:

• host: www.example.com http: paths:
  • path: / pathType: Prefix backend: service: name: app-service-v2 port: number: 80
• host: www.example2.com http: paths:
  • path: / pathType: Prefix backend: service: name: app-service-v2 port: number: 80`

[bumarcell]

From docu: "Currently a maximum of one canary ingress can be applied per Ingress rule"

[gzaripov]

Hi! Do you have any updates on this? Do you plan to support more than one canary ingress? At least for the cookie and header options.

[longwuyuan]

@gzaripov not in the near future

[gzaripov]

@longwuyuan thank you for the quick answer.

We would use this feature to deploy canary versions of our backends on PR's. By using canary we can deploy microservices that are changed in PR. Probably it is not the case the nginx canary feature was designed for. If you know the better solution, I will be very grateful if you share it with me.

Also I checked the code and looks like implementing that feature isn't so complicated. I think I can try to create PR with that feature if you will assure me there is chances to merge it in two-three weeks range.

[longwuyuan]

@gzaripov I am not a developer so wait for a developer to comment on code.

• As far as I can understand, the original creator of this issue wants to change the path field in a canary ingress, but has
  • not shown the main ingress
  • nor shown the HTTP request
  • not shown the logs of the controller pod
• A second user has commented that they want to add a second rule to the canary ingress
• Now you want multiple canary rules in one ingress resource, to help in your CI/CD process

So there is no consistency in what problem needs to be solved. Combining multiple problems in one issue creates a problem for tracking.

My awareness of canary is that one single HTTP/HTTPS request is the foundation. So I don't understand the logic behind adding multiple rules applied to the same HTTP request.

Also, CI/CD designs implement dev builds in their own environs to facilitate the feedback loop. But canary is for prod builds so I don't see the logic in the PR related comment from you.

You can look for CI/CD examples on google search or github search. But common practice is dev builds are served from dev environment from a dev URL and test builds are served from a test environment from a test URL.

[longwuyuan]

I see that the comments I made in the message just above this one still stands so will not repeat the comments.

But reading this today, an update is needed because the project has acute shortage of resources to work on the expectations described here. We are actually deprecating features that we can not support/maintain due to shortage of resources. The developers are also busy with security & Gateway-API.

Thus no action item is being tracked here and hence I am closing this issue as it adds to the tally of open issues without action items. There is no data about original ingress and the curl request etc etc but I see that its expected that that kind of info can be guessed. Unfortunately unless a CVE or a breaking change is reported with data as proof, its going to be harder to allocate resources to do research and guess work on this.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9830#issuecomment-2343672706): >I see that the comments I made in the message just above this one still stands so will not repeat the comments. > >But reading this today, an update is needed because the project has acute shortage of resources to work on the expectations described here. We are actually deprecating features that we can not support/maintain due to shortage of resources. The developers are also busy with security & Gateway-API. > >Thus no action item is being tracked here and hence I am closing this issue as it adds to the tally of open issues without action items. There is no data about original ingress and the curl request etc etc but I see that its expected that that kind of info can be guessed. Unfortunately unless a CVE or a breaking change is reported with data as proof, its going to be harder to allocate resources to do research and guess work on this. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.