search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.32k stars 685 forks source link

Change to mappings breaks canary routes #5578

Closed andrazjelenc closed 1 day ago

andrazjelenc commented 4 months ago

Describe the bug On each modification of existing mappings, creating of new mapping or deleting a mapping, all canary mappings get an additional route. Additional route is seen on Diagnostics UI, in logs and even in generated envoy.json.

To Reproduce Steps to reproduce the behavior:

  1. Apply following manifest to the k8s cluster where you have Emissary Ingress installed. This will create one canary with two services and one basic one.

    apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: single
namespace: develop
spec:
ambassador_id:
- ambassador
host: another-host
prefix: /single/
service: random-service-A:8081
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: canary-one
namespace: develop
spec:
ambassador_id:
- ambassador
host: random-host
prefix: /test/
service: random-service-B:8080
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
name: canary-two
namespace: develop
spec:
ambassador_id:
- ambassador
host: random-host
prefix: /test/
service: random-service-C:8081

  2. On Diagnostics UI inspect routes and see that everything is fine.

    Screenshot 2024-02-22 at 08 31 51

  3. Modify property of one of the existing mappings. For example we can modify service on mapping single. Any modification would trigger the issue.

  4. On Diagnostics UI inspect routes again and see additional mapping on canary.

    Screenshot 2024-02-22 at 08 33 26

  5. I know that there is a bug on UI about weights (https://github.com/emissary-ingress/emissary/issues/2142). But this is not only frontend bug, this duplicated mapping is also seen in the logs if we run Emissary Ingress with environment variable AMBASSADOR_DEBUG set to "diagd".

    ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <Mapping single.develop.1>: GID 21ca1689d6fa02eda3784f8eb1baae3ec29503f8 route_weight [0, 8, 22, 0, '/single/', 'GET', ':authority-another-host'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <InternalMapping ambassador.ambassador.1>: GID a2c3d327956b3acbfedb3662b5f50528dae653b5 route_weight [0, 26, 11, 0, '/ambassador/v0/check_alive', 'GET', ':authority-*'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <InternalMapping ambassador.ambassador.1>: GID 65f77b4cfda77405d7353cc1c83de745ee74b141 route_weight [0, 26, 11, 0, '/ambassador/v0/check_ready', 'GET', ':authority-*'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRAuth: svc auth, weight 100, grpc False, ctx_name None, location auth.ambassador.1
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 1d55ae3c39e58c6de3c526c16c37ec66c83feb2d
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning calculated weight 33 of remaining weight 100 to each of 3 weightless mappings
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 33 to weightless mapping canary-one
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 66 to weightless mapping canary-two
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to weightless mapping canary-two
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 21ca1689d6fa02eda3784f8eb1baae3ec29503f8
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping single in group
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for a2c3d327956b3acbfedb3662b5f50528dae653b5
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping internal_liveness_probe_mapping in group
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 65f77b4cfda77405d7353cc1c83de745ee74b141
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping internal_readiness_probe_mapping in group

    Exactly the same issue appears if we create new mapping or delete one of the mappings. And on each change, all canary mappings get one more duplication.

If we then restart Emissary Ingress the duplicated routes disappear and everything is fine again.

Expected behavior Creating, deleting or modifying mappings should not produce duplicated routes on canary mappings.

Versions (please complete the following information):

kflynn commented 1 day ago

I think that this is probably the same bug as #5702 -- I'm going to close this one, since the other has more information. Thanks!