Open kriptor opened 1 year ago
And then I get the IR MISMATCH nad ENVOY CONFIG MISMATCH errors and some mappings from the other environment that wasn't changed disappear from http://127.0.0.1:8877/ambassador/v0/diag/ page. Those of course stop working (returning 404).
Here when I mentioned disappeared mappings from diag
page, I meant disappeard from Ambassador Route Table
on diag
page.
This is the actual change that triggers the mismatching errors and very likely the more problematic disappearance of analogous api.stg
routes... causing 404s on the stg
environment.
ambassador, dev-api.example.com, Mapping (getambassador.io) has changed:
# Source: ambassador-setup/templates/mappings.yaml
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: dev-api.example.com
labels:
helm.sh/chart: ambassador-setup-0.1.0
app.kubernetes.io/name: ambassador-setup
app.kubernetes.io/instance: ambassador-setup-dev
app.kubernetes.io/version: "0.1.0"
app.kubernetes.io/managed-by: Helm
my-host: wildcard-subdomain
spec:
connect_timeout_ms: 1000
docs:
ignored: true
+ error_response_overrides:
+ - body:
+ content_type: application/json
+ text_format_source:
+ filename: /ambassador/ambassador-errorpages/503-maintenance-lockdown.json
+ on_status_code: 503
hostname: dev-api.example.com
precedence: 9
prefix: /v1
rewrite: /v1
- service: api.dev:8080
+ service: api-nonexistent.dev:8080
timeout_ms: 10000
ambassador, dev-api.example.com-webhooks, Mapping (getambassador.io) has changed:
# Source: ambassador-setup/templates/mappings.yaml
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: dev-api.example.com-webhooks
labels:
helm.sh/chart: ambassador-setup-0.1.0
app.kubernetes.io/name: ambassador-setup
app.kubernetes.io/instance: ambassador-setup-dev
app.kubernetes.io/version: "0.1.0"
app.kubernetes.io/managed-by: Helm
my-host: wildcard-subdomain
spec:
connect_timeout_ms: 1000
docs:
ignored: true
+ error_response_overrides:
+ - body:
+ content_type: application/json
+ text_format_source:
+ filename: /ambassador/ambassador-errorpages/503-maintenance-lockdown.json
+ on_status_code: 503
hostname: dev-api.example.com
precedence: 8
prefix: /webhooks
rewrite: /webhooks
- service: api.dev:8080
+ service: api-nonexistent.dev:8080
timeout_ms: 10000
ambassador, dev-www.example.com, Mapping (getambassador.io) has changed:
# Source: ambassador-setup/templates/mappings.yaml
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: dev-www.example.com
labels:
helm.sh/chart: ambassador-setup-0.1.0
app.kubernetes.io/name: ambassador-setup
app.kubernetes.io/instance: ambassador-setup-dev
app.kubernetes.io/version: "0.1.0"
app.kubernetes.io/managed-by: Helm
my-host: wildcard-subdomain
spec:
connect_timeout_ms: 1000
docs:
ignored: true
error_response_overrides:
- body:
content_type: text/html
text_format_source:
filename: /ambassador/ambassador-errorpages/429.html
on_status_code: 429
- body:
content_type: text/html
text_format_source:
filename: /ambassador/ambassador-errorpages/500.html
on_status_code: 500
- body:
content_type: text/html
text_format_source:
filename: /ambassador/ambassador-errorpages/502.html
on_status_code: 502
- body:
content_type: text/html
text_format_source:
- filename: /ambassador/ambassador-errorpages/503.html
+ filename: /ambassador/ambassador-errorpages/503-maintenance-lockdown.html
on_status_code: 503
- body:
content_type: text/html
text_format_source:
filename: /ambassador/ambassador-errorpages/504.html
on_status_code: 504
hostname: dev-www.example.com
precedence: 10
prefix: /
rewrite: /
- service: web.dev:80
+ service: web-nonexistent.dev:80
timeout_ms: 10000
I am having the same issue! Whenever I update working mappings, routes get screwed!, and I can see IR MISMATCH errors. @kriptor any luck with that?
Hello @Kriptor,
Do you reproduce the error on a cluster containing only those 6 mappings? Or is it a cluster in which, in addition to these mappings, there are many others (many = hundreds)?
I am investigating a problem that, I believe, may be the same as your case (but with a lot of mappings).
Describe the bug I have a cluster running
dev
andstg
namespaced services (two services per namespace). I have a working setup and all my mappings are working. Then I change the existing mappings for one environment to point to nonexisting services, to put the environment into maintenance mode (returning 503). And then I get theIR MISMATCH
nadENVOY CONFIG MISMATCH
errors and some mappings from the other environment that wasn't changed disappear fromhttp://127.0.0.1:8877/ambassador/v0/diag/
page. Those of course stop working (returning 404).To Reproduce
web.dev:80
; hostnamedev-web.example.com
, prefix/
, rewrite/
)api.dev:8080
; hostnamedev-api.example.com
, prefix/v1
, rewrite/v1
)api.dev:8080
; hostnamedev-api.example.com
, prefix/webhooks
, rewrite/webhooks
)web.stg:80
; hostnamestg-web.example.com
, prefix/
, rewrite/
)api.stg:8080
; hostnamestg-api.example.com
, prefix/v1
, rewrite/v1
)api.stg:8080
; hostnamestg-api.example.com
, prefix/webhooks
, rewrite/webhooks
)web-nonexisting.dev:80
; hostnamedev-web.example.com
, prefix/
, rewrite/
)api-nonexisting.dev:8080
; hostnamedev-api.example.com
, prefix/v1
, rewrite/v1
)api-nonexisting.dev:8080
; hostnamedev-api.example.com
, prefix/webhooks
, rewrite/webhooks
)web.stg:80
; hostnamestg-web.example.com
, prefix/
, rewrite/
)api.stg:8080
; hostnamestg-api.example.com
, prefix/v1
, rewrite/v1
)api.stg:8080
; hostnamestg-api.example.com
, prefix/webhooks
, rewrite/webhooks
)Expected behavior I should be able to modify the existing mappings without running into strange mismatching errors and without the need to restart the emissary.
Versions (please complete the following information):
Additional context I believe there is a bug (looks unrelated to the actual reason fo mismatching errors but it would be nice to see econf diff insteda of ir diffs twice): https://github.com/emissary-ingress/emissary/blob/v3.7.2/python/ambassador_diag/diagd.py#L553 Should be:
errors += self.json_diff("econf", e1, e2)