Module configured`liveness_probe` is hitting the AuthService

[joshbranham]

Describe the bug When using the liveness_probe: true config in kind: Module, traffic to the /ambassador/v0/check_alive path (served from here) is routed to our AuthService before the responding with Ambassador is ready and waiting.

To Reproduce Deploy a Module like the following

apiVersion: getambassador.io/v3alpha1
kind: Module
metadata:
  name: ambassador
spec:
  config:
    ....
    liveness_probe:
      enabled: true

Deploy an AuthService like the following

apiVersion: getambassador.io/v3alpha1
kind: AuthService
metadata:
  name: authentication
spec:
  auth_service: "some-service"

Then you can see the request in the logs of your AuthService. To confirm, you can view the route for /ambassador/v0/check_alive in the diagnostic UI has no config to bypass authentication, which would come from bypass_auth: true like any other Mapping.

Expected behavior I would expect internal health checking routes to bypass AuthService and RateLimitService configurations.

Versions (please complete the following information):

• emissary-ingress v2.5.1
• Kubernetes via kOps on EC2
• v1.24.10

Additional context A workaround we found was to manually create Mappings for the health check route like the below, then set liveness_probe: false in the Module.

Sample Mapping:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: liveness-probe
  namespace: emissary-ingress
spec:
  hostname: "*"
  prefix: /ambassador/v0/check_alive
  rewrite: /ambassad/v0/check_alive
  service: emissary-ingress-admin.emissary-ingress:8877

[cindymullins-dw]

Hi @joshbranham , both Emissary and Edge Stack ship with an authService and every call goes to the authService even for readiness and liveness probes. Bypass-auth: true doesn’t turn that off, rather it turns off Filters you may have configured in Edge Stack. Is this behavior creating an error or problem, or are you trying to reduce noisy logs perhaps?

[joshbranham]

Carrying over conversation started in Slack.

• In our testing, turning the probes off in Module and creating our own Mapping to simulate the "auto-generated" one for the probes (if they were on), and setting bypass_auth: true does in fact prevent them from reaching our AuthService.
• This behavior is causing a large influx of logs, especially when you consider our ALB is health checking all EC2 instances in our cluster via /ambassador/v0/check_alive, which funnels down to N emissary pods (where N is less than the total of nodes in the cluster)

Is the intent with authenticating the probes to validate the AuthService works? In our example, we scaled our auth processes to zero and the probes still succeeded.

[cindymullins-dw]

I think its just one of several probes to check for the liveness/health of resources Emissary expects to communicate with. Our Support notes that in 3.x we added Active Health Checking which revamped this process and may have addressed this behavior. Would you be able to try 3.x and see if it behaves the same?

[joshbranham]

We are currently on 2.x and need to figure out some of the v3 protocol change issues related to rate limiting before we can upgrade. As for Active Health Checking, that looks to be additive and not something I would configure to modify the builtin behavior of the /ambassador/v0/check_alive route.

[cindymullins-dw]

Thanks for checking that. Will mark as a feature request. Aside from the excessive logging, does this cause problems for your deployment?

[joshbranham]

Thanks for checking that. Will mark as a feature request. Aside from the excessive logging, does this cause problems for your deployment?

It is not causing issues since we have now creating custom mappings to turn the auth pieces off. Before that, it was causing lots of logs and unnecessary load on our authentication service.