We are facing an issue while updating any configuration at module object. We are trying to update circuit breaker settings at module so that it gets applied across all the mappings present in the cluster. However, rolling this change caused a complete downtime. We observed, as soon as the module change got deployed, few of the ambassador pods started giving 503 ("no healthy upstream") error for all the requests. We checked the configuration inside the emissary pod which did get updated as per the module update. On further debugging, we checked any config that we update in module object causes this 503 at all the upstream service. This gets resolved as soon as rotate all the emissary pods.
Any idea what could be the issue? It seems like any module object update causes a complete downtime.
Hello,
We are facing an issue while updating any configuration at module object. We are trying to update circuit breaker settings at module so that it gets applied across all the mappings present in the cluster. However, rolling this change caused a complete downtime. We observed, as soon as the module change got deployed, few of the ambassador pods started giving 503 ("no healthy upstream") error for all the requests. We checked the configuration inside the emissary pod which did get updated as per the module update. On further debugging, we checked any config that we update in module object causes this 503 at all the upstream service. This gets resolved as soon as rotate all the emissary pods. Any idea what could be the issue? It seems like any module object update causes a complete downtime.
Running emissary version 3.9.1