search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.24k stars 4.41k forks source link

Terminating gateway: Unable to override default 15s http timeout #20269

Open mr-miles opened 7 months ago

mr-miles commented 7 months ago

Overview of the Issue

I have a service in the mesh, that is making requests to an external service outside the mesh. The external service is registered as a "real" consul service in the catalog, not just through a service-defaults destination. The calls are going via the terminating gateway, which is set up with custom certificate for the onward connection.

All works fine until http responses take more than 15s to return - at this point I receive 504 timeout errors.

I have tried setting the http timeout by adding a service-defaults entry for the service with LocalRequestTimeoutMs set but this had no effect.

Because the external service needs a "special" certificate, I can't easily switch to tcp either.

Is it possible to set the timeout, and if so, where can I do it?

Consul info for both Client and Server

Consul 1.17.1 - via helm chart EKS 1.28 Transparent proxy enabled

karun100 commented 7 months ago

Can we have Service router configuration, and set RequestTimeout there rather than change in code as per below example:

https://developer.hashicorp.com/consul/docs/connect/config-entries/service-router#routes-destination-requesttimeout

mr-miles commented 7 months ago

For anyone else ending up here, I found a solution by digging into the codebase:

Terminating gateway does look at the RequestTimeout property of the relevant ServiceResolver - https://developer.hashicorp.com/consul/docs/connect/config-entries/service-resolver#requesttimeout - and if you set this then you can control the timeout.

Code: https://github.com/hashicorp/consul/blob/0ec7bddb9ad54e6e74c86672a3b4bac9f99a40fe/agent/xds/routes.go#L224

This is not obvious in the docs at all - terminating gateway docs mention about service-defaults destinations but say nothing about the service-resolver. IMO capturing this as example config there would be very useful.