Expand configuration for Elastic to support serverless

[dhiaayachi]

Is your feature request related to a problem? Please describe.

We were looking to use Elastic's serverless option for our visibility store, but to connect we would need to set an Authorization header, which currently doesn't appear to be supported by the config for the ES client: https://github.com/temporalio/temporal/blob/66ab754e7e8d60421678457df1914fa7ddae9dc3/common/persistence/visibility/store/elasticsearch/client/config.go#L45-L60

Describe the solution you'd like

I'd like to be able to add a configuration like the following:

elasticsearch:
  # ...
  headers:
    Authorization: "ApiKey {{ .Env.TEMPORAL_ES_VISIBIILITY_STORE_PASSWORD }}"

Describe alternatives you've considered

Another option would be to have this kind of a format:

elasticsearch:
  # ...

  auth:
    basic:
      username: "..."
      password: "..."
    # or
    elastic:
      apiKey: "..."

But it feels far less flexible and a bit more complicated to implement.

Other than that, I can't really think of any alternative, besides having to set up a proxy that adds the header. Ideally, we avoid the operational overhead of another proxy and just do it straight on the client.

[dhiaayachi]

Feature Request: Elasticsearch client configuration for Authorization headers

Is your feature request related to a problem? Please describe.

Currently, the Temporal Elasticsearch client configuration does not support adding custom headers, specifically the Authorization header, which is required for connecting to Elastic's serverless Elasticsearch option. This limits the ability to use serverless Elasticsearch as a Visibility store.

Describe the solution you'd like

The ability to add custom headers to the Elasticsearch client configuration would resolve this issue. This could be achieved by adding a new configuration option like:

elasticsearch:
  # ...
  headers:
    Authorization: "ApiKey {{ .Env.TEMPORAL_ES_VISIBIILITY_STORE_PASSWORD }}"

Describe alternatives you've considered

• Proxy solution: This involves setting up a separate proxy server that adds the Authorization header. However, this adds operational overhead and complexity to the deployment.
• Predefined authentication types: Defining a pre-defined configuration for specific authentication types (e.g., basic auth, API key) offers less flexibility and might require more code changes.

Why the proposed solution is better

The proposed solution is more flexible and efficient. It allows users to specify arbitrary headers directly in the configuration, without the need for additional proxies or complex code changes. This makes it easier to integrate with services that require custom headers for authorization.

References

• Elasticsearch Authorization
• Temporal Elasticsearch client configuration

[dhiaayachi]

Thanks for the feature request! This is a good idea. It would definitely add flexibility to how users configure the ES client.

As a workaround, you can achieve similar functionality by configuring a proxy that adds the authorization header.

Another option would be to set up the Elastic Serverless option with basic authentication and provide the username and password in the elasticsearch section of the Temporal configuration. This would not be as flexible as adding headers, but it would be simpler to implement.

Let us know if you have any other questions or requests.

[dhiaayachi]

Thanks for the feature request. You can workaround this by setting up a proxy that adds the Authorization header.