search

temporalio / docker-builds

Temporal service Docker images build
https://hub.docker.com/r/temporaliotest/auto-setup
MIT License
30 stars 59 forks source link

Using Kubernetes service discovery for ES_SEEDS #243

Open mhmtszr opened 2 months ago

mhmtszr commented 2 months ago

What are you really trying to do?

Deploying highly available Temporal cluster.

Describe the bug

We are using Kubernetes for our Temporal clusters. We don't want to give only one ip for ES_SEEDS because it can be changed thus we want to give our Kubernetes service definition. But our services contain port as well. I can give empty ES_PORT but still there is ":" character between seeds and port.

We also tried to give all IPs of the cluster seperated by comma(which is we didn't want to), but in this case Temporal server got some errors from Elasticsearch, It tried to send a request like this POST 10.10.10.1,10.10.10.2,10.10.10.3/mapping.

We can make ES_PORT totally optional in both auto-setup and Temporal server. What do you think about this? Do you have any different suggestions to us?

Or we can set ES_SERVER directly that it can overwrite.

robholland commented 1 month ago

I'm not sure I see the issue. Service names in Kubernetes will resolve to all IP addresses which have an endpoint is up. Can you not just set ES_SEEDS=myesservice.namespace?

robholland commented 1 month ago

(And ES_PORT to whatever port your service defines).

mhmtszr commented 1 month ago

It also resolves the port as well, It duplicates the port that I could not disable in k8s.

We solved the problem by defining internal dns but It's not a long term solution(because of caching problems and updating IPs).

robholland commented 1 month ago

I'm not sure what you mean that the k8s DNS resolves to the port as well. Can you show me an error to make it clearer at which layer it breaks?

mhmtszr commented 1 month ago

We are using ClusterIP for our service definitions, It contains IP addresses and port as well.

Such as:

test-service: IPs: 10.10.10.10, 10.10.10.11 Port: 8080

If I use "test-service.namespace" in my code I can directly access the application in that port, no need to define extra port definition.

In our case Temporal servers are pods, and there is a service definition for them.

Such as: IPs: 10.10.10.1, 10.10.10.2 Port: 7233

When I tried to establish a connection by using "temporal-service.namespace" I'm getting error because Temporal server automatically add ES_PORT to the end of the url.

If we look at https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh#L330 ES_SEEDS is our service definition which has port as well, I don't need to define extra port here, If I set ES_PORT empty still there is a ":" character in url which causes error.