search

jharting / openshift-rabbitmq-cluster

Deploys a RabbitMQ cluster in OpenShift
39 stars 52 forks source link

Pod Service DNS, AddressType: hostname #4

Closed ghost closed 6 years ago

ghost commented 6 years ago

Hello @jharting , Thank you for sharing. Is this config really working with OpenShift? I'm using OpenShift 3.7 and cannot get service discovery run. I'm facing 2 issues: 1.) If I'm using a headless service (clusterIP: none) and a full qualified hostname, the name resolution doesn't work: ERROR: epmd error for host rabbitmq-68-7fcdw.rabbitmq-cluster.dcrpi-omsf-dev0.svc.cluster.local: nxdomain (non-existing domain) With a clusterIP name resolution works, but it resolves the IP address of the service, not of the pod. In this case I get an epmd timeout.

2.) The cluster_formation.k8s.address_type = hostname doesn't work, as the OpenShift API doesn't return the hostname like Kubernetes does. I have already opened a feature request: https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/33

Sitenote: I'm using a DeploymentConfig, not a StatefulSet. But this shouldn't make a difference.

I think both issues are related to the same root cause, as the pod-hostname field is used for the dns discovery: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

Any help is appreciated. Thank you Roberto

jharting commented 6 years ago

@ramato-procon It does work with OpenShift. I do not have access to OpenShift 3.7 any longer but I just now tested with minishift 1.14 (matches OS 3.7.1) and the cluster deployed fine

 {running_nodes,
     ['rabbit@rabbitmq-cluster-1.rabbitmq-cluster.rabbit.svc.cluster.local',
      'rabbit@rabbitmq-cluster-0.rabbitmq-cluster.rabbit.svc.cluster.local']},

Have you tried using the origin template with StatefulSet? I wonder whether the reason the headless service does not work is because it needs a StatefulSet.

ghost commented 6 years ago

Unfortunately the beta API is deactivated in our cluster and StatefulSets are beta in OpenShift 3.7. Therefore I cannot test with a StatefulSet. After reading the Kubernetes and OpenShift documentation and testing a whole day with different settings in the DeploymentConfig, I think the reason could be that it's only working with a StatefulSet. But there is no hint in any documentation. Therefore I have created an issue to OpenShift: https://github.com/openshift/origin/issues/20632

ghost commented 6 years ago

I have analyzed the Kubernetes code and it is working with StatefulSets, only. Unfortunately it is not documented in the Kubernetes docu. Thank you.