[libcluster:examplephx] unable to connect to :"examplephx@172.17.0.6"

[manjufy]

Hey, First of all, thanks for a great Elixir example app with k8. It was very helpful.

I ran into this issue [libcluster:examplephx] unable to connect to :"examplephx@172.17.0.6"

however application runs well, just keep getting the above warning.

[bbhoss]

Does clustering work at all? Does it show it connecting to any other nodes? Please post your kubernetes config and libcluster config.

On Tue, Mar 28, 2017 at 9:21 AM -0600, "Manjunath Reddy" notifications@github.com wrote:

Hey,

First of all, thanks for a great Elixir example app with k8. It was very helpful.

I ran into this issue [libcluster:examplephx] unable to connect to :"examplephx@172.17.0.6"

however application runs well, just keep getting the above warning.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[manjufy]

Here is my libcluster config

config :libcluster, topologies: [ examplephx: [ strategy: Cluster.Strategy.Kubernetes, config: [ kubernetes_selector: "app=examplephx", kubernetes_node_basename: "examplephx"]]]

I'm exactly using your k8s config, not much difference.

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: examplephx
    tier: frontend
spec:
  type: LoadBalancer
  ports:
    # the port that this service should serve on
  - port: 8080
  selector:
    app: examplephx
    tier: frontend
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: frontend
  # these labels can be applied automatically
  # from the labels in the pod template if not set
  labels:
    app: examplephx
    tier: frontend
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 3
  # selector can be applied automatically
  # from the labels in the pod template if not set
  # selector:
  #   app: guestbook
  #   tier: frontend
  template:
    metadata:
      labels:
        app: examplephx
        tier: frontend
    spec:
      containers:
      - name: examplephx
        image: elixir:v1
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: PORT
          value: "8080"
        - name: MY_BASENAME
          value: examplephx
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: REPLACE_OS_VARS
          value: "true"
        - name: RELEASE_CONFIG_DIR
          value: /beamconfig
        - name: ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: app-config
              key: erlang-cookie
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config-volume
          mountPath: /beamconfig
      volumes:
      - name: config-volume
        configMap:
          name: vm-config

[manjufy]

I'm running the cluster locally (minikube) :-)

[manjufy]

Here is the logs

08:33:36.839 [warn] [libcluster:manju] unable to connect to :"manju@172.17.0.4"
08:33:36.839 [warn] [libcluster:manju] unable to connect to :"manju@172.17.0.5"
08:33:36.839 [warn] [libcluster:manju] unable to connect to :"manju@172.17.0.6"

[bbhoss]

Hmm, it seems that the network traffic is not making it to the other pods, although there are some other things that could go wrong as well. At least the lookup and search is working. Use kubectl exec to get a bash shell on one of the pods and see if you can ping another pod. I haven't tried using this with minikube, only GKE.

[manjufy]

Ha, Yes, I would try that. I found the issue. Its that since i'm using Kubernetes, need to name the nodes while I start the application. I have tried with the following approach

elixir --name $(MY_POD_NAMESPACE)@$(MY_POD_IP) -S mix && MIX_ENV=prod PORT=80 mix phoenix.server

node names are registered if the start the app in dev environment, however, I'm not sure how to register the node names in prod mode

[bbhoss]

It seems like the /beamconfig must not be working properly. Can you post what is at that path, if anything?

Did you perform

Create a secret and config map (I've included the proper vm.args for libcluster) see the bottom (Addendum) of this great post for details

from the readme?

[bbhoss]

kubectl create configmap vm-config --from-file=vm.args

[manjufy]

Yes, I did. Seems no luck with that.

Correct me :-), we use /beamconfig only if we are using release. For now I'm just compiling my app and starting the server.

may be its not the issue with vm-config, may be its the network. Like you mentioned, look up and search is working, but some some how nodes are not connected. will continue my investigation :-)

[manjufy]

Hey I made it work. Will update you on the working solution soon :-)

[manjufy]

https://github.com/manju16832003/phoenix-libcluster Look for branch kubernetes -> Use Minikube to test it locally

[lang100]

@bbhoss, I cant seem to connect either . getting this error:

2017-05-28T13:14:46.839169296Z 13:14:46.839 [error] [libcluster:examplephx] request to kubernetes failed!: {:failed_connect, [{:to_address, {'kubernetes.default.svc.cluster.local', 443}}, {:inet, [:inet], {:eoptions, {:undef, [{:ssl, :connect, ['kubernetes.default.svc.cluster.local', 443, [:binary, {:active, false}, {:ssl_imp, :new}, :inet, {:verify, :verify_none}], :infinity], []}, {:http_transport, :connect, 4, [file: 'http_transport.erl', line: 109]}, {:httpc_handler, :connect, 4, [file: 'httpc_handler.erl', line: 902]}, {:httpc_handler, :connect_and_send_first_request, 3, [file: 'httpc_handler.erl', line: 916]}, {:httpc_handler, :init, 1, [file: 'httpc_handler.erl', line: 243]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 247]}]}}}]}

[manjufy]

@lang100 where is your kubernetes installed? have you tried to install minikube locally and tried?

[lang100]

@manju16832003, my kubernetes is installed on google container engine. no, i dont have minikube locally. do u think it's neccery?

[lang100]

solved: https://github.com/bitwalker/libcluster/issues/19