akka / akka-management

Akka Management is a suite of tools for operating Akka Clusters.
https://doc.akka.io/docs/akka-management/
Other
255 stars 160 forks source link

Help configuring an akka cluster in kubernetes #263

Closed afrancoc2000 closed 6 years ago

afrancoc2000 commented 6 years ago

Hi,

I've been trying to create an akka cluster in kubernetes but I'm having a few problems, I tried it on my local machine using the kubernetes that comes with docker for windows and everything works great, but when I upload my yml file to my development environment, a kubernetes on a rancher I get a lot of trouble.

First the problem was the RBAC but creating a ClusterRole and a ClusterRoleBinding I managed to get by, but now even without this problems the cluster form, it does discover it self, but probing fails and I get like a cluster per pod. I have a singleton in my cluster and every pod I get has the singleton running that's how I guess the cluster is not forming.

I see there are some problems with #187 and #236 in the discovery, but I'm not sure if that is what is happening to me, it seems like the other pods are found but not been able to probe them, and I get a TCP alert.

The big difference between my local environment and dev is, I guess the use of namespaces.

I really don't get why after discovering the other pods the cluster doesn't form and every pod joins itself.

Some times I don't get the TCP error but still, the cluster is not formed.

Can you help me? I'm adding here my configuration files.

application.conf:

akka {
  loglevel: INFO

  actor {
    provider = "cluster"
  }

  cluster {
    seed-nodes = []
    seed-nodes = ${?SEED_NODES}
    shutdown-after-unsuccessful-join-seed-nodes = 120s
  }

  coordinated-shutdown.terminate-actor-system = on

  remote {
    log-remote-lifecycle-events = on
    netty.tcp {
      hostname = "127.0.0.1"
      hostname = ${?HOSTNAME}
      bind-hostname = 0.0.0.0
      port = 2551
      port = ${?PORT}
    }
  }

  discovery {
    method = kubernetes-api
    method = ${?DISCOVERY_METHOD}
    kubernetes-api {
      pod-namespace = "default" // in which namespace cluster is running
      pod-namespace = ${?K8S_NAMESPACE}
      pod-label-selector = "app=akka-simple-cluster"
      pod-label-selector = ${?K8S_SELECTOR}
      pod-port-name = "management" // name of cluster management port
      pod-port-name = ${?K8S_MANAGEMENT_PORT}
    }
  }

  management {
    http {
      hostname = "127.0.0.1"
      hostname = ${?HOSTNAME}
      bind-hostname = "0.0.0.0"
      port = 8558
      bind-port = 8558
    }
    cluster.bootstrap {
      contact-point-discovery {
        required-contact-point-nr = 2
        required-contact-point-nr = ${?REQUIRED_CONTACT_POINTS}
      }
    }
  }

}

akka-cluster.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: simple-cluster
spec:
  replicas: 3
  selector:
    matchLabels:
      app: akka-simple-cluster
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: akka-simple-cluster
    spec:
      containers:
      - image: myserver.com/akkacluster/simple-cluster:0.1.0
        imagePullPolicy: IfNotPresent
        name: akka-simple-cluster
        env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: K8S_NAMESPACE
            value: "simple-cluster"
        ports:
        - name: remoting
          containerPort: 2551
        - name: management
          containerPort: 8558
      imagePullSecrets:
      - name: ana-registry
---
kind: Service
apiVersion: v1
metadata:
  name: akka-simple-cluster
spec:
  type: NodePort
  selector:
    app: akka-simple-cluster
  ports:
  - protocol: TCP
    name: management
    port: 8558
    targetPort: management
  - protocol: TCP
    name: api
    port: 2551
    targetPort: api
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: ["*"] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: simple-cluster
subjects:
# Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
# For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
- kind: User
  name: system:serviceaccount:simple-cluster:default
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Dockerfile:

FROM openjdk:8
WORKDIR /home
COPY target/scala-2.12/SimpleCluster-assembly-0.1.jar /home/simple-cluster.jar
CMD ["java","-jar","/home/simple-cluster.jar"]

Master.scala:

package com.myserver.simplecluster

import akka.actor.{Actor, ActorLogging, ActorSystem, Props}
import akka.cluster.{Cluster, ClusterEvent}
import akka.cluster.ClusterEvent.ClusterDomainEvent
import akka.management.AkkaManagement
import akka.management.cluster.bootstrap.ClusterBootstrap
import akka.stream.ActorMaterializer
import com.myserver.simplecluster.actors.{Master, Worker}
import com.typesafe.config.ConfigFactory

import scala.concurrent.ExecutionContextExecutor

object KubernetesMain extends App {

    val role = "manager"

    val config =ConfigFactory.parseString(s"akka.cluster.roles=[$role]")
            .withFallback(ConfigFactory.load("application.kubernetes.conf"))

    implicit val system: ActorSystem = ActorSystem("akka-simple-cluster", config)
    implicit val materializer: ActorMaterializer = ActorMaterializer()
    implicit val executionContext: ExecutionContextExecutor = system.dispatcher
    implicit val cluster: Cluster = Cluster(system)

    AkkaManagement(system).start()
    ClusterBootstrap(system).start()

    system.actorOf(
        ClusterSingletonManager.props(
            Master.props(workTimeout),
            PoisonPill,
            ClusterSingletonManagerSettings(system).withRole(role)
        ),
        "master")

        system.actorOf(Worker.props(), "worker")

}

build.sbt:

name := "SimpleCluster"

version := "0.1"

scalaVersion := "2.12.6"

lazy val akkaVersion = "2.5.14"

lazy val akkaManagementVersion = "0.16.0"

Compile/mainClass := Some("com.myserver.simplecluster.Main")

libraryDependencies ++= Seq(
    "com.typesafe.akka"     %% "akka-cluster"           % akkaVersion,
    "com.typesafe.akka"     %% "akka-cluster-tools"     % akkaVersion,
    "com.typesafe.akka"     %% "akka-cluster-metrics"   % akkaVersion,

    "com.lightbend.akka.discovery" %% "akka-discovery-kubernetes-api" % akkaManagementVersion,
    "com.lightbend.akka.management" %% "akka-management" % akkaManagementVersion,
    "com.lightbend.akka.management" %% "akka-management-cluster-http" % akkaManagementVersion,
    "com.lightbend.akka.management" %% "akka-management-cluster-bootstrap" % akkaManagementVersion,
    "com.lightbend.akka.discovery" %% "akka-discovery-dns" % akkaManagementVersion,
    "com.github.TanUkkii007" %% "akka-cluster-custom-downing" % "0.0.12"
)

Logs:

[INFO] [08/09/2018 13:02:17.659] [main] [akka.remote.Remoting] Starting remoting
[INFO] [08/09/2018 13:02:17.966] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://akka-simple-cluster@10.42.13.151:2551]
[INFO] [08/09/2018 13:02:17.971] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://akka-simple-cluster@10.42.13.151:2551]
[INFO] [08/09/2018 13:02:18.017] [main] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Starting up...
[INFO] [08/09/2018 13:02:18.362] [main] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [08/09/2018 13:02:18.362] [main] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Started up successfully
[INFO] [08/09/2018 13:02:18.471] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - No seed-nodes configured, manual cluster join required
[INFO] [08/09/2018 13:02:19.569] [main] [ClusterBootstrap(akka://akka-simple-cluster)] Bootstrap using default `akka.discovery` mechanism: KubernetesApiSimpleServiceDiscovery
[INFO] [08/09/2018 13:02:19.695] [main] [AkkaManagement(akka://akka-simple-cluster)] Including HTTP management routes for ClusterHttpManagement
[INFO] [08/09/2018 13:02:19.833] [main] [AkkaManagement(akka://akka-simple-cluster)] Including HTTP management routes for ClusterBootstrap
[INFO] [08/09/2018 13:02:19.840] [main] [ClusterBootstrap(akka://akka-simple-cluster)] Got self contact point address: http://10.42.13.151:8558
[INFO] [08/09/2018 13:02:19.867] [main] [AkkaManagement(akka://akka-simple-cluster)] Binding Akka Management (HTTP) endpoint to: 0.0.0.0:8558
[INFO] [08/09/2018 13:02:20.352] [main] [ClusterBootstrap(akka://akka-simple-cluster)] Initiating bootstrap procedure using akka.discovery method...
[INFO] [08/09/2018 13:02:20.371] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiSimpleServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider]
[INFO] [08/09/2018 13:02:20.409] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/user/worker] Worker Path: worker
[INFO] [08/09/2018 13:02:20.455] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/user/$a] Worker Path: $a
[INFO] [08/09/2018 13:02:20.493] [akka-simple-cluster-akka.actor.default-dispatcher-15] [AkkaManagement(akka://akka-simple-cluster)] Bound Akka Management (HTTP) endpoint to: 0.0.0.0:8558
[INFO] [08/09/2018 13:02:20.528] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:21.572] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:21.932] [akka-simple-cluster-akka.actor.default-dispatcher-16] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.9.94:46192: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:22.184] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558, 10-42-9-94.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:22.189] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.202] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.206] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-9-94.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.211] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558, 10-42-9-94.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:22.221] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.223] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.225] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-9-94.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:22.389] [akka-simple-cluster-akka.actor.default-dispatcher-19] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.9.94:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:22.402] [akka-simple-cluster-akka.actor.default-dispatcher-18] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [3] contact points, confirmed [1], which is less than the required [2], retrying
[INFO] [08/09/2018 13:02:22.491] [akka-simple-cluster-akka.actor.default-dispatcher-4] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:22.507] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[ERROR] [08/09/2018 13:02:22.850] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558] Probing [http://10-42-4-124.simple-cluster.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(10-42-4-124.simple-cluster.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of Connection refused
[INFO] [08/09/2018 13:02:23.430] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:23.495] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:23.496] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:23.497] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:23.737] [akka-simple-cluster-akka.actor.default-dispatcher-2] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:23.751] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[ERROR] [08/09/2018 13:02:24.140] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558] Probing [http://10-42-4-124.simple-cluster.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(10-42-4-124.simple-cluster.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of Connection refused
[INFO] [08/09/2018 13:02:24.388] [akka-simple-cluster-akka.actor.default-dispatcher-18] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[INFO] [08/09/2018 13:02:24.538] [akka-simple-cluster-akka.actor.default-dispatcher-19] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:24.664] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:24.671] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:24.673] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:24.932] [akka-simple-cluster-akka.actor.default-dispatcher-2] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:24.966] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:25.400] [akka-simple-cluster-akka.actor.default-dispatcher-18] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[ERROR] [08/09/2018 13:02:25.530] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558] Probing [http://10-42-4-124.simple-cluster.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(10-42-4-124.simple-cluster.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of Connection refused
[ERROR] [08/09/2018 13:02:25.530] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558] Overdue of probing-failure-timeout, stop probing, signaling that it's failed
[INFO] [08/09/2018 13:02:25.549] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Received signal that probing has failed, scheduling contact point probing again
[INFO] [08/09/2018 13:02:26.244] [akka-simple-cluster-akka.actor.default-dispatcher-2] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:26.264] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:26.388] [akka-simple-cluster-akka.actor.default-dispatcher-18] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[ERROR] [08/09/2018 13:02:26.517] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-9-94.simple-cluster.pod.cluster.local-8558] Probing [http://10-42-9-94.simple-cluster.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Probing timeout of [http://10-42-9-94.simple-cluster.pod.cluster.local:8558]
[ERROR] [08/09/2018 13:02:26.517] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-9-94.simple-cluster.pod.cluster.local-8558] Overdue of probing-failure-timeout, stop probing, signaling that it's failed
[INFO] [08/09/2018 13:02:27.389] [akka-simple-cluster-akka.actor.default-dispatcher-18] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[INFO] [08/09/2018 13:02:27.477] [akka-simple-cluster-akka.actor.default-dispatcher-18] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:27.491] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:27.581] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:27.819] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:27.828] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:27.835] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[ERROR] [08/09/2018 13:02:27.908] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator/contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558] Probing [http://10-42-4-124.simple-cluster.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(10-42-4-124.simple-cluster.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of Connection refused
[INFO] [08/09/2018 13:02:28.416] [akka-simple-cluster-akka.actor.default-dispatcher-4] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[INFO] [08/09/2018 13:02:28.560] [akka-simple-cluster-akka.actor.default-dispatcher-4] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:28.576] [akka-simple-cluster-akka.actor.default-dispatcher-5] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:28.875] [akka-simple-cluster-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:28.917] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Located service members with name: [akka-simple-cluster]: [10-42-4-124.simple-cluster.pod.cluster.local:8558, 10-42-13-151.simple-cluster.pod.cluster.local:8558]
[INFO] [08/09/2018 13:02:28.923] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-4-124.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:28.925] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Ensuring probing actor: contactPointProbe-10-42-13-151.simple-cluster.pod.cluster.local-8558
[INFO] [08/09/2018 13:02:29.377] [akka-simple-cluster-akka.actor.default-dispatcher-3] [LowestAddressJoinDecider(akka://akka-simple-cluster)] Discovered [2] contact points, confirmed [1], which is less than the required [2], retrying
[INFO] [08/09/2018 13:02:29.854] [akka-simple-cluster-akka.actor.default-dispatcher-19] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.13.151:60520: Contact Point returning 0 seed-nodes ([Set()])
[INFO] [08/09/2018 13:02:29.894] [akka-simple-cluster-akka.actor.default-dispatcher-4] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.13.151:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:29.957] [akka-simple-cluster-akka.actor.default-dispatcher-2] [akka.actor.ActorSystemImpl(akka-simple-cluster)] Querying for pods with label selector: [app=akka-simple-cluster]
[INFO] [08/09/2018 13:02:30.082] [akka-simple-cluster-akka.actor.default-dispatcher-18] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Contact point [akka.tcp://akka-simple-cluster@10.42.4.124:2551] returned [0] seed-nodes []
[INFO] [08/09/2018 13:02:30.408] [akka-simple-cluster-akka.actor.default-dispatcher-4] [akka.tcp://akka-simple-cluster@10.42.13.151:2551/system/bootstrapCoordinator] Initiating new cluster, self-joining [akka.tcp://akka-simple-cluster@10.42.13.151:2551]. Other nodes are expected to locate this cluster via continued contact-point probing.
[INFO] [08/09/2018 13:02:30.604] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] is JOINING, roles [manager, dc-default]
[INFO] [08/09/2018 13:02:30.651] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] dc [default] is the new leader
[INFO] [08/09/2018 13:02:30.688] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Leader is moving node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] to [Up]
[INFO] [08/09/2018 13:02:31.337] [akka-simple-cluster-akka.actor.default-dispatcher-2] [HttpClusterBootstrapRoutes(akka://akka-simple-cluster)] Bootstrap request from 10.42.4.124:59702: Contact Point returning 1 seed-nodes ([Set(ClusterMember(akka.tcp://akka-simple-cluster@10.42.13.151:2551,1935708835,Up,Set(manager, dc-default)))])
[INFO] [08/09/2018 13:02:31.858] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Received InitJoin message from [Actor[akka.tcp://akka-simple-cluster@10.42.4.124:2551/system/cluster/core/daemon/joinSeedNodeProcess-1#80687263]] to [akka.tcp://akka-simple-cluster@10.42.13.151:2551]
[INFO] [08/09/2018 13:02:31.858] [akka-simple-cluster-akka.actor.default-dispatcher-16] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Sending InitJoinAck message from node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] to [Actor[akka.tcp://akka-simple-cluster@10.42.4.124:2551/system/cluster/core/daemon/joinSeedNodeProcess-1#80687263]]
[INFO] [08/09/2018 13:02:32.414] [akka-simple-cluster-akka.actor.default-dispatcher-15] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Node [akka.tcp://akka-simple-cluster@10.42.4.124:2551] is JOINING, roles [manager, dc-default]
[INFO] [08/09/2018 13:02:32.483] [akka-simple-cluster-akka.actor.default-dispatcher-3] [akka.cluster.Cluster(akka://akka-simple-cluster)] Cluster Node [akka.tcp://akka-simple-cluster@10.42.13.151:2551] - Leader is moving node [akka.tcp://akka-simple-cluster@10.42.4.124:2551] to [Up]
[WARN] [SECURITY][08/09/2018 13:02:35.241] [akka-simple-cluster-akka.remote.default-remote-dispatcher-39] [akka.serialization.Serialization(akka://akka-simple-cluster)] Using the default Java serializer for class [com.myserver.simplecluster.actors.GiveMeWork] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][08/09/2018 13:02:36.033] [akka-simple-cluster-akka.remote.default-remote-dispatcher-39] [akka.serialization.Serialization(akka://akka-simple-cluster)] Using the default Java serializer for class [com.myserver.simplecluster.actors.GotWork$] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][08/09/2018 13:02:36.053] [akka-simple-cluster-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://akka-simple-cluster)] Using the default Java serializer for class [com.myserver.simplecluster.actors.ProcesarBDSap] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][08/09/2018 13:02:39.824] [akka-simple-cluster-akka.remote.default-remote-dispatcher-39] [akka.serialization.Serialization(akka://akka-simple-cluster)] Using the default Java serializer for class [com.myserver.simplecluster.actors.Done] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][08/09/2018 13:02:40.949] [akka-simple-cluster-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://akka-simple-cluster)] Using the default Java serializer for class [com.myserver.simplecluster.actors.ProcesarBDGuideWire] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
afrancoc2000 commented 6 years ago

Found my problem, I was missing the bind-port property in the remote configuration, adding it solved the issue.

remote {
  log-remote-lifecycle-events = on
  netty.tcp {
    hostname = "127.0.0.1"
    hostname = ${?HOSTNAME}
    port = 2551
    port = ${?PORT}
    bind-hostname = 0.0.0.0
    bind-port = 2551
  }
}

Thanks!

adnanfaizan commented 5 years ago

Another visible problem is creating singleton right after ClusterBootstrap start. If service discovery is is in progress then singleton would not get created. So better to put singleton creation in the memberUp event.

Cluster(system).registerOnMemberUp({
 system.actorOf(
        ClusterSingletonManager.props(
            Master.props(workTimeout),
            PoisonPill,
            ClusterSingletonManagerSettings(system).withRole(role)
        ),
        "master")

        system.actorOf(Worker.props(), "worker")
})
lonnie-intercax commented 9 months ago

Ana Maria should be encouraged to write up this configuration for Clustering in Kubernetes as a blog post or how-to article for LightBend.

There's much good information located here that is otherwise spread out through Akka documentation.

johanandren commented 9 months ago

For the record the samples in Akka projection nowadays contain complete Kubernetes yamls (including ingress load balancer for k8 on AWS):

https://github.com/akka/akka-projection/tree/main/samples/grpc/shopping-cart-service-java https://github.com/akka/akka-projection/tree/main/samples/grpc/shopping-cart-service-scala

afrancoc2000 commented 9 months ago

Hi @lonnie-intercax, thank you for the proposal, this is an issue from 5 years ago and I have not worked in Akka for a while, so I would need to regain context in the latest changes and updates that have happened. But I will keep it in mind for when I have some extra time.

lonnie-intercax commented 9 months ago

;-) v/r Lonnie Click to schedule a call https://calendly.com/lonnie-vanzandt/30min

On Wed, Jan 10, 2024 at 1:27 PM Ana Maria Franco @.***> wrote:

Hi @lonnie-intercax https://github.com/lonnie-intercax, thank you for the proposal, this is an issue from 5 years ago and I have not worked in Akka for a while, so I would need to regain context in the latest changes and updates that have happened. But I will keep it in mind for when I have some extra time.

— Reply to this email directly, view it on GitHub https://github.com/akka/akka-management/issues/263#issuecomment-1885397596, or unsubscribe https://github.com/notifications/unsubscribe-auth/A67C37C7VPXJ5U43P4N55E3YN3MSPAVCNFSM4FOW75O2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBYGUZTSNZVHE3A . You are receiving this because you were mentioned.Message ID: @.***>