Cluster over LAN: bad request-replay performance

[vasily-kirichenko]

Server:

open System.IO
open Akka.FSharp
open Contracts
open System

[<EntryPoint>]
let main _ = 
    let system = 
        Configuration.parse (File.ReadAllText "akka.hocon")
        |> System.create "ActorSystem1"

    let _actor =
        spawn system "Server" <| fun mb ->
            let startTime = DateTime.UtcNow
            let rec loop count = actor {
                let! msg = mb.Receive()
                match msg with
                | Request.Add (x, y) -> 
                    mb.Sender() <! Response.Added (x + y)
                if count % 1000 = 0 then
                    Logging.logInfof mb "%d total, %f /s" 
                                        count 
                                        (1000. * float count / (DateTime.UtcNow - startTime).TotalMilliseconds)
                return! loop (count + 1)
            }
            loop 0
    Console.ReadKey() |> ignore
    0

config:

akka {
  actor {
    provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
  }

  remote {
    log-remote-lifecycle-events = DEBUG
    helios.tcp {
      hostname = "server1"
      port = 2551
    }
  }

  cluster {
    seed-nodes = [ "akka.tcp://ActorSystem1@server1:2551" ]
    auto-down-unreachable-after = 30s
  }
}

Client:

open System.IO
open Akka.FSharp
open System
open Contracts

[<EntryPoint>]
let main _ = 
    let system = 
        Configuration.parse (File.ReadAllText "akka.hocon")
        |> System.create "ActorSystem1"

    let _actor =
        spawn system "Client" <| fun (mb: Actor<Response>) ->
            let server = select "akka.tcp://ActorSystem1@server1:2551/user/Server" system
            let startTime = DateTime.UtcNow
            let rec loop count = actor {
                server <! Request.Add (1, 2)
                let! _msg = mb.Receive()
                if count % 1000 = 0 then
                    Logging.logInfof mb "%d total, %f /s" 
                                        count 
                                        (1000. * float count / (DateTime.UtcNow - startTime).TotalMilliseconds)
                return! loop (count + 1)
            }
            loop 0

    Console.ReadKey() |> ignore
    0

config:

akka {
  actor {
    provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
  }

  remote {
    log-remote-lifecycle-events = DEBUG
    helios.tcp {
      hostname = "server2"
      port = 0
    }
  }

  cluster {
    seed-nodes = [ "akka.tcp://ActorSystem1@server1:2551" ]
    auto-down-unreachable-after = 30s
  }
}

Messages:

namespace Contracts

type Request =
    | Add of int * int

type Response =
    | Added of int

Full code here: https://github.com/vasily-kirichenko/AkkaNetTest

Server output:

[INFO][05.10.2015 10:07:25][Thread 0005][[akka://ActorSystem1/user/Server]] 927000 total, 326.696727 /s
[INFO][05.10.2015 10:07:28][Thread 0006][[akka://ActorSystem1/user/Server]] 928000 total, 326.695203 /s
[INFO][05.10.2015 10:07:31][Thread 0004][[akka://ActorSystem1/user/Server]] 929000 total, 326.689090 /s
[INFO][05.10.2015 10:07:34][Thread 0008][[akka://ActorSystem1/user/Server]] 930000 total, 326.673229 /s
[INFO][05.10.2015 10:07:37][Thread 0007][[akka://ActorSystem1/user/Server]] 931000 total, 326.678275 /s
[INFO][05.10.2015 10:07:40][Thread 0006][[akka://ActorSystem1/user/Server]] 932000 total, 326.682159 /s

Client output:

[INFO][05.10.2015 10:07:16][Thread 0004][[akka://ActorSystem1/user/Client]] 924000 total, 327.153086 /s
[INFO][05.10.2015 10:07:19][Thread 0021][[akka://ActorSystem1/user/Client]] 925000 total, 327.158118 /s
[INFO][05.10.2015 10:07:22][Thread 0008][[akka://ActorSystem1/user/Client]] 926000 total, 327.165574 /s
[INFO][05.10.2015 10:07:25][Thread 0006][[akka://ActorSystem1/user/Client]] 927000 total, 327.170581 /s
[INFO][05.10.2015 10:07:28][Thread 0006][[akka://ActorSystem1/user/Client]] 928000 total, 327.168550 /s
[INFO][05.10.2015 10:07:31][Thread 0007][[akka://ActorSystem1/user/Client]] 929000 total, 327.161907 /s
[INFO][05.10.2015 10:07:34][Thread 0021][[akka://ActorSystem1/user/Client]] 930000 total, 327.145497 /s
[INFO][05.10.2015 10:07:37][Thread 0021][[akka://ActorSystem1/user/Client]] 931000 total, 327.150053 /s
[INFO][05.10.2015 10:07:40][Thread 0008][[akka://ActorSystem1/user/Client]] 932000 total, 327.153446 /s
[INFO][05.10.2015 10:07:43][Thread 0006][[akka://ActorSystem1/user/Client]] 933000 total, 327.145933 /s
[INFO][05.10.2015 10:07:46][Thread 0008][[akka://ActorSystem1/user/Client]] 934000 total, 327.151386 /s
[INFO][05.10.2015 10:07:49][Thread 0021][[akka://ActorSystem1/user/Client]] 935000 total, 327.162778 /s

[rogeralsing]

This bug have been corrected. turned out to be a problem with our dedicated thread pool. We are now trying out some new optimizations and have an experimental transport pushing 90k messages per sec.

[vasily-kirichenko]

I retested it with the latest nuget packages and Wire serializer. On same machine it performs at about 4800 messages/second, which is a great improvement. I will test it on the same LAN as before tomorrow.

[vasily-kirichenko]

860 messages/second over LAN :(

[Horusiath]

@vasily-kirichenko I guess, that using StreamsTransport from v1.5 branch will improve things considerably.

[Petabridge-CI]

Give me some time to upgrade to the latest version of Helios here - been giving reports on it for weeks and released on Tuesday. Cool it.

Sent from my iPhone

On May 8, 2016, at 12:05 PM, Vasily Kirichenko notifications@github.com wrote:

860 messages/second over LAN :(

— You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub