Strange timeouts when implementing simple Proxy

[jedws]

I wrote a simple proxy to test the scalability of spray for an eval, and it isn't very promising. It tends to die around 8230 connections into an 'ab' test:

extra:test jwesleysmith$ ab -n 10000 -c 50 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
apr_poll: The timeout specified has expired (70007)
Total of 8238 requests completed

Yet when I run it against the original proxied page it returns fine.

The code is from the examples:

object Main extends App {
  Supervisor(
    SupervisorConfig(
      OneForOneStrategy(List(classOf[Exception]), 3, 100),
      List(
        Supervise(Actor.actorOf(new RequestActor("test-endpoint", new Proxy())), Permanent),
        Supervise(Actor.actorOf(new HttpServer()), Permanent),
        Supervise(Actor.actorOf(new HttpClient()), Permanent)
      )
    )
  )
}

class RequestActor(id: String, handle: (RequestContext => Unit))
  extends Actor {
  self.id = id

  protected def receive = {
    case ctx: RequestContext => handle(ctx)
    case Timeout(method, uri, _, _, _, complete) => complete {
      HttpResponse(status = 500).
        withBody("The " + method + " request to '" + uri + "' has timed out...")
    }
  }
}

class Proxy extends (RequestContext => Unit) {
  val log = LoggerFactory.getLogger(getClass)

  def apply(ctx: RequestContext) {
    HttpDialog("server-test", 8081).
      send(HttpRequest(method = GET, uri = "/")).
      end.
      onComplete {
        _.value match {
          case Some(Right(response)) => {
              ctx.responder.complete(new HttpResponse(body = response.body))
          }
          case error                 => log.error("Error: {}", error)
        }
      }
  }
}

No errors are being reported.

[sirthias]

What spray-can version are you running on what OS?

This is what I get when running ab under OSX against the server-example from the current spray-can/0.9.2-SNAPSHOT:

~ > abx -k -n 100000 -c 100 127.0.0.1:8080/                                                                                              ruby-1.9.3
This is ApacheBench, Version 2.3 <$Revision: 1139530 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        spray-can/0.9.2-SNAPSHOT
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /
Document Length:        464 bytes

Concurrency Level:      100
Time taken for tests:   1.899 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    100000
Total transferred:      62400000 bytes
HTML transferred:       46400000 bytes
Requests per second:    52659.88 [#/sec] (mean)
Time per request:       1.899 [ms] (mean)
Time per request:       0.019 [ms] (mean, across all concurrent requests)
Transfer rate:          32089.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0  13.2      0    1012
Processing:     1    2   0.7      2      31
Waiting:        1    1   0.7      1      31
Total:          1    2  13.3      2    1023

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      2
  95%      3
  98%      3
  99%      4
 100%   1023 (longest request)

Apparently the problem you are seeing lies on the client side. We recently fixed two memory leaks in HttpClient, maybe you could try again with the latest SNAPSHOT?

[jedws]

was running 0.9.0 on MacOSX didn't realise you had removed the Scala version from 0.9.1 and hadn't seen it. Added the ScalaTools Snapshot repo and 0.9.2-SNAPSHOT was able to reproduce the same behaviour.

[sirthias]

I assume you are running OSX 10.6 (Snow Leopard), right? On Lion the version of ab from your stdout dump doesn't properly run unpatched...

[jedws]

turns out it was just socket exhaustion