Rack and Rails Grape numbers

I read your analysis, which is very nice, but I found it odd that the Rails performance numbers were so high relative to standalone Rack. So I ran the benchmarks myself and I think the Rails numbers you list are incorrect.

Here's my Rack run:

ab -c 10 -n 1000 http://localhost:9292/projects\?key\=61c2339c1bc92bc48120b55513cd568b
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        2211 bytes

Concurrency Level:      10
Time taken for tests:   14.675 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2303000 bytes
HTML transferred:       2211000 bytes
Requests per second:    68.14 [#/sec] (mean)
Time per request:       146.754 [ms] (mean)
Time per request:       14.675 [ms] (mean, across all concurrent requests)
Transfer rate:          153.25 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    46  141 196.1    131    5051
Waiting:       39  134 195.8    121    5042
Total:         47  141 196.1    131    5051

Percentage of the requests served within a certain time (ms)
  50%    131
  66%    148
  75%    153
  80%    157
  90%    168
  95%    177
  98%    192
  99%    279
 100%   5051 (longest request)

Here's my first Rails run, which indeed shows very good performance:

ab -c 10 -n 1000 http://localhost:9292/projects\?key\=61c2339c1bc92bc48120b55513cd568b  
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        9 bytes

Concurrency Level:      10
Time taken for tests:   1.886 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Non-2xx responses:      1000
Total transferred:      199000 bytes
HTML transferred:       9000 bytes
Requests per second:    530.35 [#/sec] (mean)
Time per request:       18.855 [ms] (mean)
Time per request:       1.886 [ms] (mean, across all concurrent requests)
Transfer rate:          103.07 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:     9   19  42.1     12     437
Waiting:        9   18  42.1     11     437
Total:          9   19  42.1     12     437

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     13
  75%     13
  80%     15
  90%     29
  95%     31
  98%     40
  99%    426
 100%    437 (longest request)

However, I think these are all 404s. There is a setting in Grape to prefix your API URLs, and it's set to 'api', so here's another run with that prefix in place:

ab -c 10 -n 1000 http://localhost:9292/api/projects\?key\=61c2339c1bc92bc48120b55513cd568b
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests

Server Software:
Server Hostname:        localhost
Server Port:            9292

Document Path:          /api/projects?key=61c2339c1bc92bc48120b55513cd568b
Document Length:        2211 bytes

Concurrency Level:      10
Time taken for tests:   17.681 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      2492000 bytes
HTML transferred:       2211000 bytes
Requests per second:    56.56 [#/sec] (mean)
Time per request:       176.812 [ms] (mean)
Time per request:       17.681 [ms] (mean, across all concurrent requests)
Transfer rate:          137.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:    41  176  12.4    172     217
Waiting:       41  176  12.4    172     217
Total:         41  176  12.3    172     218

Percentage of the requests served within a certain time (ms)
  50%    172
  66%    183
  75%    185
  80%    186
  90%    189
  95%    192
  98%    197
  99%    204
 100%    218 (longest request)

The numbers are a little more down-to-earth here, so I'm guessing you encountered the same problem in your figures.

This doesn't really affect your overall findings, just thought I'd point it out as I found it confusing based on my Rack/Rails experience.

steventen / grape-vs-martini

Rack and Rails Grape numbers #2