add vapor 2

[tanner0101]

Hello! Vapor 2 is going to be released soon and has some significant performance improvements. I wanted to put up this PR before we release it.

This PR contains:

• Vapor 1.0 moved to vapor-1. (This can probably just be deleted)
• Vapor 2.0 added as vapor
• Concurrency reduced to 256
• Adds brew openssl linker flags to Makefile (this may need to be macOS only?)

Re: concurrency reduction: 5000 seems pretty high. I looked at TechEmpower's popular benchmark and they start at 256. It's also worth noting the majority of the web frameworks tested by TechEmpower achieve their highest performance rating at 256 concurrency. Let me know what you think about this change. Ideally this test would run subsequent rounds at increasing concurrency and take the best result.

[tanner0101]

Initial results from my computer

MacBook Pro (Retina, 15-inch, Mid 2014)
2.8 GHz Intel Core i7
16 GB 1600 MHz DDR3

Gyda:which_is_the_fastest tanner$ bin/benchmarker swift
Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
swift                     vapor                            0.350998        0.326072        0.337858
swift                     vapor-1                          0.966406        0.805082        0.894411
swift                     perfect                          0.427684        0.405412        0.414113
swift                     kitura                           2.327408        0.454126        1.046605

 -- Ranking (Language) --
1. swift (vapor) 0.3378576

 -- Ranking (Framework) --
1. vapor 0.3378576
2. perfect 0.4141132
3. vapor-1 0.894411
4. kitura 1.0466052

[tanner0101]

Finally got all of the frameworks working.

Gyda:which_is_the_fastest tanner$ bin/benchmarker swift python elixir node rust go crystal ruby
Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
ruby                      rails                            7.526120        7.420932        7.466198
ruby                      sinatra                          2.285661        2.260198        2.267690
ruby                      roda                             0.799984        0.788682        0.792571
crystal                   kemal                            0.222531        0.207475        0.218673
crystal                   router_cr                        0.161381        0.150687        0.158364
go                        echo                             0.257169        0.250905        0.254168
go                        gorilla_mux                      0.248023        0.225687        0.234917
go                        fasthttprouter                   0.182162        0.165422        0.172072
rust                      iron                             0.198380        0.195860        0.196840
rust                      nickel                           0.184839        0.167415        0.175215
rust                      rocket                           0.220559        0.201314        0.213475
node                      express                          0.933137        0.794135        0.830836
elixir                    plug                             0.263257        0.241566        0.249871
elixir                    phoenix                          0.273045        0.253058        0.258990
swift                     vapor                            0.274026        0.254649        0.263687
swift                     vapor-1                          0.737742        0.682268        0.705383
swift                     perfect                          0.301576        0.290027        0.293078
swift                     kitura                           2.265872        0.371374        0.804359
python                    sanic                            0.467176        0.453436        0.459454

 -- Ranking (Language) --
1. crystal (router_cr) 0.1583636
2. go (fasthttprouter) 0.1720718
3. rust (nickel) 0.1752152
4. elixir (plug) 0.2498706
5. swift (vapor) 0.2636872
6. python (sanic) 0.459454
7. ruby (roda) 0.7925706
8. node (express) 0.8308364

 -- Ranking (Framework) --
1. router_cr 0.1583636
2. fasthttprouter 0.1720718
3. nickel 0.1752152
4. iron 0.19684
5. rocket 0.2134748
6. kemal 0.218673
7. gorilla_mux 0.2349174
8. plug 0.2498706
9. echo 0.254168
10. phoenix 0.25899
11. vapor 0.2636872
12. perfect 0.2930782
13. sanic 0.459454
14. vapor-1 0.705383
15. roda 0.7925706
16. kitura 0.804359
17. express 0.8308364
18. sinatra 2.2676902
19. rails 7.4661982

[OvermindDL1]

How many physical cores do you have? Can you try it with at least 6 physical cores?

[tanner0101]

  Model Name:           MacBook Pro
  Model Identifier:     MacBookPro11,2
  Processor Name:       Intel Core i7
  Processor Speed:      2.8 GHz
  Number Processors:        1
  Total Number of Cores:    4

I don't have access to a Mac w/ 6 cores and I'm consistently getting Killed during the test on Ubuntu.

[tanner0101]

Hey @tbrand thanks for the feedback. I've updated the PR accordingly. It looks like some other files such as the Gemfile.lock have been changed after I ran the benchmark against all frameworks. Let me know if those files need to be removed as well.

Updated Results (with -r 5000)

Seems like my computer is just dying under the intense load. (or Rails is now the fastest framework)

Gyda:which_is_the_fastest tanner$ bin/benchmarker
Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
ruby                      rails                            5.790080        4.918701        5.481310
ruby                      sinatra                          6.075413        5.217747        5.789235
ruby                      roda                             5.893200        5.372578        5.676756
crystal                   kemal                            5.856132        5.084523        5.529858
crystal                   router_cr                        6.376590        5.505239        5.968543
go                        echo                             6.136766        5.172770        5.574827
go                        gorilla_mux                      5.894664        5.201269        5.631014
go                        fasthttprouter                   5.755849        5.371901        5.582804
rust                      iron                             5.848245        5.054677        5.527529
rust                      nickel                           5.827571        5.368569        5.580561
rust                      rocket                           6.106785        5.282349        5.648829
node                      express                          5.877766        5.537620        5.661982
elixir                    plug                             6.047065        5.145600        5.727440
elixir                    phoenix                          6.040569        5.357334        5.758597
swift                     vapor                            6.048253        5.315112        5.673224
swift                     perfect                          6.056318        5.173561        5.612143
swift                     kitura                           6.117649        5.720446        5.933327
python                    sanic                            6.134792        5.739315        5.923018

 -- Ranking (Language) -- 
1. ruby (rails) 5.4813104
2. rust (iron) 5.5275288
3. crystal (kemal) 5.529858
4. go (echo) 5.574827
5. swift (perfect) 5.6121432
6. node (express) 5.6619818
7. elixir (plug) 5.7274398
8. python (sanic) 5.9230176

 -- Ranking (Framework) -- 
1. rails 5.4813104
2. iron 5.5275288
3. kemal 5.529858
4. echo 5.574827
5. nickel 5.5805614
6. fasthttprouter 5.582804
7. perfect 5.6121432
8. gorilla_mux 5.6310136
9. rocket 5.6488292
10. express 5.6619818
11. vapor 5.673224
12. roda 5.6767558
13. plug 5.7274398
14. phoenix 5.7585966
15. sinatra 5.7892346
16. sanic 5.9230176
17. kitura 5.9333272
18. router_cr 5.9685432

[tbrand]

@tanner0101 Great PR!!!:tada:

It looks like some other files such as the Gemfile.lock have been changed

Never mind, the building always update them. So lock files mean nothing actually:smile:

Seems like my computer is just dying under the intense load.

I've faced similer situation. Then, I failed to stop previous server (The process was aliving). You can confirm it by sudo lsof -i:3000.:+1:

Finally, I'll try this and merge if it work. Thanks!!!

[tbrand]

@tanner0101

I dont' familiar with swift, could you tell me how to fix this?

crystal/which_is_the_fastest/swift/vapor/Packages/Crypto-2.0.0-beta.1/Sources/Crypto/Cipher/Cipher+Method.swift:1:8: error: could not build Objective-C module 'CTLS'
import CTLS
       ^

I've updated my packages by swift package update and run swfit build -c release.

[tanner0101]

@tbrand the CTLS package is available through Vapor's Homebrew tap and APT repo.

Check out the documentation for getting that installed here:

• macOS: https://docs.vapor.codes/2.0/getting-started/install-on-macos/
• Ubuntu: https://docs.vapor.codes/2.0/getting-started/install-on-ubuntu/

[tanner0101]

Aggh sure enough. You were right. :)

Gyda:~ tanner$ lsof -i :3000
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
server_sw 9044 tanner    3u  IPv4 0xd06613d8c4356e17      0t0  TCP *:hbci (LISTEN)
server_go 9215 tanner    3u  IPv6 0xd06613d8976c8f0f      0t0  TCP *:hbci (LISTEN)
server_go 9233 tanner    5u  IPv4 0xd06613d8c119ae17      0t0  TCP localhost:hbci (LISTEN)
Gyda:~ tanner$ 

[tbrand]

@tanner0101 Thanks! I'll try Vapor's Homebrew tomorrow:smile: (It is midnight in Japan.)

Aggh sure enough. You were right. :)

Have you tried latest benchmark.cr? Previous version contains some bugs like that, sorry...

[tbrand]

@tanner0101

<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/CoreGraphics.h"
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CoreGraphics.h:8:10: note: in file included fro
m /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CoreGraphics.h:8:
#include <CoreGraphics/CGBase.h>
         ^
<unknown>:0: error: build had 2 command failures

I still got an error... 😢

[tanner0101]

Hmm, I've never seen that error. Can you provide the full output w/ what command you are running?

[tbrand]

brew tap vapor/homebrew-tap
brew update
brew install vapor
cd swift/vapor
swift package update
swift build -c release

Then

/Users/taicsuzu/Documents/workspace/hobby/crystal/which_is_the_fastest/swift/vapor/Packages/Node-2.0.0-beta.7/Sources/Node/Fuzzy/FuzzyConverter.swift:21:40: error: same-type requirement makes generic parameter 'Element' non-generic
extension Array where Iterator.Element == FuzzyConverter.Type {
                                       ^
/Users/taicsuzu/Documents/workspace/hobby/crystal/which_is_the_fastest/swift/vapor/Packages/Node-2.0.0-beta.7/Sources/Node/Fuzzy/Dictionary+Convertible.swift:31:23: warning: cast from '_' to unrelated type 'String' always fails
            nodes[key as! String] = try Node.fuzzy.represent(value, in: context)
                  ~~~ ^   ~~~~~~
/Users/taicsuzu/Documents/workspace/hobby/crystal/which_is_the_fastest/swift/vapor/Packages/Node-2.0.0-beta.7/Sources/Node/Fuzzy/FuzzyConverter.swift:41:64: error: ambiguous reference to member '??'
            if let data = try fuzzy.represent(any, in: context ?? Node.defaultContext) {
                                                               ^~
Swift.??:33:13: note: found this candidate
public func ??<T>(optional: T?, defaultValue: @autoclosure () throws -> T) rethrows -> T

... < Too Long > ...

#define API_AVAILABLE(...) __API_AVAILABLE_GET_MACRO(__VA_ARGS__,__API_AVAILABLE4, __API_AVAILABLE3, __API_AVAILABLE2, __API_AVAILABLE1)(__VA_ARGS__)
                           ^
<unknown>:0: error: too many errors emitted, stopping now
<unknown>:0: warning: imported declaration 'dispatch_suspend' could not be mapped to 'DispatchObject.suspend(self:)'
<unknown>:0: warning: imported declaration 'dispatch_resume' could not be mapped to 'DispatchObject.resume(self:)'
<unknown>:0: warning: imported declaration 'dispatch_sync' could not be mapped to 'DispatchQueue.sync(self:execute:)'
<unknown>:0: warning: imported declaration 'dispatch_queue_create' could not be mapped to 'DispatchQueue.init(__label:attr:)'
<unknown>:0: warning: imported declaration 'dispatch_set_target_queue' could not be mapped to 'DispatchObject.setTarget(self:queue:)'
<unknown>:0: warning: imported declaration 'dispatch_group_create' could not be mapped to 'DispatchGroup.init()'
<unknown>:0: warning: imported declaration 'dispatch_group_enter' could not be mapped to 'DispatchGroup.enter(self:)'
<unknown>:0: warning: imported declaration 'dispatch_group_leave' could not be mapped to 'DispatchGroup.leave(self:)'
<unknown>:0: warning: imported declaration 'dispatch_semaphore_create' could not be mapped to 'DispatchSemaphore.init(value:)'
<unknown>:0: warning: imported declaration 'dispatch_io_create' could not be mapped to 'DispatchIO.init(__type:fd:queue:handler:)'
<unknown>:0: warning: imported declaration 'dispatch_io_create_with_path' could not be mapped to 'DispatchIO.init(__type:path:oflag:mode:queue:handler:)'
<unknown>:0: warning: imported declaration 'dispatch_io_create_with_io' could not be mapped to 'DispatchIO.init(__type:io:queue:handler:)'
<unknown>:0: warning: imported declaration 'dispatch_io_barrier' could not be mapped to 'DispatchIO.barrier(self:execute:)'
<unknown>:0: warning: imported declaration 'dispatch_io_get_descriptor' could not be mapped to 'getter:DispatchIO.fileDescriptor(self:)'
<unknown>:0: warning: imported declaration 'dispatch_io_set_high_water' could not be mapped to 'DispatchIO.setLimit(self:highWater:)'
<unknown>:0: warning: imported declaration 'dispatch_io_set_low_water' could not be mapped to 'DispatchIO.setLimit(self:lowWater:)'
<unknown>:0: error: could not build Objective-C module 'CoreFoundation'
<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/CoreGraphics.h"
        ^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CoreGraphics.h:8:10: note: in file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/CoreGraphics.framework/Headers/CoreGraphics.h:8:
#include <CoreGraphics/CGBase.h>
         ^
<unknown>:0: error: build had 2 command failures

[tbrand]

May be my swift version is too old?

Apple Swift version 3.0.2 (swiftlang-800.0.63 clang-800.0.42.1)
Target: x86_64-apple-macosx10.9

[tanner0101]

May be my swift version is too old?

Aha, yes that's the issue. You need 3.1+ for Vapor 2.

[tbrand]

Will update it.

[r6m]

Waiting to see the results in the photo. 👍

[tbrand]

@rezam90 Very soon.

[tbrand]

I have to update mac os, haha

[tbrand]

This commit has been included new release:tada:( https://github.com/tbrand/which_is_the_fastest/releases/tag/v1.1.2 ) Thanks again and waiting for your new commit! :smile: