[WIP] Extendability

[coss]

Earlier this year I wanted to benchmark my internal Router implementation with your benchmarking suite. I looked for a way to disable other Routers, so I don't have to go get the packages. But as everything was basicly implemented in one big file, there was no way to avoid these dependencies. So I split everything into seperate files with build tags. During this step I discovered that not all implementations work correctly (you fixed most of them as you included a simple test case), which led me to implement a new benchmark runner.

The new benchmark and test drivers allow for better extendability. Benchmarks are split into different types of benchmarks, e.g. ParameterReadWrite which benchmarks the reading and writing of a named path segment. These types are the only shared information between benchmarks and Router implementations. A Router implementor specifies the types of benchmarks a Router can run as well as other metadata, such as Router name and homepage URL, during registration. Further allows the new driver implementation to run tests for each benchmark, which uncovered more errors than the current test implementation.

Errors found by the new test driver:

• Possum. router.Simple is not sufficient for dynamic path segment routing. Further makes Possum’s current implementation of router.RegEx it impossible to benchmark ParameterReadWrite.

Also, I removed the RegEx call inside Beego's constructor - it does not interfere with the benchmarks but is still unnecessary.

Todo:

• [ ] Add CLI usage info
• [ ] Fix highlighting algorithm
• [ ] Refactor runner.go as it is quit messy
• [ ] Add RegEx based white- or blacklisting (prefix with ! to invert) for Router
• [ ] Move the driver inside the new internal package (Go 1.5)

[coss]

Here is a preview of the generated markdown and CLI output. Note that the highlighting algorithm is broken.

[julienschmidt]

Awesome! I haven't looked over all your changes yet, but the idea is great.

You can already run certain benchmarks only with e.g. go test -bench="HttpRouter|Gin", but yes, you still need all the dependencies (just go get -u github.com/julienschmidt/go-http-routing-benchmark really)

[coss]

You can already run certain benchmarks only with e.g. go test -bench="HttpRouter|Gin"

Yes, I know. I switched PCs quite a bit at that time and I tried to avoid downloading all dependencies on every machine.

While I ran the benchmarks I noticed that most of the execution time (~50%-60%) seems to come from invoking the GC to calculate the memory consumption of a Router object. Changing Fixtures to not contain a pointer may reduce the GC time significantly because the collector will not scan slices, channels, and as of 1.5 (CL 3288), maps if they don´t contain any pointers.

Note that even with the new concurrent GC in 1.5, invoking runtime.GC() still causes a STW collection.

I profiled the benchmark runner and found that the overhead in relation to the ideal execution time is not from the GC, which runs in 2ms to 5ms. The estimator in the testing package is sometimes way off. I observed the benchmark running from 1.7s to 3.1s, with some rare outlier running for 5s to 10s, for -time=1s.

[julienschmidt]

?

[coss]

Whoops, delete the wrong branch and thanks to wrong aliasing on the current machine I pushed it upstream.