Issues with Vert.x setup

[purplefox]

I just looked at the Vert.x setup... At first glance there appear to be some issues, including:

1. The command line used to start Vert.x needs to include -instances where n is number of cores on the machine, or it will be crippled to one thread.
2. I can't see anywhere where the MongoDB module is being started. How is this being done? Shouldn't there be a call to vertx.deployModule() in the script?

Other than that there seem to be more general issues with the testing approach, unless I am missing something including:

1. The JSON serialization test seems to be testing the performance of the Jackson JSON tool (not part of Vert.x)
2. The database tests seem to be testing the performance of MongoDB.

[bhauer]

Thanks @purplefox!

1. Is this necessary in addition to the configuration of the number of Verticles and Mongo Persistors in App.groovy? https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/vertx/App.groovy
2. In the same App.groovy file, deployModule('vertx.mongo-persistor-v1.2.1', persistorConf, 8). Is this doing it wrong or do I misunderstand? (Note, I didn't write the Vert.x test, so I apologize if my response is way off the mark.)

As for the testing approach:

1. Yes, it's a test of (in this case) what we consider a platform and a best-of-breed JSON serializer for Java. We do not presently have plaintext response tests, but I just modified #133 to reflect the need to add such a test in the future. I've not previously made it a priority because when we started this project, we quickly observed that it's effortless to run into the network bottleneck of our gigabit Ethernet with just plaintext. A resulting chart will show a bunch of roughly-equal bars, with small variances due to response header differences. Incidentally, that's exactly what we saw for high-performance platforms and frameworks with our JSON test; although perhaps a little less dramatically.
2. Yes, the database tests indeed are quite limited by the performance of the databases themselves. But also the database connection pool, driver, and ORM (where available). The results show that it is possible to run into the performance barrier of the database server or network--the barrier where it's simply not possible to send any more queries to the database server and expect responses. But interestingly, not a whole lot of frameworks get to that barrier because their connection pool, driver, or ORM is also a bottleneck.

Thanks for taking the time to review this. We especially appreciate it when the framework/platform authors themselves can help us improve the tests.

[purplefox]

I must be going blind. I didn't even see the App.groovy, just saw the WebServer.java! Apologies.

[purplefox]

Regarding the general points

1. Get a 10 Gib/s ethernet! Otherwise the results just tell me that Jackson is quite a fast JSON serializer (but maybe not quite as fast as other ones)
2. I think the database used must be the same across all runs, otherwise I'm not sure how any of the results can be compared. I notice that some frameworks don't run the database benchmarks. I think I would like to see the database benchmarks for Vert.x removed too.

[bhauer]

Ha! We'd be happy to have 10 GBE but it's not in the cards, so to speak.

Several tests do not yet implement the database tests. But as for dropping the Vert.x database test, I'll discuss this with my colleagues since that's an unprecedented request and I'd like some more opinions about that. The request certainly carries weight coming from the author of a framework, but I'm not sure I agree with removing data for that reason alone. I'll follow up later.

Incidentally, you can filter the results to view only platforms using Mongo:

http://www.techempower.com/benchmarks/#section=data-r5&d=1

[purplefox]

Don't worry about removing the results, after all it's a free(ish) world and you can publish whatever you want ;)

However, I do still have real concerns about the benchmarks and whether they're actually benchmarking the system under test, and whether they really reflect what people would actually do with the framework.

[purplefox]

Haven't tried this, but you might get somewhat better perf in the vert.x database tests by increasing the number of mongodb module instances you deploy from 8 to somewhere around 20 or 30.

[bhauer]

@purplefox Thanks for the recommendation and for being open to seeing the data live on despite your misgivings with the project as a whole.

The approach we are taking exercises building blocks of web applications in admittedly unrealistic scenarios of repeated requests to do the same trivial operations. As I have written elsewhere, it means we receive criticism from fans of full-stack frameworks for not testing a realistic fully-conceived web application (e.g., a blog or a web store) and criticism from platform developers for needing to snap on best-of-breed components such as JSON serializers or database drivers in order to fulfill our quite-small requirements.

I believe for all the caveats, it remains valuable to know the rough performance characteristics of the building blocks one is likely to use with a particular selection, whether they come as part of the package "off the shelf" or must be selected from a suite of bolt-on options. Where there is not yet consensus on the preferred building blocks, we have been happy to see the addition of more variations (e.g., Play+Anorm and Play+JPA).

With time, we will gradually add more tests to exercise more pieces, and some of those tests may make progress toward the more fully-conceived application test desired by full-stack fans.

[bhauer]

@pfalls1 If you find the time prior to Round 6, can you give a quick test to the recommendation of 20-30 mongodb module instances?

[bhauer]

I am just cleaning up old issues. I believe this issue is superseded by Round 7 vert.x work.