Global cache flag: Should have a two runs with cache enabled and cache disabled.

[agentgt]

There is a fundamental problem with this benchmark in that it is testing the compiling AND execution on every request for many of the libraries. Some of them that isn't the case which is also rather unfair (For example JSP only compiles once so long as the file does not change where as Handlebars compiles on every request when setCache(false)).

There are two phases in most templating languages.

• Compiling - when the templates are turned into ASTs or bytecode etc
• Executing - a model is then passed to the compiled template and run

In a production setting compiling only happens once.

Thus most folks are truly only concerned with how fast the templates execute and not compile and then execute.

Furthermore I believe there are templating languages that are probably doing some internal caching similar to how JSP is only compiling once.

Anyway my recommendation is to make some global property that enables cache for all the template engines.

Then you can test both scenarios.

[jreijn]

Hi Adam,

Thanks for your valuable insights . You are completely right. I did not see that 8 years ago when I created this repo. A lot of engines have been added in the mean time with varying levels of configuration.This project is up for a refresh. Perhaps a better benchmark would be to test the engines with jmh instead of the integration with Spring. That however was the original intent to compare the engines in a Spring Context.

I like the suggestion to add a global caching flag. I'll see if I can spent some time over the weekend to give it a go.

Jeroen

[Vest]

Hello all,

I disabled the caching a long time ago, because cached pages produced same or similar results. When I disabled the caching, I profiled the frameworks and wrote in notes why some of them are slow in multithreaded tests. I think, it is helpful for people, who is planning to use one of the tested frameworks.

There is another project, similar to the current one, where the compilation within these engines is tested with JMH. You can check that project, if you can find it. There you can get different results, than in this repo.

But if you ask my opinion, I don’t want to enable the caching in this project. There is no need to test static pages.

[agentgt]

But if you ask my opinion, I don’t want to enable the caching in this project. There is no need to test static pages.

It depends on what the template library caches. I don't think most of them have idempotent cache like that. Besides you can just make the model have some dynamic data on every request. I'll go check what the controller does and add random data.

Anyway I did some updating to JDK 17 and latest Spring Boot.

From the runtests.sh with cache disabled

jsp Time taken for tests: 0.336 seconds
freemarker Time taken for tests: 0.389 seconds
thymeleaf Time taken for tests: 0.717 seconds
mustache Time taken for tests: 0.475 seconds
handlebars Time taken for tests: 1.025 seconds
htmlFlow Time taken for tests: 0.268 seconds
trimou Time taken for tests: 0.269 seconds
rocker Time taken for tests: 0.278 seconds
ickenham Time taken for tests: 0.573 seconds

With cache enabled as best as I could tell (actually handlebars could be improved more as it has a reload flag on its template cache that avoids checking for file changes.. there are probably other ones that do this as well).

jsp Time taken for tests: 0.341 seconds
freemarker Time taken for tests: 0.423 seconds
thymeleaf Time taken for tests: 0.629 seconds
mustache Time taken for tests: 0.459 seconds
handlebars Time taken for tests: 0.448 seconds
htmlFlow Time taken for tests: 0.325 seconds
trimou Time taken for tests: 0.326 seconds
rocker Time taken for tests: 0.199 seconds
ickenham Time taken for tests: 0.377 seconds

Which is exactly the results I expected. I expected Rocker to trash others because it uses zero byte copy and thread locals. I expected the interpreted ones to be similar in speed (mustache, handlebars, thymeleaf) and the compiles ones similar (JSP, htmlFlow, rocker etc). I'm surprised htmlFlow didn't fair better as I thought it did the same tricks rocker did.

The above was done on a lower power atom processor server.... so basically templating speed... there are probably other things to optimize.

[Vest]

It works fine for situations, when the page stays unchanged during the test. But if you plan to render dynamic pages (e.g. a news feed), the caching will not be fully utilized (because different frameworks can cache either fragments of pages, or maybe even the final result only). It means that partially you might face the scenario, where you need the fastest possible template engine, plus an efficient caching can be achieved with Nginx/Apache/something else.

I haven’t repeated your tests (with enabled cache) on my machine, because when I did this a long time ago, I found out that some engines use caching even if you disable this explicitly, and I found this unfair.

Apache tests are very inaccurate, because of many factors, so you shouldn’t rely on them. If you think that a template engine is slow without the enabled caching, probably it has other advantages, that is why people will choose it. Additional performance boost can be added either via CDN, or caching web servers, or anything else.

I will close this issue, but not because I am rude. I have honestly read your posts, thought about them and so on. But because I think that the goal of this repo was to show: different engines combined together within Spring ecosystem. And additional benefit was to measure raw template processing speed, without any hidden tweaks such as caching.

[agentgt]

Well then you should remove the benchmarks from the readme because its incredibly disingenuous and no doubt will lead people astray that somehow Handlebars is 5x slower than Thymeleaf when the reality is it usually executes faster.

On the first load... On the first hit. JSP is the slowest of all because it has to compile (The examples I showed were after warming up and running several times). Why doesn't the benchmark show this?

Thus the safest thing is to remove the benchmarks.

I will close this issue, but not because I am rude. I have honestly read your posts, thought about them and so on. But because I think that the goal of this repo was to show: different engines combined together within Spring ecosystem. And additional benefit was to measure raw template processing speed, without any hidden tweaks such as caching.

It is clearly its not doing that (showing different engines) as well as it should because most of the template engines are not tuned correctly nor is it using their full capabilities or the latest JDK or the latest Spring Boot or Spring or the template engines.

I'm going to be a little rude. I have some doubts on the expertise.

And additional benefit was to measure raw template processing speed, without any hidden tweaks such as caching

The caching of template engines is saving the precompiling and obviously leveraging static strings. If you can please point me to an engine that some how has idempotent caching like CDN as in same model or same request -> use cache -> output. As far as I know none of them do that. They always execute the engine as they don't know how the model has changed.

Anyway at least do the the honest and remove the benchmarks or at least point to the JMH benchmarks.