rust-sailfish / sailfish

Simple, small, and extremely fast template engine for Rust
https://rust-sailfish.github.io/sailfish/
MIT License
801 stars 54 forks source link

I keep winning. Make an article like the one you did explaining why. #48

Closed zzau13 closed 3 years ago

zzau13 commented 3 years ago

https://www.techempower.com/benchmarks/#section=test&runid=646af5ea-0fae-448d-b77d-bc27b13d73e5 While in your benchmarks and these https://github.com/djc/template-benchmarks-rs , they said otherwise. It would be very honorable to write an article explaining why to the community, which I think has the right to know why you lie in this way as reflected in the results of techempower.

It really is outrageous and an insult to me that you had put those results.

And remember math doesn't fail, don't bet against it.

alleonhardt commented 3 years ago

I hope you have read the benchmark code techempower used for this benchmark. The template they tested is this one:

<!DOCTYPE html>
<html>
  <head><title>Fortunes</title></head>
  <body>
    <table>
      <tr><th>id</th><th>message</th></tr>
      <% for item in items { %><tr><td><%= item.id %></td><td><%= &*item.message %></td></tr><% } %>
    </table>
  </body>

There are also only 12 entries inside the database table they use to instantiate the template. The second RUST ONLY benchmark you bring up inserts 100 entries. I dont know which one is faster and I also dont want to judge, but math is not comparing two completely different benchmarks with each other. If you care that much about speed just benchmark the libraries in your use case and use the one that's fastest for your particular case.

The second point is the benchmark you quote, tests all kind of framworks written in Rust but also written in C, C++ etc. There are only two rust frameworks that perform better than sailfish and both only by a small margin.

I think it is very rude to insult someone who created a great library which you can use for free in his/her free time.

d4h0 commented 3 years ago

I think it is very rude to insult someone who created a great library which you can use for free in his/her free time.

I agree.

Besides, @botika, at least you could include enough information, so people know what you are talking about.

Especially, if you write something like:

It would be very honorable to write an article explaining why to the community, which I think has the right to know why you lie in this way as reflected in the results of techempower.

zzau13 commented 3 years ago

@ShadowItaly Look at the benchmark implementations, the commit history is a lot of fun. The top two in rust are implemented with yarte. I have not insulted anyone, and it comes from here https://github.com/Kogia-sima/sailfish/issues/19 because according to him I am the enemy. And how do you know that the library is great if you don't know what the Techempower frameworks are implemented in? Have you looked at the code? I do, man. And based on the benchmark results, I can say that he have reinvented the wheel to no avail, as the benchmarks show. In addition, it does not follow the SOLID principles, more exactly the O, look for what the O is that will be good for you.

If you implement yarte well in big table or teams benchmark, after knowing the degree program, you will see that the result is the same as techempower.

What surprises me the most is that without knowing what Techempower is implemented in, you have the courage to comment on it, it is truly enviable.

@d4h0 For the same article that I put above. I recommend reading the history of issues and commits.

Basically because for a long time these two benchmarks have given wrong results saying that yarte was slower, when yarte is the fastest. And more exactly because of the article that the enemy calls me. And the uncanny resemblance of the escape algorithm.

Kogia-sima commented 3 years ago

@ShadowItaly @d4h0 Thanks for answering.

@botika At first please answer both indications from @ShadowItaly. I agree that those two benchmarks are totally different. Taking the top on some specific benchmark doesn't mean it always tops in another benchmarks.

Also I'd like to add three points; First in general benchmark results depend on CPU family, micro architecture, Operating System, compiler, and even which of the Standard C implementation is used. For example, the results you paste were collected on the self-hosted environment (Citrine). Critine is hosted on Ubuntu 18.04.3 4.15.0-88-generic, which uses older version of libc implementation. If you REALLY want feasible benchmark results, why don't you take the benchmark on the latest operating system by yourself?

Second, even in the techempower benchmark, you take a shortcut to top the benchmark. I checked your implementation and found that you pre-allocates buffer by constant value, which allows compiler to skip some capacity checks or even pipelining some operations. I measured the performance with constant pre-allocation and black-boxed pre-allocation (simply wraps the pre-allocation size by black_box function).

techempower   518.59 ns  [517.75 ns, 519.42 ns]
techempower   536.68 ns  [535.32 ns, 538.05 ns]
Source code * constant pre-allocation ```rust #[inline(never)] fn techempower(b: &mut Bencher) { let items = vec![ Fortune { id: 0, message: "Additional fortune added at request time.", }, ... (snip) ... Fortune { id: 12, message: "フレームワークのベンチマーク", }, ]; let items = black_box(items); b.iter(|| { let mut buf = Buffer::with_capacity(2048); Techempower { items: &items }.render_once_to(&mut buf).unwrap(); buf.into_string() }); } ``` * `black_box`ed pre-allocation ```rust #[inline(never)] fn techempower(b: &mut Bencher) { let items = vec![ Fortune { id: 0, message: "Additional fortune added at request time.", }, ... (snip) ... Fortune { id: 12, message: "フレームワークのベンチマーク", }, ]; let items = black_box(items); b.iter(|| { let mut buf = Buffer::with_capacity(black_box(2048)); Techempower { items: &items }.render_once_to(&mut buf).unwrap(); buf.into_string() }); } ```

Benchmark environment

hardware/software version
CPU model name Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
micro architecture Sky lake
operating system Lubuntu 20.04.1 LTS
Rust compiler 1.48.0 stable
Standard C implementsion glibc 2.31.0

You see that constant pre-allocation gets 3-4% performance improvements, although it is not an applicable idea in most of real-world applications. So even if yarte benchmark is faster than sailfish, that's not because yarte is faster, rather due to an impractical idea used for benchmarking yarte.

Third, taking top two times is not statistically enough to prove that yarte is faster. You need to experiment t-test or some another comparison tests to prove that. btw ntex-sailfish took top on another trial:

https://github.com/TechEmpower/FrameworkBenchmarks/issues/5709#issuecomment-754576921

Also I executed the benchmark on my local environment, and ntex-sailfish got better performance in all cases.

benchmark results in local environment target commit: https://github.com/TechEmpower/FrameworkBenchmarks/commit/0ae216eb34f2ccf02089da2f3057b6df02c3177f * ntex-db ```json { "ntex-db": [ { "latencyAvg": "690.18us", "latencyMax": "20.10ms", "latencyStdev": "391.70us", "totalRequests": 353361, "startTime": 1611471969, "endTime": 1611471984 }, { "latencyAvg": "1.17ms", "latencyMax": "20.54ms", "latencyStdev": "713.70us", "totalRequests": 420216, "startTime": 1611471986, "endTime": 1611472001 }, { "latencyAvg": "2.06ms", "latencyMax": "30.35ms", "latencyStdev": "1.41ms", "totalRequests": 486686, "startTime": 1611472003, "endTime": 1611472018 }, { "latencyAvg": "3.67ms", "latencyMax": "37.13ms", "latencyStdev": "2.55ms", "totalRequests": 544323, "startTime": 1611472020, "endTime": 1611472035 }, { "latencyAvg": "6.55ms", "latencyMax": "38.00ms", "latencyStdev": "3.81ms", "totalRequests": 588439, "startTime": 1611472037, "endTime": 1611472052 }, { "latencyAvg": "12.08ms", "latencyMax": "54.23ms", "latencyStdev": "5.33ms", "totalRequests": 624051, "startTime": 1611472054, "endTime": 1611472070 } ] } ``` * ntex-sailfish ```json { "ntex-sailfish": [ { "latencyAvg": "685.81us", "latencyMax": "16.02ms", "latencyStdev": "376.82us", "totalRequests": 355537, "startTime": 1611472260, "endTime": 1611472275 }, { "latencyAvg": "1.16ms", "latencyMax": "17.76ms", "latencyStdev": "712.35us", "totalRequests": 425044, "startTime": 1611472277, "endTime": 1611472292 }, { "latencyAvg": "2.01ms", "latencyMax": "28.31ms", "latencyStdev": "1.39ms", "totalRequests": 497609, "startTime": 1611472294, "endTime": 1611472309 }, { "latencyAvg": "3.61ms", "latencyMax": "32.31ms", "latencyStdev": "2.48ms", "totalRequests": 553405, "startTime": 1611472311, "endTime": 1611472326 }, { "latencyAvg": "6.42ms", "latencyMax": "38.72ms", "latencyStdev": "3.72ms", "totalRequests": 598947, "startTime": 1611472328, "endTime": 1611472344 }, { "latencyAvg": "11.90ms", "latencyMax": "51.47ms", "latencyStdev": "5.35ms", "totalRequests": 636665, "startTime": 1611472346, "endTime": 1611472361 } ] } ```

And based on the benchmark results, I can say that he have reinvented the wheel to no avail, as the benchmarks show.

Performance is just one aspect of the software. For example I got ~10% reduction in code bloats by re-implementing buffer from scratch. Also it allows further improvement using branch prediction (although it is only available in nightly compiler).

If you implement yarte well in big table or teams benchmark, after knowing the degree program, you will see that the result is the same as techempower.

If you mention that, why don't you measure the performance by yourself? As I told you many times in other issues, if you have any complaint about this crate, what you should do is to provide a concrete example or some pertinent evidence with it.

And more exactly because of the article that the enemy calls me.

If you are saying about this article, it is not intended to demean other libraries or to show that any particular library is better than others. I added a note in the article in order to avoid misconception.

And the uncanny resemblance of the escape algorithm.

It seems you're not still convinced of what I said in this issue. If you have any question, please clarify which part of my comments you didn't understand. I'll explain about that in more detail.

Kogia-sima commented 3 years ago

Btw recently my lab-mates told me that yarte contains a direct copy of sailfish source code.

https://github.com/botika/yarte/commit/ceb3a55ac445d87092ccf81749bddb8506629575

MIT license does not allow direct copy and modifiction of the source code without an acknowledgement to the original software. I have no intention to send DMCA due to such a small copy, but I'll be happy if you add a proper copyright notation in your repo :)

zzau13 commented 3 years ago

https://github.com/botika/yarte/blob/master/yarte_helpers/src/helpers/integers/mod.rs#L1 Yes of course, you did this too: https://github.com/miloyip/itoa-benchmark are you @miloyip? Why don't you look like him

Kogia-sima commented 3 years ago

"you did this too" is a common style of red herring. I already moved the implementation to https://github.com/Kogia-sima/itoap, which has proper copyright notice in both README.md and LICENSE file.

The problem is that yarte doesn't include acknowledgement despite direct copy-and-pastes from the sailfish source code.

zzau13 commented 3 years ago

They are tests that test if 2 strings are equal, I don't think an assert can be licensed,

It's like I license a test, assert_eq! (foo, bar). It doesn't make a lot of sense.

I'm not going to give you credit for a test, if it was anything that affected behavior or performance, yes.

The iota implementation is from @miloyip, which I mention in the code. Also mine uses simd and yours doesn't. I do not understand

zzau13 commented 3 years ago

Well, it doesn't matter. Do not make the article on the interesting topic of re-inventing the wheel. I close but before doing it again, look at this PR, rent a good server, start it up and enjoy.

Kogia-sima commented 3 years ago

They are tests that test if 2 strings are equal, I don't think an assert can be licensed,

Directly copying the original source code and modifying based on the copy is forbidden by MIT License. Simple string comparison doesn't prove your innocent.

Also even in the string comparison, yarte still have copy from original source code (e.g. trait definitions).

Furthermore, whether or not a test is protected by copyright varies from country to country. At least in Japan, source code for testing program constitute the software itself, and thus copyrightable (著作権法第10条第3項). What is the basis for your statement?

Also mine uses simd and yours doesn't

We are not talking about algorithms.

zzau13 commented 3 years ago

Well now nothing will remove the tests.

Congratulations on the tests, the best you have ever done.

It's great! I've had a great time with you but you have to go to work. Honestly the best since the IRC fashion passed, I haven't had such a good time for a long time.

And remember I win. :smiling_face_with_three_hearts: :smiling_face_with_three_hearts:

Kogia-sima commented 3 years ago

@d4h0 @ShadowItaly Sorry for the spam. I decided to block @botika due to copyright issue.

d4h0 commented 3 years ago

@Kogia-sima: You mean, you blocked @botika from the repo? I would have done the same (seeing the issues/comments he has posted here).

Kogia-sima commented 3 years ago

@d4h0 Yes, he is blocked from all my repositories.