search

EcoG-io / josev

The community version of Josev - An Operating System for the V2G charging Stations
Apache License 2.0
125 stars 18 forks source link

Performance Results #8

Open danielpeintner opened 1 year ago

danielpeintner commented 1 year ago

Hi,

I noticed that you provide some performance numbers (Rust codec vs. EXIficient). Do you have some description about the test setup?

The reason why I am asking. Does the EXIficient time include the schema loading phase as well? I think this would not really be a fair comparison. Instead the schema should be loaded before the time counts since this in a real world application needs to be done only once.

Note1: in general it would be better to compare the time measurements against OpenV2G instead of the generic EXIficient codec. Note2: The statement "REXI - A Rust version of the EXI en/decoder, which is at least x100 faster than the available open source solutions" would need to include OpenV2G (and others?) as well.

shalinnijel2 commented 1 year ago

Hi Daniel, Yes, the grammar is loaded only once in this case. But I agree, comparing against OpenV2G would have been fairer. We hadn't done that at the time though - it is something we intend to do.

danielpeintner commented 1 year ago

Yes, the grammar is loaded only once in this case.

I think the time for loading a grammar should not be counted at all for the time comparison.

But I agree, comparing against OpenV2G would have been fairer. We hadn't done that at the time though - it is something we intend to do.

That makes sense since your claim "at least x100 faster than the available open source solutions" is simply not true.

shalinnijel2 commented 1 year ago

This comparison was done only for 15118-20 messages. Not sure if open source OpenV2G supports -20 messages though.

gerf-autoshart commented 1 year ago

I would be interested in more details in the methodology of this testing. How many iterations of each message is in the benchmark? What are the deviations? Others have mentioned OpenV2G as being a more fair comparison and I would agree.

I think it might also be a fair comparison to test multiple architectures as many implementors are probably going to target embedded systems with less capable processors than an i5. For example, a Raspberry Pi 4b, NXP IMX8, and various STM variants might shed some light into the true performance that the Rust EXI encoder/decoder offers.

Other testing variables might include memory consumption of the decoder/encoder in question. If the Rust solution is fast but allocates 500MB of memory then that limits its application in the field as does the size of the generated binary. Further comparisons might include memory allocations, if any and freeing activity.