Performance and Packaging Benchmarking and Integration Tests

[vkresch]

Describe the feature

In accordance to the WP Performance and Packaging we need benchmarks to decide if certain features are worth considering. Like flatbuffers (https://github.com/OpenSimulationInterface/open-simulation-interface/pull/427), delta encodings or type optimization. Furthermore OSI does not have any unit and integration tests (only format/linter checker and smoke test). Developers do not currently know when they add new features to OSI if it adds overhead and slows it down or speeds it up.

Describe the solution you would like

A solution to that would be implementing a base integration test pipeline into the CI which decodes and encodes differtent type of OSI files and writes them to disk. See the graph below:

We would than measure the time of the green highlighted part of the pipeline and compare them accordingly. Also the size of the file can be measured.

Todos would be (for the base pipeline):

• Implement a open source scenario generator (python scenario generator for generating for example traces with 1000 moving objects/1000 stationary objects ...). Here is a simple way of generating scenarios which would need to be extended in a user friendly way. The scenario generator would be also a good way to show how to generate scenarios which can be visualized by OSI visualizer for new user. See this issue (https://github.com/OpenSimulationInterface/open-simulation-interface/issues/398).
• Implement a base model for a simple transformation to the host vehicle in python or/and C++ (at the start a simple mapping could also be done). It's important to keep the model implementation as generic as possible so that it can be reused in each pipeline hence the encoding and decoding should be modified according to the piepline type.
• Add the pipeline to the current CI

Describe alternatives you have considered

Another way of testing would not only be local but through a communication protocol like TCP and then measure the time. But it would be not easy to implement it in the current CI.

Describe the backwards compatibility

The backwards compatibility would also be tested by the integration tests and can be thus detected. This feature would not break anything since it is a benchmark test.

[vkresch]

@pmai let me know your thoughts if it would make sense to implement a benchmark this way in accordance to the WP

[doganulus]

Where can I find discussion/documentation regarding deltaencode feature (if exists)? I think that would be a great feature beyond optimization but probably not backwards-compatible so a discussion would be required.

[vkresch]

@doganulus currently there is no documentation about it. But the idea is to encode just fields which change in an OSI trace file over time. Hence would reduce the file size (because no redundant fields are copied over to each single message like lane boundaries) and better the performance of any simulation tool (osi-visualizer) which uses this kind of delta encoding. To make sure it's backward compatible I would leave this option of encoding as optional or would implement a converter between those types of encoders at first.

You are welcome to share if you have any specific ideas, use cases, suggestions and improvements on how to improve the performance of OSI.

[vkresch]

@engelben like @pmai said, maybe we would need for the scenario generation tool a separate repository like osi-generator in the organization so that we can distinguish between the OSI standard and the tools which use the standard. I'm also thinking of maybe creating an osi-benchmark repo to collect use cases which need to be benchmarked. The osi-benchmark repo would run a travis CI with all current known usecase and time them. Let me know what you think.

[engelben]

@vkresch great suggestion. I like the idea of having an automated generator in the repo for example traces.

What is the reasoning behind keeping the encoding from .osi to the new formats (flatbuffer/deltaencoding) out of the benchmark?

In the last ccb meeting we discussed naming conventions for the repositories to clarify exactly your point. We did not finish this discussion, but if we have the main repo as osi-specification then your suggestions for naming the other repos. make sense.

[vkresch]

@engelben keeping flatbuffer/deltaosi encodings out of the benchmark is because these are only converters. With the benchmarking we want to proof that deltaosi/flatbuffers in itself (without protobuf osi) are improving the performance. Furthermore it's also possible to generate directly flatbuffer files without any conversion.

[engelben]

@vkresch thanks for the clarification! But is this conversion process not also part of the comparison? It is the whole process that needs to be more performant is it not?

[pmai]

@engelben the comparison is intended to show that a world where osi is flatbuffer-based (i.e. there is no protobufs around) is faster (and by how much) than the current world; so the conversion itself should not be part of the benchmark, as @vkresch says.

One thing that would be even nicer, since it would also capture the effort to create the SensorView, would be to convert the Scenario Generator to directly emit flatbuffers; then the benchmark would include both versions of the generator as well - even more ideally there would also be a consumer of the SensorData for both proto and flat versions. However all of this looks like a lot of effort, which might not be worth it. The more narrow SV decode + SD encode benchmark should already be sufficient to gauge the order of magnitude of improvements.

[pmai]

@pmai let me know your thoughts if it would make sense to implement a benchmark this way in accordance to the WP

Sorry for the delayed response, too many distractions... Yes, a benchmark of this sort would be relevant for the WP. One alternative to a new generator would be to enhance the OSMPDummySource FMU to provide more content - which could also be ported to flatbuffers directly... If we switch, we would need to port the examples anyway...

[engelben]

ah ok thanks @pmai and @vkresch for the breakdown!

[vkresch]

@pmai ok thanks for the feedback! I will start for now implementing the base pipeline and will open a PR in the near future :).

[kmeids]

@pmai @vkresch I am currently using the following workflow (please see picture below) for Radar-Model simulation. Currently I am exporting the "osi_dynamic_sv.pb.txt" for the entire scenario from our own driving simulator and only considering the moving objects (consecutive SV messages are separated by the following "$$__$$" characters). I can automatically generate scenarios with an increasing number of traffic participants, and record the time at each process (time_process_1 ... time_process_4). Please let me know if this is something that would be helpful for the Benchmarking activity.