search

quickfix-j / quickfixj

QuickFIX/J is a full featured messaging engine for the FIX protocol. - This is the official project repository.
http://www.quickfixj.org
Other
955 stars 611 forks source link

including benchmarking code for QFJ within another module in the code base #576

Closed busy-spin closed 2 months ago

busy-spin commented 1 year ago

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Provide additional maven module to include all performance regression. Any code changes would additionally have to pass the performance regression.

Example , having this benchmark code from @chrjohn, within code base is useful. https://github.com/quickfix-j/quickfixj/pull/559

Describe alternatives you've considered N/A

Additional context N/A

chrjohn commented 1 year ago

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

jvirtanen commented 1 year ago

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

Running microbenchmarks reliably in GitHub Actions is probably hard, as they are very sensitive to their runtime environment. However, just having the microbenchmarks in an executable Maven module is itself a good goal, even if it is not part of the CI.

For Philadelphia, I run the microbenchmarks manually towards the end of the release cycle. JMH provides machine-readable output formats, such as JSON, if you pass it the -rf and -rff options. In Philadelphia, I use this to generate a human-readable performance regression test report (sample). (Incidentally, the sample shows how susceptible these things are; these are from an otherwise idle laptop, which itself should be quite a bit more stable than the standard GitHub Actions runners. Earlier I used to spin up a c4.xlarge Amazon EC2 instance just for this purpose.)

busy-spin commented 1 year ago

Sounds like a sensible idea. Are you aware of anything that could also trigger a github action in case there is a regression? Or will the build just fail?

Running microbenchmarks reliably in GitHub Actions is probably hard, as they are very sensitive to their runtime environment. However, just having the microbenchmarks in an executable Maven module is itself a good goal, even if it is not part of the CI.

For Philadelphia, I run the microbenchmarks manually towards the end of the release cycle. JMH provides machine-readable output formats, such as JSON, if you pass it the -rf and -rff options. In Philadelphia, I use this to generate a human-readable performance regression test report (sample). (Incidentally, the sample shows how susceptible these things are; these are from an otherwise idle laptop, which itself should be quite a bit more stable than the standard GitHub Actions runners. Earlier I used to spin up a c4.xlarge Amazon EC2 instance just for this purpose.)

@chrjohn now I think about it it might be difficult to come up performance regression as we need to come up with sensible assertion for the benchmark. But I totally agree with @jvirtanen , if we have executable maven module based on JMH we can provide way for contributors to manually benchmark them.

Can we take this as a feature, I'd like to make a PR on this.

chrjohn commented 1 year ago

The assertion probably would be that after each commit the performance should not decrease. But as @jvirtanen said (thank you for your thoughts on this) it could be problematic. Please go ahead and create a PR. Many thanks in advance. 👍

busy-spin commented 1 year ago

hi @chrjohn / @jvirtanen

I made PR for quickfixj-per-test , kindly provide your feedback

https://github.com/quickfix-j/quickfixj/pull/582