search

pragmaticminds / crunch

Industrial data evaluation engine
https://pragmaticminds.de/
Apache License 2.0
13 stars 3 forks source link

Performance Issues for streams with many messages #10

Open JulianFeinauer opened 5 years ago

JulianFeinauer commented 5 years ago

Currently, we observe a performance problem when processing many messages in a Stream with man evaluation functions or with evaluation functions that "look" at many signals. I already did some analysis in the branch "performance-test".

The class TestCrunchPerformanceNT contains the main to run the tests (and also a setup to run it with JMH with the IntelliJ Plugin).

The file referenced there is 30MB large and thus not in the repo but is shared in the crunch-oss channel in Slack (or downloaded here: https://pragmaticminds.slack.com/files/U1Q8RMFDL/FFEAR60TS/sample2.txt)

ottobackwards commented 5 years ago

any chance you can host this outside your company slack?

JulianFeinauer commented 5 years ago

Hi @ottobackwards... This is indeed a valuable concern which we'd like to address. A good and publicly available infra is one of the reasons why we'd like to move to asf :)

ottobackwards commented 5 years ago

Well, I wanted to run the test, but I'm not sure I want to join your slack

JulianFeinauer commented 5 years ago

Yeah I agree. And we already have a fix now but it's not really rolled out because we had other issues we focused on...

Von meinem Mobiltelefon gesendet

-------- Ursprüngliche Nachricht -------- Betreff: Re: [pragmaticminds/crunch] Performance Issues for streams with many messages (#10) Von: Otto Fowler An: pragmaticminds/crunch Cc: Julian Feinauer ,Author

Well, I wanted to run the test, but I'm not sure I want to join your slack

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/pragmaticminds/crunch/issues/10?email_source=notifications&email_token=AECOQCQ7J7P2E56GINA3Q2LQGFMRXA5CNFSM4GQ5QJW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CDAAA#issuecomment-524562432, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AECOQCTORCUHVOZACCTFQA3QGFMRXANCNFSM4GQ5QJWQ.