Telemetry batching and throttling

akhenry commented 10 months ago

Closes https://github.com/akhenry/openmct-yamcs/issues/213

Describe your changes:

Moves WebSocket handling to a worker per the Chromium team's recommendation. Partly implemented as API in Open MCT so that other backends can benefit. The yamcs-specific details are handled in the Yamcs realtime provider.
Batches parameter values and emits them to views periodically (every 1000ms right now) to improve performance.
When processing batches, always processes remote clock parameter first so that subsequent values are compared against the latest time.
For performance reasons does not parse JSON messages and instead does string comparisons where needed. In testing this was much more performant.

All Submissions:

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same update/change?
[x] Is this change backwards compatible? For example, developers won't need to change how they are calling the API or how they've extended core plugins such as Tables or Plots.

Author Checklist

[x] Changes address original issue?
[ ] Tests included and/or updated with changes?
[x] Command line build passes?
[x] Has this been smoke tested?
[ ] Testing instructions included in associated issue OR is this a dependency/testcase change?

Reviewer Checklist

[ ] Changes appear to address issue?
[ ] Reviewer has tested changes by following the provided instructions?
[ ] Changes appear not to be breaking changes?
[ ] Appropriate automated tests included?
[ ] Code style and in-line documentation are appropriate?
[ ] Has associated issue been labelled unverified? (only applicable if this PR closes the issue)
[ ] Has associated issue been labelled bug? (only applicable if this PR is for a bug fix)

akhenry commented 10 months ago

CPU usages seems surprisingly higher after moving to a WebWorker. Here's before without the Web Worker (with data ~10Hz) with CPU ranging 30-60%:

and here's after on the same data, with CPU ranging 55-85%:

The heap size keeps more flat with the Web Worker implementation, probably due to the batching/throttling.

Interesting. This was not my experience with Quickstart or VIPER. Can you share the display you used here? Maybe there's something I missed.

Also, CPU usage may be higher if batching is not happening because the worker just adds overhead in that case. I wonder if there's something peculiar going on with your setup...

Oh, interestingly, I note that you are measuring CPU usage differently to me. I have been using task manager. Let me do some testing with the performance monitor...

scottbell commented 10 months ago

Oh, interestingly, I note that you are measuring CPU usage differently to me. I have been using task manager. Let me do some testing with the performance monitor...

@akhenry I'll test again with activity monitor too and see if anything changes. I can test on a standalone box too.

scottbell commented 10 months ago

Here's before:

https://github.com/akhenry/openmct-yamcs/assets/9853862/797e2d20-5976-4a83-ba23-118a8089f8ce

Here's after:

https://github.com/akhenry/openmct-yamcs/assets/9853862/104a8fe0-bdda-4c79-a30f-2de018fd756d

looks to be a much better speedup sans table. Good work!

akhenry commented 10 months ago

Here's before: before.mov

Here's after: after.mov

looks to be a much better speedup sans table. Good work!

Awesome! Yeah so it seems that the tables were the actual bottleneck rather than the telemetry processing. A PR exists to address this - https://github.com/nasa/openmct/pull/7392

akhenry commented 10 months ago

@scottbell Good to go again

scottbell commented 10 months ago

Retested and looks good!

https://github.com/nasa/openmct/assets/9853862/9959219a-b627-4f6c-9488-06e88ceed365

@akhenry can wait until next sprint, but probably want to follow a fileup for the websocket reconnection and plot data:

akhenry / openmct-yamcs