nasa / openmct

A web based mission control framework.
https://nasa.github.io/openmct/
Other
12.09k stars 1.26k forks source link

Display became inoperable #7857

Closed davetsay closed 1 month ago

davetsay commented 2 months ago

Summary

In a production deployment, a complex display - which relies on a Remote Clock - became inoperable. Troubleshooting resulted in the following analysis. 1) the remote clock never receives a time via historical request. 2) The main event loop is getting blocked with style recalculations, so it fails to process telemetry values, including the current time via subscriptions. The Remote Clock relies on getting time via telemetry request or subscription. Other telemetry requests rely on Remote Clock having a time set, so they in turn get stalled.

Fix these

  1. Remote Clock never receives telemetry from a historical request, due to a code regression.
  2. Progress bars are very CPU intensive.
  3. Plans are polling to resize internal components due to window resizes.

Expected vs Current Behavior

The Remote Clock gets a time set and telemetry begins to flow and views render properly

Steps to Reproduce

See VIPERGC-415 for full instructions, or...

Poor Man's Steps to Reproduce

  1. Build a complex display with all the things
  2. Configure to use Remote Clock
  3. Open complex display on multiple (at least 3) windows
  4. Refresh displays in sequence until they get stuck loading and never resolving

Environment

Impact Check List

Additional Information

davetsay commented 1 month ago

Testing Instructions:

unlikelyzero commented 1 month ago

after syncing with @davetsay ,

  1. Progress bars are already tested with a visual test
  2. Planview resizing is already tested with a visual test
  3. We need to figure out how to add a test to cover the remote clock / timeContext changes
akhenry commented 1 month ago

@shefalijoshi to test

shefalijoshi commented 1 month ago

Verified fixed over VPN. @unlikelyzero to verify this in the dev lab