nasa / openmct

A web based mission control framework.
https://nasa.github.io/openmct/
Other
12.02k stars 1.24k forks source link

Display became inoperable #7857

Open davetsay opened 5 days ago

davetsay commented 5 days ago

Summary

In a production deployment, a complex display - which relies on a Remote Clock - became inoperable. Troubleshooting resulted in the following analysis. 1) the remote clock never receives a time via historical request. 2) The main event loop is getting blocked with style recalculations, so it fails to process telemetry values, including the current time via subscriptions. The Remote Clock relies on getting time via telemetry request or subscription. Other telemetry requests rely on Remote Clock having a time set, so they in turn get stalled.

Fix these

  1. Remote Clock never receives telemetry from a historical request, due to a code regression.
  2. Progress bars are very CPU intensive.
  3. Plans are polling to resize internal components due to window resizes.

Expected vs Current Behavior

The Remote Clock gets a time set and telemetry begins to flow and views render properly

Steps to Reproduce

See VIPERGC-415 for full instructions, or...

Poor Man's Steps to Reproduce

  1. Build a complex display with all the things
  2. Configure to use Remote Clock
  3. Open complex display on multiple (at least 3) windows
  4. Refresh displays in sequence until they get stuck loading and never resolving

Environment

Impact Check List

Additional Information

davetsay commented 3 days ago

Testing Instructions:

unlikelyzero commented 2 days ago

after syncing with @davetsay ,

  1. Progress bars are already tested with a visual test
  2. Planview resizing is already tested with a visual test
  3. We need to figure out how to add a test to cover the remote clock / timeContext changes