search

nasa / openmct

A web based mission control framework.
https://nasa.github.io/openmct/
Other
12.09k stars 1.26k forks source link

[Performance] Add minimal performance test to detect telemetry eviction #7312

Open unlikelyzero opened 11 months ago

unlikelyzero commented 11 months ago

Summary

scottbell commented 11 months ago

Getting 7 memory leaks, though none look related to imagery telemetry:

⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s1.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s2.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct/e2e/test-data/snapshots/data/cur/s3.heapsnapshot
MemLab found 7 leak(s)
Number of clusters loaded: 0

--Similar leaks in this run: 17--
--Retained size of leaked objects: 10KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --__VUE_INSTANCE_SETTERS__ (property)--->  [Array] (object) @254719 [164 bytes]
  --0 (element)--->  [<closure>] (closure) @254721 [72 bytes]
  --context (internal)--->  [<function scope>] (object) @65935 [13KB]
  --n (variable)--->  [Object] (object) @224927 [4.4KB]
  --get SU (property)--->  [SU] (closure) @252267 [72 bytes]
  --context (internal)--->  [<function scope>] (object) @71073 [24.5KB]
  --Be (variable)--->  [WeakMap] (object) @259471 [8.4KB]
  --table (internal)--->  [<array>] (array) @259787 [8.4KB]
  --36 / part of key (Object @319849) -> value (system / JSProxy @319851) pair in WeakMap (table @259787) (internal)--->  [system / JSProxy] (hidden) @319851 [268 bytes]
  --1 (hidden)--->  [Object] (object) @319849 [252 bytes]
  --onChange (property)--->  [p] (closure) @354829 [76 bytes]
  --context (internal)--->  [<function scope>] (object) @354849 [312 bytes]
  --l (variable)--->  [destroy] (closure) @354933 [48 bytes]
  --context (internal)--->  [<function scope>] (object) @355163 [20 bytes]
  --c (variable)--->  [Object] (object) @354143 [10.9KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @60335 [10KB]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60361 [9.8KB]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60353 [196 bytes]
  --6 (element)--->  [Detached Text] (native) @60351 [108 bytes]
  --4 (element)--->  [Detached HTMLDivElement] (native) @60347 [368 bytes]
  --9 (element)--->  [Detached InternalNode] (native) @12940 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @12942 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @14560 [112 bytes]
  --1 (element)--->  [Detached EventListener] (native) @10192 [112 bytes]

--Similar leaks in this run: 48--
--Retained size of leaked objects: 5.5KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --forms (property)--->  [I] (object) @373367 [2.8KB]
  --formController (property)--->  [E] (object) @373573 [2.6KB]
  --controls (property)--->  [Object] (object) @491191 [2.4KB]
  --file-input (property)--->  [Object] (object) @319867 [2.1KB]
  --destroy (property)--->  [destroy] (closure) @491245 [28 bytes]
  --context (internal)--->  [<function scope>] (object) @349391 [2.1KB]
  --i (variable)--->  [destroy] (closure) @349925 [2KB]
  --context (internal)--->  [<function scope>] (object) @349927 [2KB]
  --c (variable)--->  [Object] (object) @344171 [2KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @60435 [144 bytes]
  --5 (element)--->  [Detached HTMLDivElement] (native) @60421 [576 bytes]
  --7 (element)--->  [Detached HTMLDivElement] (native) @60411 [124 bytes]
  --5 (element)--->  [Detached HTMLFormElement] (native) @60389 [888 bytes]
  --14 (element)--->  [Detached InternalNode] (native) @6344 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @3910 [112 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @3912 [112 bytes]
  --1 (element)--->  [Detached EventListener] (native) @14640 [112 bytes]

--Similar leaks in this run: 46--
--Retained size of leaked objects: 4.8KB--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --objects (property)--->  [g] (object) @75753 [5.4KB]
  --eventEmitter (property)--->  [o] (object) @68377 [16 bytes]
  --_events (property)--->  [Object] (object) @68181 [9.2KB]
  --mine:name (property)--->  [Array] (object) @68955 [164 bytes]
  --4 (element)--->  [n] (object) @473673 [24 bytes]
  --fn (property)--->  [<closure>] (closure) @473553 [27.3KB]
  --context (internal)--->  [<function scope>] (object) @473233 [27.3KB]
  --this (variable)--->  [system / JSProxy] (hidden) @370163 [27.2KB]
  --1 (hidden)--->  [Object] (object) @370177 [27.2KB]
  --_ (property)--->  [Object] (object) @370187 [21.6KB]
  --subTree (property)--->  [Object] (object) @370437 [12.2KB]
  --el (property)--->  [Detached HTMLDivElement] (native) @61617 [124 bytes]
  --3 (element)--->  [Detached HTMLDivElement] (native) @180 [96 bytes]
  --2 (element)--->  [Detached HTMLUListElement] (native) @61641 [124 bytes]
  --4 (element)--->  [Detached Text] (native) @61645 [108 bytes]
  --4 (element)--->  [Detached HTMLLIElement] (native) @61691 [124 bytes]
  --6 (element)--->  [Detached HTMLLIElement] (native) @61683 [124 bytes]
  --6 (element)--->  [Detached HTMLLIElement] (native) @61675 [124 bytes]
  --4 (element)--->  [Detached HTMLDivElement] (native) @61681 [204 bytes]
  --3 (element)--->  [Detached Text] (native) @14986 [80 bytes]

--Similar leaks in this run: 3--
--Retained size of leaked objects: 432 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --indicators (property)--->  [c] (object) @373351 [1.1KB]
  --indicatorObjects (property)--->  [Array] (object) @373545 [820 bytes]
  --2 (element)--->  [Object] (object) @373815 [76 bytes]
  --destroy (property)--->  [destroy] (closure) @374087 [48 bytes]
  --context (internal)--->  [<function scope>] (object) @374291 [20 bytes]
  --c (variable)--->  [Object] (object) @75675 [1KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @62011 [144 bytes]

  ✘  1 [chrome-memory] › memory/eviction.memory.perf.spec.js:91:8 › Telemetry eviction for › imagery (27.7s)
--Retained size of leaked objects: 324 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --webpackHotUpdateopenmct (property)--->  [<closure>] (closure) @315253 [416 bytes]
  --context (internal)--->  [<function scope>] (object) @315255 [344 bytes]
  --previous (internal)--->  [<function scope>] (object) @69777 [140 bytes]
  --n (variable)--->  [Object] (object) @381347 [386.2KB]
  --61120 (element)--->  [Object] (object) @258701 [640 bytes]
  --exports (property)--->  [<closure>] (closure) @257753 [114KB]
  --context (internal)--->  [<function scope>] (object) @207947 [113.9KB]
  --_r (variable)--->  [e] (closure) @257767 [1KB]
  --_link (property)--->  [Detached HTMLAnchorElement] (native) @61879 [324 bytes]
  --3 (element)--->  [Detached DOMTokenList] (native) @6078 [56 bytes]

--Similar leaks in this run: 1--
--Retained size of leaked objects: 144 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --3 (shortcut)--->  [Window / http://localhost:8080] (object) @6211 [154.3KB]
  --openmct (property)--->  [v] (object) @69049 [63.4KB]
  --time (property)--->  [c] (object) @470183 [3.4KB]
  --_events (property)--->  [Object] (object) @470221 [1.8KB]
  --tick (property)--->  [Array] (object) @471837 [80 bytes]
  --0 (element)--->  [n] (object) @472099 [24 bytes]
  --fn (property)--->  [<closure>] (closure) @370015 [28 bytes]
  --context (internal)--->  [<function scope>] (object) @369905 [5.8KB]
  --e (variable)--->  [native_bind] (closure) @271541 [5.8KB]
  --bound_this (internal)--->  [system / JSProxy] (hidden) @369907 [5.7KB]
  --1 (hidden)--->  [Object] (object) @370013 [5.6KB]
  --_ (property)--->  [Object] (object) @66385 [5.5KB]
  --appContext (property)--->  [Object] (object) @369917 [1.4KB]
  --app (property)--->  [Object] (object) @369801 [1KB]
  --_container (property)--->  [Detached HTMLDivElement] (native) @62021 [144 bytes]

--Similar leaks in this run: 1--
--Retained size of leaked objects: 132 bytes--
[<synthetic>] (synthetic) @1 [13.6MB]
  --6 (element)--->  [C++ Persistent roots] (synthetic) @15448 [25.7KB]
  --5 (element)--->  [HTMLInputElement] (native) @62085 [1.2KB]
  --<symbol _vei> (property)--->  [Object] (object) @402719 [28 bytes]
  --onClick (property)--->  [i] (closure) @62087 [84 bytes]
  --context (internal)--->  [<function scope>] (object) @402715 [24 bytes]
  --previous (internal)--->  [<function scope>] (object) @67085 [10.2KB]
  --B (variable)--->  [Detached HTMLTemplateElement] (native) @61867 [132 bytes]
scottbell commented 11 months ago

Running against YAMCS yield 4 unrelated memory leaks:

     1 [chromium] › yamcs/eviction.e2e.spec.js:43:5 › Telemetry eviction for @yamcs › gyro plot
[WebServer] <i> [webpack-dev-server] [HPM] Upgrading to WebSocket
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s1.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s2.heapsnapshot
⏲️  Waiting 2000ms...
🚮 Running garbage collection...
📸 Capturing heap snapshot to /Users/scott/workspace/openmct-yamcs/tests/e2e/test-data/snapshots/data/cur/s3.heapsnapshot
MemLab found 4 leak(s)
Number of clusters loaded: 0

--Similar leaks in this run: 3--
--Retained size of leaked objects: 420 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --set _assetPath (property)--->  [set] (closure) @335437 [32 bytes]
  --context (internal)--->  [<function scope>] (object) @348613 [80 bytes]
  --previous (internal)--->  [<function scope>] (object) @119391 [51.1KB]
  --$s (variable)--->  [Detached HTMLDivElement] (native) @59129 [420 bytes]
  --3 (element)--->  [Detached HTMLAnchorElement] (native) @3994 [296 bytes]

--Similar leaks in this run: 2--
--Retained size of leaked objects: 324 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  ✘  1 [chromium] › yamcs/eviction.e2e.spec.js:43:5 › Telemetry eviction for @yamcs › gyro plot (22.1s)
  --context (internal)--->  [<function scope>] (object) @196197 [356 bytes]
  --previous (internal)--->  [<function scope>] (object) @75671 [103.4KB]
  --e (variable)--->  [Object] (object) @240881 [50.9KB]
  --61120 (element)--->  [Object] (object) @163641 [24 bytes]
  --exports (property)--->  [<closure>] (closure) @259621 [113.9KB]
  --context (internal)--->  [<function scope>] (object) @192971 [113.9KB]
  --wr (variable)--->  [t] (closure) @259657 [1KB]
  --_link (property)--->  [Detached HTMLAnchorElement] (native) @59107 [324 bytes]
  --3 (element)--->  [Detached DOMTokenList] (native) @108 [56 bytes]

--Similar leaks in this run: 2--
--Retained size of leaked objects: 200 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --layout (property)--->  [s] (object) @286411 [8.9KB]
  --$refs (property)--->  [Object] (object) @389901 [376 bytes]
  --browseObject (property)--->  [s] (object) @287429 [7.3KB]
  --viewContainer (property)--->  [Detached HTMLDivElement] (native) @58145 [200 bytes]
  --5 (element)--->  [Detached InternalNode] (native) @6708 [56 bytes]
  --1 (element)--->  [Detached InternalNode] (native) @992 [56 bytes]
  --1 (element)--->  [Detached DOMTokenList] (native) @994 [56 bytes]

--Similar leaks in this run: 1--
--Retained size of leaked objects: 144 bytes--
[<synthetic>] (synthetic) @1 [12.9MB]
  --2 (shortcut)--->  [Window / http://localhost:9000] (object) @6207 [37.7KB]
  --openmct (property)--->  [v] (object) @107339 [45.4KB]
  --__ob__ (property)--->  [vt] (object) @324249 [24 bytes]
  --dep (property)--->  [lt] (object) @288229 [216 bytes]
  --subs (property)--->  [Array] (object) @391569 [196 bytes]
  --6 (element)--->  [di] (object) @352463 [548 bytes]
  --vm (property)--->  [s] (object) @316495 [3.9KB]
  --$parent (property)--->  [yi] (object) @344783 [2.6KB]
  --$options (property)--->  [Object] (object) @353327 [672 bytes]
  --el (property)--->  [Detached HTMLDivElement] (native) @58223 [144 bytes]
scottbell commented 11 months ago

Last state of this is we've got a test in the openmct-yamcs repo, and it works if the telemetry is being emitted at ~100Hz. Below that, memlab doesn't seem to see that unbounded object growth of the plot series.

The thing I'd like to add next is to get a trace of the object, find if they're openmct objects, and collect only those for the test instead of relying on hardcoded object types.

scottbell commented 10 months ago

Some open questions @akhenry @unlikelyzero @ozyx

  1. Do we want this in the openmct-yamcs branch? I think this approach should also work with SWGs. Though maybe we’re also worried about the adapter itself? Do we want to test every object that can take telemetry for memory growth over time?
  2. I set a threshold for memory growth at 500kb. Is that reasonable? Should we set it higher/lower?
  3. How long do we want to look at these objects? I’m currently looking at them for 30s per snapshot, so 150s total.
scottbell commented 10 months ago

note i’ve also synthetically tested this approach by just adding a quick return in the PlotSeries’s purge method. the test correctly identifies that telemetry is stacking up.

scottbell commented 10 months ago

Approach after conferring with @akhenry: When we receive the first telemetry value for both request & subscribe (in the Telemetry API), register the telemetry javascript objects with the finalization method, run view for a minute, then check to see if both telemetry javascript objects are gone.

  1. want to test every view that can take telemetry (one big display?)
  2. check to see if we can use telemetry api low level
  3. do it in openmct-yamcs
scottbell commented 10 months ago

After talking with @akhenry, putting this one on hold for a bit. Current status:

  1. Interception of all TelemetryAPI request calls is working. At page load + 5s, all returned telemetry objects from the request calls are registered with the FinalizationRegistry.
  2. Interception of all TelemetryAPI subscribe calls is working.At page load + 5s, all telemetry objects from the subscribe callbacks are registered with the FinalizationRegistry.
  3. The test passes once all telemetry objects are garbage collected, and thus "evicted".

The reason we're putting this on hold is that the telemetry objects themselves, at least within the Plot View, are almost immediately copied into a new array.. This means the telemetry objects are garbage collected, but also reduces the utility of this test. We should consider:

  1. Changing test to directly hook into PlotSeries to track the newly copied data (i.e., in the add method). To do this we'd need to allow a function to be added to the in PlotSeries via a $ref that feed references into the FinalizationRegistry. This seems really ugly!
  2. Longer term, stop copying telemetry data into a new array.
scottbell commented 9 months ago

Digging into this more, I think I'm wrong about this. I did a bunch of testing in this branch adding finalization registers inside of the Telemetry API and the actual PlotSeries, and found the data being garbage collected at the same rate, and the same number of items:

https://github.com/nasa/openmct/assets/9853862/47a5d7cc-4ed2-403c-b252-2f9bb78a2dc1

So I think this PR is probably fine as is.