test(core): Remove cpu usage/heap total performance measuring

jpinkney-aws commented 2 months ago

Problem

We're getting a signal that cpu usage/heap total can't be used to performance measure vscode extensions. This is because all vscode extensions run in a single process, causing other extensions to interfere with our monitoring.

Solution

Remove cpu usage/memory performance measuring
Keep duration measuring for now to see if this can be useful for regression testing (at least its better than nothing?)

License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

hayemaxi commented 2 months ago

Couldn't this be a little beneficial in automated CI, where the extension is only one running (if it even is)?

jpinkney-aws commented 2 months ago

Its technically running alongside whatever microsoft provides by default (which may or may not also contribute to the cpu usage/memory)

When I start up a brand new vscode insiders instance with no extensions installed I already have:

2024-08-30 11:11:04.466 [info] Extension host with pid 3697 started
2024-08-30 11:11:04.639 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git
2024-08-30 11:11:04.674 [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '*'
2024-08-30 11:11:04.697 [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: '*'
2024-08-30 11:11:04.792 [info] Eager extensions activated
2024-08-30 11:11:04.822 [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: false, activationEvent: 'onStartupFinished'
2024-08-30 11:11:04.825 [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
2024-08-30 11:11:11.852 [info] ExtensionService#_doActivateExtension vscode.emmet, startup: false, activationEvent: 'onLanguage'

Its maybe worth a try but from what I've seen so far everything points to a lack of consistency in the extension host

jpinkney-aws commented 2 months ago

As I was writing the PR that would move performance tests to their own CI I just realized that unit tests disable all other extensions and the performance tests which run in there still fail occasionally ☹️ https://github.com/aws/aws-toolkit-vscode/blob/master/packages/core/scripts/test/launchTestUtilities.ts#L89

jpinkney-aws commented 2 months ago

I'll close this for now so it stays around. George gave me a couple more ideas

aws / aws-toolkit-vscode

test(core): Remove cpu usage/heap total performance measuring #5524

Problem

Solution