Performance regression while loading large levels

[srikappa-amzn]

Describe the bug It's been noticed that for loading large levels (with 10k entities), loading times seems to have drastically fallen down. What used to be loaded in a few seconds is now taking minutes (I've killed the editor when I crossed the 3 minute mark)

Steps to reproduce Steps to reproduce the behavior:

1. Get latest development
2. Build the AutomatedTesting project
3. Open the AutomatedTesting\Levels\Performance\10KEntityCpuPerfTest level

Expected behavior Level opens and loads within a few seconds

Actual behavior Level loading takes many minutes

Found in Branch development

Desktop/Device (please complete the following information):

• Device: PC
• OS: Windows
• Version 10
• CPU 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz 2.50 GHz
• GPU NVIDIA GeForce RTX 3070 Laptop GPU
• Memory 32 GB

[srikappa-amzn]

Did a quick profile on a large level, a lot of calls are coming from PrefabPublicHandler::GenerateUndoNodesForEntityChangeAndUpdateCache which is coming from LocalViewBookmarkLoader::FindOrCreateLocalViewBookmarkComponent(), which in turn is coming from EditorViewportWidget::SetDefaultCamera(). And in almost all of these cases, the patch produced is empty and the function just returns. But I believe that the function is probably called an large number of times unnecessarily.

Confirmed the assumption with testing. It is definitely much much faster to not let GenerateUndoNodesForEntityChangeAndUpdateCache () get called during level loading. (It just makes sense to skip that because why would you want undo/redo support during the process of loading a level?) I was able to load the 'AutomatedTesting\Levels\Performance\10KEntityCpuPerfTest\10KEntityCpuPerfTest.prefab' level within a few seconds as opposed to minutes(3 minutes and still counting)

Next steps:

1. Prevent ToolsApplication::CreateUndosForDirtyEntities() from getting called during level loading. Listen to the PrefabPublicNotifications::OnRootPrefabInstanceLoaded() notification to know when a level is ready
2. Ensure that the issue doesn't occur when open a new level after a level is loaded
3. Ensure that the issue doesn't occur when saving existing level as a new level (level save as)

Bonus points: Investigate how the change in FindOrCreateLocalViewBookmarkComponent() contributed to such large delays. Why is each and every single one of the 10k entities marked as dirty? is it probably because of selection component deprecation?

[srikappa-amzn]

Potentially related issue: If I open my 10 k cubes level just selecting a parent (taht has 1000 children) freezes the editor up

[AMZN-daimini]

Potentially related issue: If I open my 10 k cubes level just selecting a parent (taht has 1000 children) freezes the editor up

That specific part may be due to this https://github.com/o3de/o3de/issues/12682

[nick-l-o3de]

Theres at least 2 problems here, one is the console spam but the other is that its creating undos for the entities just loaded, which is unnecessary.

[srikappa-amzn]

Reopening this since https://github.com/o3de/o3de/pull/13081 improves level loading times but still is not fast enough. Next step is to investigate why a lot of entities are being marked during level loading. @michabr , fyi

[michabr]

@srikappa-amzn The reason for entities being marked as dirty during level loading comes from components marking the entity dirty during entity activation. The TransformComponent and EditorMaterialComponent both do this. I am testing out a fix for this that only marks dirty entities if a valid current undo batch exists.