Editor: Performance Issues With 18000 Entities

[galibzon]

Describe the bug

1. When the level is almost empty it takes only 0.02 seconds to add an entity and its components. By the time there are 9000 entities it takes around 15 seconds to add an entity and its components. As each entity is added by the attached python script "populate_static_objects.py" the memory consumption increases and after a thousand entities have been added, VS2019 reports 20GB extra to be taken from the starting used memory.

2. Sometimes: Initially, upon loading the goals_9k_1kd_prepopulated level, VS2019 reports memory consumption of 13GB. When adding a "Comment" component to the Camera entity (child of "Atom Default Environment" entity), and upon saving the level is gets stuck for a while and memory consumption spikes to 37.6GB.

3. Enabling "Helpers" and "Icons" visualization in the viewport when there are thousand of visible entities can bring the performance down to 2 fps. There should be a way to only enable auxiliary geometry for the entity in focus.

4. Why is the goals_9k_1kd_prepopulated.prefab file 62MB Large? All entities are identical and they have 4 components referring the same assets.

Assets required The following zip file: goal_9ks_1kd_TWO_levels.zip
Contains two levels goal_9ks_1kd & goal_9ks_1kd_prepopulated. The level goal_9ks_1kd only has a few entities, and you have to enter Game Mode to see it spawning the static entities that will be placed evenly on top of the terrain, and later dynamic entities will be spawned.
The level goal_9ks_1kd_prepopulated already has 9000 entities. It was prepopulated using the script goal_9ks_1kd/Scripts/populate_static_objects.py. You can use this level to debug why it takes 15 seconds to add or remove a new entity.

Steps to reproduce Load goal_9ks_1kd_prepopulated and try to add or remove entities.

Expected behavior It should be almost instantaneous.

Actual behavior It takes around 15 seconds to add or remove a single entity.

Screenshots/Video If applicable, add screenshots and/or a video to help explain your problem.

Found in Branch commit 6ec1466f63b476cfea6a8391b8340f39ff4d5679 (HEAD -> development, upstream/development)

commit 6ec1466f63b476cfea6a8391b8340f39ff4d5679 (HEAD -> development, upstream/development)
Merge: ceff3d230a 6be944190b
Author: Gene Walters <32776221+AMZN-Gene@users.noreply.github.com>
Date:   Thu Jul 21 04:57:34 2022 -0700

Desktop/Device (please complete the following information):

• Device: PC
• OS: Windows
• Version 10
• CPU AMD Ryzen Threadripper 3970X 32-Core Processor 3.70 GHz
• GPU NVidia RTX 3070
• Memory 64GB

Additional context Issues listed here are related with this other issue: https://github.com/o3de/o3de/issues/10699.
To use goal_9ks_1kd/Scripts/populate_static_objects.py you'll have first to comment the code in CryEditPy.cpp PyIdleWaitFrames:

    void PyIdleWaitFrames([[maybe_unused]] uint32 frames)
    {
        //struct Ticker : public AZ::TickBus::Handler
        //{
        //    ...
        //    ...
        //};
        //
        //QEventLoop loop;
        //Ticker ticker(&loop, frames);
        //loop.exec();
    }

In case you want to add thousand of entities to a level. Use goal_9ks_1kd/Scripts/populate_static_objects.py with pyRunFile command. Example: Add 1000 entities above the terrain saving the level every 100 entities (to not lose the work in case of a crash)

pyRunFile D:\GIT\o3de\AutomatedTesting\Levels\goal_9ks_1kd\Scripts\populate_static_objects.py -m 1000 -o 0 -c 1000 -s 100

[galibzon]

Assigned it to @lsemp3d & @monroegm for further re-assignment.

[lsemp3d]

@galibzon thanks for the research! I think we should break this up into one GHI per issue listed above, it will make it easier to divide the work between prefabs, editor and possibly viewport (depends what is causing the camera issue).

[galibzon]

@galibzon thanks for the research! I think we should break this up into one GHI per issue listed above, it will make it easier to divide the work between prefabs, editor and possibly viewport (depends what is causing the camera issue).

@lsemp3d:
Created https://github.com/o3de/o3de/issues/10921 & https://github.com/o3de/o3de/issues/10922

[hultonha]

@galibzon FYI we did add an option to only show helpers for the selected entities a little while ago, that might have at least solved point 3 (although it would be nice to make things faster overall - I still wonder if we should just remove all icons in the viewport as I'm not convinced how much value they add - I'm not sure other game engines have them)

[nick-l-o3de]

It now takes ~2.1gb of memory to load this level instead of the original 13gb. Adding a comment node adds an aditional 100mb of memory, but adding additional comment nodes to other things does not move the memory amount. It still takes like 3 seconds to update when you do add a component, or move a single entity though

So I think the memory problem is gone, but the performance is still fairly poor with this many entities at the root level. Performance is actually way better when the entities are in prefabs that group them into reasonable clumps (like, 100 prefab instances, each containing 100 prefab instances, of 100 entities each) as opposed to a giant flat list to deal with.

[nick-l-o3de]

I did a profile of what the CPU was doing during an operation of selecting 1 entity and duplicating it.

• 29% actually duplicating prefab instances (half capturing undo, half generating patch)
• 66% updating instances in the level

the instance updates are

• 38% - actually updating the entities, via the JSON serialization system
• 28% - creating and activating C++ side components (as in, AZ::Entities, AZ::Components, and activating them)

This would infer to me that its still doing global operations (as in, operating on the entire level worth of entities) on local edits (modifying one entity or duplicating one instance)

[nick-l-o3de]

Summary basically: The current prefab algorithms all operate such that they touch the data inside the prefab that owns the entity being modified. (Levels are prefabs). so if you have a prefab with 10,000 entities in it, N in O(N) is at least 10,000. This is true whether its a prefab with 10,000 entities in it, or a level with 10,000 entities in it. Edit operations essentially involve the owning prefab. So one workaround is to keep your owning prefab sizes down, split your world into a heirarchy of prefabs and perform operations on them to keep the "scope" of the edit small.

A lot of the improvements are there, but the fundamental slowness with large levels is not going to go away until improvements are made that reduce the scope of modification so that modifications to a single entity no longer involve its siblings or its owning prefab (taking it from O(N) to O(1))

Whether this counts as a bug still or a improvement/feature to work towards... not sure. Its not actually bugged. Its just the algorithm is what it is...