Entity creation / destruction slows down over time?

[morphogencc]

I've found a very odd situation that I'm trying to troubleshoot right now using entityx.

I'm using OpenCV to detect and track objects from frame-to-frame in a video, and then using entityx to manage the physics of each tracked object. I've found that as my project runs, the framerate drops -- it starts at 60 fps, then within 30 minutes it's dropped down to 48 or so, and by the 90 minute mark it's barely hitting 20 fps. After several rounds of surgically removing / adding code back into the project to identify the issue, I've found that the creation / destruction of objects completely slows down the software:

void setup(void* display) {
    Display* parentDisplay = static_cast<Display*>(display);
    mEntityHandle = parentDisplay->mEntities.create();
    mEntityHandle.assign<Id>(-1);
}

void update(void* display) {
    Display* parentDisplay = static_cast<Display*>(display);

    if (mEntityHandle.valid()) {
        auto id_component = mEntityHandle.component<Id>();
        id_component->mId = getLabel();
    }
}

void destroy(void* display) {
    Display* parentDisplay = static_cast<Display*>(display);
    entityx::ComponentHandle<sitara::Id> id;
    //entityx::ComponentHandle<sitara::ecs::Attractor> force;
    for (auto entity : parentDisplay->mEntities.entities_with_components(id)) {
        if (id->mId == getLabel()) {
            entity.destroy();
        }
    }
}

These three functions are simply hooks that are called by my object tracker, and input a void* display that represents the parent class of my entityx::EntityManager objects. But I've found that removing these few lines of code removes my slowdown problem completely, and re-adding them causes the issue I'm observing.

So my question is: what could possibly be causing this? While running in VS2019 I don't see any memory leaks (memory usage looks pretty steady); it could be that there's a very expensive memory alignment operations that occurs somewhere due to the constant destruction / creation of entities, but if so, I'm not sure how to compensate for it.

For further troubleshooting information, my Id Component is very simple:

    struct Id {
        Id(int id) {
            mId = id;
        }

        int mId;
    };

So I doubt that there's any issues with this being a very expensive operation...

[alecthomas]

Interesting. Are you increasing the total number of entities over time or does it remain roughly constant?

Can you run a profiler over it? That would be very helpful.

FYI when an entity is deleted it is added to a free list. Subsequent entity creation calls will first consume from this list before extending the underlying entity component vectors

[morphogencc]

Like I mentioned, nothing crazy shows up on the profiler (no memory leaks, no excessive CPU load), but the relevant function that looks noticeable is that kill() is taking up an abnormal amount of CPU time:

Function Name   Total CPU [unit, %] Self CPU [unit, %]
| - sitara::FishFollower::kill  42819 (5.83%)   220 (0.03%)
| - entityx::EntityManager::ViewIterator<entityx::EntityManager::UnpackingView<sitara::Id>::Iterator,0>::next   33553 (4.57%)   2563 (0.35%)
| - entityx::EntityManager::ViewIterator<entityx::EntityManager::UnpackingView<sitara::Id>::Iterator,0>::operator++ 31995 (4.36%)   88 (0.01%)
| - entityx::EntityManager::ViewIterator<entityx::EntityManager::UnpackingView<sitara::Id>::Iterator,0>::predicate  22055 (3.00%)   1950 (0.27%)

Those three functions are the most-called functions from entityx, taking up even more runtime than my System update() calls.

I also checked the entity numbers, and it looks like it's roughly constant -- I have 3 EntityManagers for 3 different FBOs, and each one manages roughly 80 - 140 items.

[alecthomas]

Can you try periodically printing EntityManager.size() and EntityManager.capacity()?

[alecthomas]

Also are you compiling in release mode? In debug mode there are a lot of assertions.

[morphogencc]

Compiling in Release x64.

I ran it my application for a few hours and the frame rate's down to 13 fps... and it looks like the capacity for all three managers has been constant at 144, whether the EntityManager has 80 or 130 entities.

Does the iterators over sitara::Id mean that looping during the kill function is taking up a lot of CPU time? I didn't think 130 entities should be noticeably large, but it surprises me to see that the iterator is over the component...

[alecthomas]

130 entities is inconsequential. The example included with EntityX has about 20k entities and I get ~50fps on my laptop. That said, I haven't run it for hours, but I find it very surprising to be honest.

I can't think of anything that would cause this, and I think it will be hard to debug this further without code to reproduce it.

[alecthomas]

Actually with a full release build I get 400+fps - around 8M entities/s. That is with 8 systems, collisions, etc. It's not complex logic, but it is representative of the raw throughput of EntityX.

[morphogencc]

Thought sounds right to me -- I'd honestly be surprised if 140 entities was a lot. I was hoping there was something simple like a vector reserve() that was causing my problems, but I suppose I'll keep digging and see if I can produce a minimal example.

Thanks for your help alec!

[alecthomas]

I'll run the example for an hour this morning to see what happens.

Wild question, but is it possible your machine is overheating?

[alecthomas]

FWIW I ran the example for an hour and didn't see any slowdown at all, still a solid ~400fps 🤷‍♂️

[morphogencc]

@alecthomas Wild! I have no idea what's going on here, but it seems clear that it's on my end... thanks for the troubleshooting help, and at least helping me narrow down where the issue might be.

[alecthomas]

No problem and good luck. I'd be interested in hearing what the cause was if/when you find it, just out of curiosity.

[morphogencc]

So I seem to have found a resolution... In my old code, I would mark tracked objects for destruction and call destroy() on each object individually.

So basically, I was iterating:

for (auto& obj : markedObjects) {
    obj.destroy();
}

Now on every frame, I simply destroy all entities and then recreate them next frame by calling setup().

This change has completely removed the slow-down issues... I can't really understand why that would be, except that destroy() iterates over all of the entities, so there's a nested for-loop... but the size of each list is in the dozens, at most, so I would assume the inefficiency is negligible.

Happy to share more (or even my project source) if it'd help figure out what exactly was causing the slowdown here, but it seems to be resolved for now.

[alecthomas]

When you say

Now on every frame, I simply destroy all entities

What do you mean? Calling obj.destroy(), but on all entities rather than just marked ones?

[morphogencc]

Correct -- what I used to do was each object would call it's destroy() function, which then filters through all components with an Id component and searches for one with a matching id and destroys it. Across all objects, this snippet looks like:

for (object in markedObjects) {
    for (entity in Entities_with_Id) {
        if (object.Id == entity.component<Id>()->mId) {
            entity.destroy();
        }
    }
}

Now my code, once per frame, just filters through the entities and destroys them all:

for(entity in Entities_with_Id) {
    entity.destroy();
}

In theory I would expect this to have a higher cost from both creation and destruction (as they both occur more frequently) but my application very happily chugs along at a constant framerate for 22+ hours now.