[Feature request] Clone some of the component pools into a new registry

[indianakernick]

I may have found a way to clone specific component pools into a new registry while preserving entity identifiers. sparse_set should only be cloneable if the component is copyable. The following simple test prints 14 14 if Component is copyable and fails to compile with Cannot clone non-copyable component if Component is not copyable.

struct Component {
  int value;

  // Component(Component &&) = default;
  // Component &operator=(Component &&) = default;
  // explicit Component(int value)
  //  : value{value} {}
};

int main() {
  entt::registry<> reg;
  auto e = reg.create();
  reg.assign<Component>(e, 14);
  std::cout << reg.get<Component>(e).value << ' ';
  entt::registry<> cloned = reg.clone<Component>();
  std::cout << cloned.get<Component>(e).value << '\n';
  return 0;
}

Move-only components can still be used within the registry, they just cannot be cloned. CRTP is used to give the sparse_set a clone_to function if the component is_copy_constructible.

@@ -534,6 +534,12 @@
         reverse.clear();
         direct.clear();
     }
+
+    void basic_clone_to(sparse_set<entity_type> *other) {
+        assert(other);
+        other->reverse = reverse;
+        other->direct = direct;
+    }

 private:
     std::vector<entity_type> reverse;
@@ -541,6 +547,25 @@
 };

+template <typename Derived, bool Copyable>
+class clone_base;
+
+template <typename Entity, typename Type>
+class clone_base<sparse_set<Entity, Type>, false> : public sparse_set<Entity> {
+    // no clone
+};
+
+template <typename Entity, typename Type>
+class clone_base<sparse_set<Entity, Type>, true> : public sparse_set<Entity> {
+public:
+    void clone_to(sparse_set<Entity> *other) {
+        assert(other);
+        sparse_set<Entity>::basic_clone_to(other);
+        auto *otherDerived = static_cast<sparse_set<Entity, Type> *>(other);
+        otherDerived->instances = static_cast<sparse_set<Entity, Type> *>(this)->instances;
+    }
+};
+
 /**
  * @brief Extended sparse set implementation.
  *
@@ -564,7 +589,10 @@
  * @tparam Type Type of objects assigned to the entities.
  */
 template<typename Entity, typename Type>
-class sparse_set<Entity, Type>: public sparse_set<Entity> {
+class sparse_set<Entity, Type>: public clone_base<sparse_set<Entity, Type>, std::is_copy_constructible_v<Type>> {
+    template <typename Derived, bool Copyable>
+    friend class clone_base;
+
     using underlying_type = sparse_set<Entity>;
     using traits_type = entt_traits<Entity>;

A new member function on the registry creates a new registry by cloning the given components.

@@ -1472,6 +1472,28 @@

         return { (*this = {}), assure };
     }
+
+private:
+
+    template <typename Component>
+    std::unique_ptr<sparse_set<entity_type>> clone_pool() {
+        static_assert(std::is_copy_constructible_v<Component>, "Cannot clone non-copyable component");
+        auto poolPtr = std::make_unique<sparse_set<entity_type, Component>>();
+        static_cast<sparse_set<entity_type, Component> *>(&pool<Component>())->clone_to(poolPtr.get());
+        return poolPtr;
+    }
+
+public:
+    template <typename... Components>
+    registry<entity_type> clone() {
+        registry<entity_type> newReg;
+        newReg.pools.reserve(sizeof...(Components));
+        (newReg.pools.push_back(clone_pool<Components>()), ...);
+        newReg.entities = entities;
+        newReg.available = available;
+        newReg.next = next;
+        return newReg;
+    }

 private:
     std::vector<std::unique_ptr<sparse_set<Entity>>> handlers;

It might be faster to clone_to a given registry to avoid a few memory allocations. But then you might get entity IDs mixed up.

Copying the whole registry without having to list off all of the components is difficult. If the clone_to function was virtual, you'd need to provide an implementation for when the component is not copyable. You can't static_assert(false) in there because then you can't build the vtable. The best you can do is assert(false) but I'd rather catch it at compile-time.

If you're "double-buffering" the registry then you probably only need to clone a few components. Maybe you could clone the rendering components and render the current frame while you're running game logic for the next frame.

If you only need to copy one component pool and you don't care about entity IDs then you could memcpy a raw_view.

[indianakernick]

Using a similar strategy with CRTP and std::is_empty_v, it might be possible to optimize for empty components but that's a separate discussion.

[skypjack]

Before to discuss a solution, I want to know why on the earth a registry should support full/partially cloning pools. :smile:
I mean, what is the actual problem? Do you really need to copy the internal arrays of a sparse set for that? If the goal is to store aside things for a frame so that you can work on them offline, hardly you need the overhead of a sparse set.

[indianakernick]

@skypjack There is more than one way of copying components. Each has its own strengths and weaknesses. I suggest you discuss this in the wiki somewhere because it seems like a common request. This is just one way of doing it. There are other ways that don’t require changes to EnTT.

[indianakernick]

I personally don’t actually use EnTT for any kind of threading 😄 so it may be more constructive to hear from those who do.

[skypjack]

Yeah, indeed. I'm not sure that cloning a registry/pool is the best solution.
As an example, if you want to put aside things so as to render in parallel, you don't need the overhead of a sparse set. It's a waste of memory and CPU cycles for something you won't use anyway.
Because of this I'm asking the why behind this. Let's see if others pop up to join the discussion.

[pgruenbacher]

I brought this up with kerndog on gitter. 1) Network syncing. My client simulation runs 10 steps, copies itself at time 10, and then does 10 more "predictive steps". Client then gets a state update from server for the time 10 state. It compares that against the time 10 state cache. If there's errors, it'll paste the state update into the cache, then fast-forward this updated state to time 20, at which point it becomes the master state. These corrections may ultimately lead to rubber-banding or teleporting of agents, but at least the state is now more correct between users. This requires being able to copy the whole registry state.

2) My EnTT simulation is compiled separately from my godot engine. The engine will still be using EntityId's though to keep track of data though. For instance a physics object will have a entity id associated with it. On a frame update it may want to send a request through a middleware api to access the position associated with that EntityId, and the easiest way for me to to implement this is to just have a copy of the registry. About 1/2 of the components are being accessed by the engine, and so it seems to me to just be simpler to copy the whole registry.

2 has work-arounds maybe, but I'm pretty sure copying each sparse-set is fine for now. I went into this with the plan of being able to copy easily, so all components I care about are trivially copyable for that reason. That's also why I excluded std:: containers in the components as well. 1 requires reliably reproducing the registry, but Idk if that's my ultimate networking strategy.

[pgruenbacher]

I mean for 2, I guess I can just do a SuperAgent struct that contains all the information I know I'll need for the engine, (positin, velocity, state, health) and store it as a frozen vector with an EntityId -> index vector as well to make the data easily indexable. Tbh I still haven't explored how spare sets work intenally so idk the performance differences of storing as a dense vector vs jsut copying a spare set. memory is least of my concerns though.

[vblanco20-1]

There is also a 3rd option. You want to memcopy entity render data to ship into a few Rendering Systems to run in a pipeline with the rest of the engine. This is something i wanted to do myself but couldnt. Without pipelining, you cant reach true near 100% parallelism.

[pgruenbacher]

true, for #2 that can work, kinda . I just memcpy the agent components into a superant agent struct. but #1 is still up in the air. plus it's not obvious to me how to memcpy the component data + entity data in a way different from just copying the sparse set itself.

[pgruenbacher]

To make it more clear for the reason 1, i'm not saying copying the registry for network communication, but rather for rollbacks. http://www.gabrielgambetta.com/client-side-prediction-server-reconciliation.html E.g. client keeps copy of registry at time 10, gets server update at time 20 with time 10 timestamp, checks for equivalency in time 10 registry cache, and if divergent, will fast-forward the fixed timestemp 10 registry to the current time to become the master registry state.

[dbacchet]

Hi @pgruenbacher, @Kerndog73, I'm very interested about learning other problems/use-cases, thanks for the discussion! The rollback case is what led me to ask for the Save/Restore functionality (see issue #27 ), and we use it extensively in my company. For my use-case I save the entire state of the simulation (i.e. everything in the registry) every few seconds and restore whenever I need. I voluntarily chosen not to save/restore a subset because is very error prone on a system where new components can be added dynamically, and could require saving as well. The only case I can think skipping some of the components could make sense is for heavy and static resources (meshes, images, etc.). The solution I used for that was to use a separate ResourceManager and only keep the reference (a custom smart ptr in my case) in the components.

The client/server use case you pointed to is interesting, but I'm still thinking if is not better to have a proper protocol between the client and the server to communicate changes. I'm using EnTT in a distributed simulation framework, and for that need we use gRPC (any RPC lib will do the job). This has the nice side effect that I don't require the clients to actually use an ECS, so it makes it easier to integrate existing libraries/functionalities.

Just my 2 cents, sorry for the rant :)

[pgruenbacher]

hm well I intend to simulate thousands of moving agents all the time, so each client is running their semi-deterministic simulation with lockstep user commands, but with the server sending master updates to everyone to make sure any divergences in the simulations are resolved. RPC to send agent updates would be too difficult on the bandwidth I think for hundreds of nearby agents. though who knows, I still avoid networking prototypes out of pure fear.

[skypjack]

Ok, sorry if I'm late, I was out of home during the weekend.
@pgruenbacher

Bullet 1. Just out if curiosity: did you consider using snapshot stuff for reconciliation? If you excluded it, can I ask you why?
The problem of sending over the network a registry as-is is that you are going to waste a lot of resources. Snapshot tries to reduce the overhead to some extents. Then you can just load the snapshot in an empty registry and you've your local copy to compare with.

Bullet 2. Sparse sets consume memory. If you don't need them, if you don't need all the features of a registry, I wouldn't recommend to keep up and running a copy of a registry. I personally use multiple registries though, so just a consideration: take under control your memory usage if you do it. :+1:

@vblanco20-1

There is also a 3rd option. You want to memcopy entity render data to ship into a few Rendering Systems to run in a pipeline with the rest of the engine. This is something i wanted to do myself but couldnt.

Why you couldn't? You can memcpy components at once with entity. What was the problem?

[pgruenbacher]

For bullet 1, I'm not sending snapshot over network. I'm saving a cache of the game state in memory, and then upon server updates, I reconcile the game state and fast forward. Basically a rollback.

[pgruenbacher]

But in order to achieve rollback, I need to keep a copy of the game state to fast forward from, since my sim code cant reverse operations

[skypjack]

Ok, got it. The question remains anyway: why snapshot doesn't work for that? It's slower than a memcpy, granted, but you can easily create a copy of a registry with it.
Another question so as to have a full overview of the problem: do you plan to make a deep copy of the whole registry or just some components? Approaches are different in the two cases.

[pgruenbacher]

full deep copy when doing the cache, but I'm willing to manually list the components during the clone call since it's pretty easy to know all the components that are part of it. if you think snapshot does the job I'm fine with trying it, I just wanted to make sure it makes sense.

[skypjack]

Tbh I've never tried it, but probably it could work with an archive implementation that just forwards everything to a loader.
I'll keep the issue open for a while anyway. I want to think to it twice to see if there is an easier way to do it.

[skypjack]

@Kerndog73 @pgruenbacher

I thought a bit about this. I see some problems with blindly copying only some component types from a registry to another one.

In particular:

• You cannot copy only the sparse sets. If you want it to work properly you must copy also the list of entities and destroyed entities of the registry itself. This will probably lead to orphans that is in charge to the user to clean up if required.
• You cannot copy things on a registry that isn't empty, unless you decide to completely overwrite it. That's because in all the other cases there is the risk to have inconsistencies on identifiers. As an example, you copy component A at step 1, then component B at step 2. If an entity identifier has been recycled and was assigned to A at step 1 and to B at step 2, what you obtain is a broken instance of a registry.

For point 2, you can still move-assign an empty registry to the target one and it will erase everything at once. Then you can copy-assign the pools you want. Point 1 is arguable the fact that an hypothetical full/partial copy function should also get rid of orphans. It means to waste CPU cycles for something that could be pointless, if all what an user wants is to put aside some (let me say) constant memory to reuse later.

Btw at a first glance it's possible to implement a partial copy of a registry with the limitations above mentioned. I'm still not that sure it's a good idea to do that in a real world software.
I mean, if all what I want is to create a memory of some components for a step back, I'd just use a dedicated component. In case of multiple steps back, well, in case the number is fixed it doesn't change much and dedicated component could work fine as well, otherwise I'd rather use a different data structure.
The problem is that a registry occupies more memory than what you would need for a backlog, that's all.

What about?

[pgruenbacher]

The point 2) is no longer relevant to me really. I'm moving most engine code into my simulation code (rendering and maybe even physics) and using dual/triple buffers to handle.

The point 1) still stands in terms of a backlog cache for fast-forwarding. I'm not sure how copying entities of registry along with all of the components would be an issue? All the data is laid out easily enough to copy it seems...

std::vector<std::unique_ptr<sparse_set<Entity>>> handlers;
std::vector<std::unique_ptr<sparse_set<Entity>>> pools;
std::vector<entity_type> entities;
size_type available{};
entity_type next{};

Kerndog's implementation seemed to cover it after looking at it again. Handlers shouldn't be copied cause I think they have pointers by their nature, they'd just be re-implemented on the other registry after it's been created and copied to. But once a registry's entities and pools are copied into a new registry, both registries are now completely independent of one another. no pointers. no containers. if I were to run a simulation step on each registry, they'd both get the same results deterministically without affecting each other. To achieve determinism requires pretty much copying everything, so might as well copy everything at that point. The whole point of a simulation registry state is to contain all information that will reproduce a simulation, other stuff like caching should really happen outside of it.

and again to clarify the idea of the networking. https://i.stack.imgur.com/Ya1JT.png I'm probably doing a mix of lock-step. let's say i have 10,000 agents simulating, each has a position. I'm at time 5. I send a message to the server that I want to move 5,000 agents of group 1 relation to a circle goal. I send message to server saying "give all agents of group 1 circle goal action" I don't do anything client-side yet other than maybe audio + visual response to the action. For times 6,7,8,9 I get server messages saying "no new actions". I simply step the client simulation. At time 10 everyone get message back from server saying "player 1 is moving all agents of group 1 to circle goal" My client simulation starts simulating the time 10 step with the new player 1 (me) action applied. Because it's time 10, I also make a copy of the simulation state in memory. ... more steps go by At time 19 I get an packet from he server with positional data for agents 1-100 at time 10. I reconcile that positional data on my copy of time 10, and then run that copy up to time 19. I discard my original state since it's invalid. If there were any differences between server and user simulations, the user sees rubber-banding for agents 1-100.

The point of this sort of networking is to keep server/client entity states corrected with only infrequent updates and mostly locks-step actions, without having to worry about having perfectly deterministic simulations (no floating-point issues, no fatal desyncs). sorry for wall of text.

[vblanco20-1]

Gonna go a bit more in-depth with the "for pipelining" use case of registry clone. With an ECS, paralelism is already quite easy to do. You can fairly trivially run systems at the same time, and you can also use parallel for. But both of those have sync points where all the threads need to wait for other to finish.

When you run 2 systems in parallel, the final time would be whatever takes the most time to execute, since one thread ends early, and just needs to wait until the other one finishes.

When you perform parallel-for, there is allways going to be a case where some tasks are longer than others, and they take time to execute. A bigger number of tasks can allow less "difference" becouse it averages out, but then you need to deal with bigger task overhead wich is very bad. This makes the case where 7 threads are waiting for the 8th of them to finish its task.

Apart from that, adding and removing entities and components can be unsafe, so you need to put them in a queue or similar to "defer" them, and then do it singlethread. While this happens, all the other hreads cant do anything.

A very common pattern, used everywhere, is to do pipelining. In a ECS engine, you can have a "simulation" thread, and a render thread. The simulation does almost everything, and then the rendering thread prepares the DirectX or similar calls to send to the driver. If you pipeline simulation with rendering,all the cases ive talked above (where stuff is waiting) will be filled with the tasks from the rendering. This can be done with audio, but luckily audio doesnt need nearly the same amount of interoperability with the ECS itself. Similar case with asset loading.

If you can do a "fast" copy of the simulation registry into a rendering related registry, then you can pipeline your rendering with your simulation almost trivially. I wanted to do that in my own experiment, but got bottlenecked on those sync points i commented about.

Of course, this can be fairly niche, and its not that easy to do the pipelining properly, but it does allow you to implement pipelining, while it wasnt possible to do before.

[skypjack]

To keep track of the discussion on gitter, I add that persistent views can be a problem in case of partial copies - ie a view for A/B would break in case an user decides to copy only one of the components.

Possible solutions:

• Do not copy persistent views.
• Runtime asserts in the codebase and warnings in the documentation.
• Copy only acceptable persistent views if possible.

[skypjack]

I'm taking a look at this these days. Maybe I found a suboptimal but less impactful solution. Let's see.
The API will be something like this:

a_registry.clone<A, B, C>(another_registry);

I must admit that clone doesn't sound good and I'm open to suggestions for a proper name.
The list of components is something in which I'm interested because it's unlikely that I want to copy all the instances of all the components with such a tool. I mean, it's something that rarely makes sense at all and this approach works in both cases, so...

As a side note, this function won't support filters as it happens now for the persistent views. It's a bit tricky to introduce them and I cannot figure out a case in which it would make sense to use filters and for which a slightly different approach doesn't solve the problem in an easier way.

I'm looking forward to comments and feedback. :+1:

---

Well, probably copy is just fine for the naming part.

[vblanco20-1]

I think that "WIP API" looks perfectly fine, i like how its just obvious. Clone as the verb is precisely what you are doing with the word, and then the logical type list of the components. There is only one thing i think on that, and is a way to maybe typedef the typelist.

For example: ` //somewhere, maybe on the top of a file or in a class? using RenderTypelist = entt::componentlist<A,B,C,D>;

//later down the line, as part as the update code a_registry.clone(another_registry);

//what about...

using TransformComponentList = typelist<Position,Rotation,Scale,Transform,RenderTransform>; using RenderableComponentList = typelist<3Dmesh,Sprite,Material,OpenglState0>; using GeneralComponentList = typelist<Parent,Static,Disabled,Name>; a_registry.clone<TransformComponentList, RenderableComponentList,GeneralComponentList>(another_registry); `

The type list of the clone is very likely to grow into a lot of components, so being able to "define" them somewhere for usage later over different clones would be VERY useful. Of course, this is just syntactic sugar, and would be doable with macros.

[pgruenbacher]

I'd benefit from that alot ^ https://i.kym-cdn.com/photos/images/newsfeed/000/176/345/ALOT2.png?1316561264

[skypjack]

@vblanco20-1 @pgruenbacher

At a first glance, there are several places where a type list could replace the raw parameters list.
This isn't even the first time someone asked for something like this.

From my point of view, consistency of an API is important, so I won't do that only for a very specific case and ignore all the others.
Does it mean that I won't do that? Actually no, it means that it makes sense and I'll do it throughout the whole entity part.
There is a trivial trick that can be used to avoid breaking backward compatibility.

Probably this would deserve also its own issue.

[skypjack]

I see a problem with the cloning functionality. Probably something that can be solved easily. Roughly speaking, the destination registry must be empty, at least the pools of the components to clone.
The loader member function solved the problem by cleaning it up before to return. The same approach can be used here. Btw all the entities and the components are erased in this case.

[skypjack]

Please, see branch experimental, in particular the registry::clone member function.
I'm looking forward to your feedback before to merge it on master. Thanks.

[pgruenbacher]

I checked it out and it compiles fine for me. unittests worked. emptyRegistry.clone(registry) clones no components which is to be expected, though Idk if we want to be more explicit in saying sizeof...Component > 0. Idk how useful it is cloning a registry of just the ents and not any components...

I'm trying to test to see if a full_clone method was possible (clone all component pools) without having to explicitly list the components. This may be possible by iterating over the pools that exist with valid pointers, but I'm having trouble at the moment figuring out how to do the valid copies and instantiation, simply because understanding the internal workings of registry is a bit complex for my testing timeframe.

full_clone should help with a) large registries where everything needs to be copied for simulation replication purposes and b) dynamic components too

From what I can tell, copying the component pools without using a template list would mean having the component pool tracking the size of it's component type internally, and then having the registry just do static copy of the data blindly. Pretty messy.

[pgruenbacher]

looking around a little more, full_clone doesn't seem possible actually. disregard. type list compatibility though would be useful I think, even better will be to allow for multiple type lists to be passed in. From what I can tell clone is missing the type_list functionality.

[skypjack]

Idk how useful it is cloning a registry of just the ents and not any components...

Fair enough.

looking around a little more, full_clone doesn't seem possible actually.

The main problem with a full copy is that the clone functionality should be pushed down to the sparse set in the form of a virtual method. However, sparse sets have only the requirements that components be movable. Therefore it would break a bit the design with two possible solutions:

• Get rid of the requirements and turn it to movable + copyable.
• Detect if a type of component is copyable and override the virtual function accordingly: if it's copyable everything works like a charm, if it's not... well, what would you expect? A runtime assert? An empty member function that does nothing? Definitely fragile.

even better will be to allow for multiple type lists to be passed in

Good point. I'll add some code to merge type lists.

From what I can tell clone is missing the type_list functionality.

Everything is missing type list functionality. I've yet to introduce them all around and I can guarantee clone will be involved by that change. Btw that's another issue. I prefer to keep the two sets of changes separated.

[BrodyHiggerson]

Just found this issue when searching for clone/copy.

In my use case, I require a full/deep copy operation - I am holding a different perspective of the game world for every unit (as their views are warped by the speed of light and the resulting delays in information), where that view includes a full ECS copy.

As time passes, events arrive at each unit and I update their own ECS instances based on the event. I cannot share an ECS instance (entt::registry) among the objects - modifying a component in one must not touch that same component in another.

I'd be interested in any ideas/suggestions on this.

[skypjack]

After a discount on gitter, I had an idea (inspired by what @Kerndog73 proposed initially, but without the CRTP stuff) to get rid of the type list for deep copies. Hopefully, it will be consistent and won't need to change the moveable-only requirement for components. :+1:
Let's see if it works. I'm reopening the issue to keep track of changes and have feedback.

[skypjack]

Please, see branch experimental. Feedback are really, really, really appreciated. :+1: