Expose diagnostic primitives for remote tooling

[heavyrain266]

Hi, I think that it could be really useful to have light abstractions or general serialization with e.g. nanoserde (I made it no_std last year).

Ex. you could use tracing + metrics to compose diagnostic packages and send them over to companion inspector that is either web or GUI app (see flecs). It is generally useful for remote debugging on consoles.

[zakarumych]

What exactly do you want to serialize? Content of the World or its internals (number and sizes of archetypes, entity <-> location mapping etc)?

The former can be partially done manually, but I was going to explicitly support entity serialization someday.

The later, I think, may be implemented as World::metrics function that returns current world serializable metrics structure that can be populated with whatever data makes sense.

[heavyrain266]

I was thinking about serializing entire World and processing the data with Apache Arrow or similar.

[zakarumych]

So you are interested in World content. It is fairly straightforward to serialize all components of all entities where component type define serialization process for the component instances. The Component trait can be extended to support serialization and deserialization. Same goes for resources. Then World could be serialized, excluding components that don't support it. On deserialization the World would be restored, if all component and resources types are serializable then it would be equivalent to the original.

Alternatively a user-defined set of serializable components of all entities may be serialized. This will produce easier to analyze serialized data, especially if binary format is used. It would be possible to deserialize data not into new World, but individual entities and components. The schema would look like Vec<(EntityId, Option<ComponentA>, Option<ComponentB>, ...)>

I think both kinds of World serialization are useful and both should be supported.

[heavyrain266]

Good explaination, thank you.

Despite that I already built a simple game with edict last year, I still feel new to ECS. At the moment I'm in the process of constructing architecture for AAA engine that will be used for linear action-rpg.

My interest in serialization is mostly for external tooling and data echange between game runtime and the editor. The idea is based on publications from Remedy Entertainment, that you can find here: https://youtu.be/FI2pyzTOvaQ

tl;dr: Remedy's "World Editor" is written in C# and communicates with game runtime in C++ by sending ECS commands through gRPC along with diffs that contains Pixar's USD format to synchronize changes in the World.

[zakarumych]

This use-case is somewhat similar to requirements for netcode.

I have another crate evoke (it's a bit outdated) that works with edict. It extracts data from the World using the second method (tuple of options) that it then serializes using trait implementation for those components. Basically it can extract data from the server and add, remove and modify any component by sending signals to the client.

With change-detection it can be made more efficient by iterating only over modified components, building a "diff". This is not exactly a diff, it can be applied to state not older that "diff" start version to produce "diff"'s end version which is perfect for unreliable data channel.

Probably this system integrated directly into edict would fit your use-case.

[heavyrain266]

Thanks a lot!