Sending array data

[LuisEGomezV]

Introduction

The current state of the MessagesSystem relies completely on unmanaged messages. This is perfect to avoid GC, but takes away the possibility of sending an array through the network inside of a message struct.  

Problem

  The need for sending array data through network, for synchronizing things like   • Inventory content • Creature stats • Player abilities set • Player skills • Player buffs • Etc.  

Solution

  One posible solution could be to have a 

Send<T,U>(int connectionId, T metaData, NativeArray<U> messageData); 
    where T:unmanaged, NetworkMessage
    where U: unmanaged, INetworkElementData 

method in NetworkServer System. Where T is a NetworkMessage as currently known. Used to have the data which is not required to repeat, like the message ID, or the netId of the target, etc. And U is a new INetworkElementData, for now, just an unmanaged struct.

The system will then serialize them together, and send them in one package

The receiver must also implement a system

class NetworkServerArrayMessageSystem<T,U> or class NetworkClientArrayMessageSystem<T,U>

To handle the message accordingly. The receiver could also have a pool of NativeArray<U> or reuse a single one, in the case of receiving the same message a lot.

[miwarnec]

Hey @LuisEGomezV , check out Unity's new Bytes16/Bytes510/Bytes4960 etc. structs. You can put them into the messages, and we still get free serialization. It's really cool.

Feel free to reopen if that doesn't work :)

[LuisEGomezV]

@vis2k Well, that might work for most cases, I don't think it is an ideal solution because most of the time it will have unused bytes, and also the data conversion OriginalData--> BytesX --> OriginalData but if there is not too many different "array" messages, maybe it should be fine. I'll try it this week and see how it works. Thanks

[LuisEGomezV]

Hi @vis2k I'm having issues designing a pattern to work around this. The problem I'm having is that those BytesX structs don't have an indexer, and I cannot find a writer tool for this structs. For cases when there is only a few bytes (16/32) and also I know at compile time the number of elements there is not big issue, because I can manually assign each byte. But in cases where the number of elements it's only known at runtime, I cannot make a loop or something to programmatically fill the bytes because of the lack of indexer

(If there is an existing tool or pattern to do this, please let me know, I did not find anything).

I think it's possible to work with this, but it is definitely necessary a solution for array data

[miwarnec]

Hi @vis2k I'm having issues designing a pattern to work around this. The problem I'm having is that those BytesX structs don't have an indexer, and I cannot find a writer tool for this structs. For cases when there is only a few bytes (16/32) and also I know at compile time the number of elements there is not big issue, because I can manually assign each byte. But in cases where the number of elements it's only known at runtime, I cannot make a loop or something to programmatically fill the bytes because of the lack of indexer

(If there is an existing tool or pattern to do this, please let me know, I did not find anything).

I think it's possible to work with this, but it is definitely necessary a solution for array data

as mentioned in discord, I don't have a good solution yet. we could consider a simple struct like BufferAndIndex with Bytes4096 + index that you mentioned. would that work?

[LuisEGomezV]

Not sure if I understand, you mean like encapsulating BytesX with BufferAndIndex and "fake" the Indexer with switch or if's ?

[ghost]

Is "having unused bytes" a concern? I don't believe DOTSNET/Apathy send over the wire the "unused end" of a large message like that, do they? I seems like the Send just increments the position after the blit by the correct amount?

[LuisEGomezV]

I don't mean the unused bytes on the bytes segment on the transport, I mean the unused bytes of the Bytes4096. The serializer algorithm will add every byte from it, even if not data is actually there, because it cannot know if you are using it, just serialize the whole struct.

[miwarnec]

next dotsnet version will have custom NetworkMessage.Serialize/Deserialize again. you can then send arrays like: write(array.length); foreach entry: write(array[i]);

thanks, closing