Improve benchmark performance

[cBournhonesque]

Benchmarks show that it takes 1.3 ms to replicate 1000 entities (replicon takes 30us). Why?

With a lot of tracing spans, it's 3ms (because of the tracing overhead):

• send_entity_spawn takes 530us

  • ReplicationSend::prepare_entity_spawn is 85us (because we are allocating. The overall memory should be re-used though. But the hashmap inside the replication_group_id is probably very inefficient?)

• send_component_update is 686us

  • ReplicationSend::prepare_component_insert is 178us
  • rest is probably iteration + serialization?

• networking::send is 1.13ms

  • buffer_replication_message is 790us
  • finalize is 115us
  • then there is serializing (not tracked)
  • buffer_send_with_priority is 170us (buffering into the message manager)
  • send_packets is only 335us
  • message_manager.send_packets (that collects the messages to send from channels and builds the packets) is 190us

Also here are the ChannelSendStats:

ChannelSendStats {
        num_single_messages_sent: 1000,
        num_fragment_messages_sent: 0,
        num_bytes_sent: 27000,
},

• maybe it's not optimal to send a lot of individual messages, because we generate one MessageId per individual message?
• here the stats don't even take into account the MessageId, it's just the raw message bytes. 27 bytes seems pretty steep!
  • 1-2 bits for ReplicationMessage (but it shouldn't be needed because we are in the EntityActions Channel!)
  • group_id = u64 = 8 bytes
  • 1 bit for Action vs Updates
  • 2 bytes for MessageId (the "sequence id" for the replication group)
  • the length of the vec: encoded with GammeEncoding so probably 1 byte. (here 2 bits)
  • the entity: u64 so 8 bytes
  • SpawnAction: I think only 2 bits?
  • insert: 2 bits for the length, 2 bytes for the ComponentNetId, 4 bytes for the float
  • remove: 1 bit for the length of the empty hashset
  • updates: 1 bit for the length of the vec = 24 bytes + 11 bits = 26 bytes (I don't know where the extra 1 byte comes from). That's pretty steep. The main reason is that Entity is 8 bytes. Maybe we could gamma-code it as 2 tuples? since both the index and generation should be pretty low? Another reason is that we encode both ReplicationId and Entity which are the same here.

Potential ideas:

• send_entity_spawn

  • uses a double hashmap to store data. In particular the allocated memory of the second hashmap cannot be recovered!

• networking_send

  • buffer_replication_message
  • we serialize twice because of bitcode quirks currently
  • we allocate new EntityActions/EntityUpdates message instead of re-using existing ones
  • serialize directly into a cursor without intermediate data structures? I'm not sure it's possible if we want to keep the ReplicationGroup guarantees, which replicon doesn't have Creating some entities ahead of time to re-use allocations in prepare_component_insert seems to bring a small (5%?) improvement. But what we can do is since we already buffered the per-replication data, the final message can be written to manually using a cursor
    • they write all entity-actions into one message, which might become big and have to be split up (bad if packet-loss). We have one message per ReplicationGroup. That's also why they can write in a cursor efficiently: they write all the despawns (with entity), then all the removals (with entity), then all insertions (with entity)
    • should we just update our message-packing?
    • all the component updates for an entity are iterated through sequentially (which shouldn't make a diff for this benchmark) so they can be serialized directly in order?

[Nul-led]

Did you actually run their benchmarks or did you just trust their results? :p

[cBournhonesque]

Ran their benchmarks :) I'm trying to understand why the difference can be so big. Probably because of extra allocations? but still

[cBournhonesque]

First optimization I will try:

• Instead of building a EntityActionsMessage (which allocates to store the component insertion data, and then gets re-serialized) into a buffer, we will try to write the EntityActionsMessage directly
  • 2 options: 1) have one intermediary buffer for each entity in the replication group, and then we will stitch all the buffers in the end. Requires extra allocations but reduces bandwidth because we don't write the same Entity multiple times. Basically for each entity, we will store [Spawn, Despawn, Removals, Insertions] 2) for the replication group, write all [Spawns], then all [Despawns], then all [Removals], then all [Insertions]. This has better performance because we can write the final message directly in one go without intermediary buffers, but it requires more bandwidth because we might have to write the Entity multiple times (once in Spawn, once in Removals, etc.)

Note that these new approaches also probably require more bandwidth, because bitcode was able to use bit-compression (write individual bits instead of bytes) for the previous EntityActionsMessage.

I will try b. first to have the biggest difference in performance

[Nul-led]

imo bandwidth is still the most important thing to optimize for multiplayer games. If its impossible to optimize the serialization due to an increase in bandwidth, having this tradeoff seems worth it. Ofc finding a balance is a good idea but i feel like optimizing for bandwidth first makes more sense.

[cBournhonesque]

Agreed that bandwidth is more important but this is still a massive difference: 40X ! I would accept something like 5 or 10X (i.e. 300us to serialize the 1000 entities), but 1.3ms is way too much

I think it's mostly due to 2 things:

• they send the EntityActions in one big packet, so they can serialize it directly. It works but then the packet is split up into multiple parts, which adds a bunch of latency if any of the fragment is dropped or delayed. Instead we send a lot of individual messages, which is more flexible but we use more CPU because we keep some data around in memory. (but maybe we can pre-allocate some memory or something?)
• they serialize their message directly while because of bitcode I serialize stuff twice, and have extra allocations. In general I think I want to move away from bitcode. It's good for bandwidth but it's too much of a headache compared with an easier serialization approach

[Nul-led]

Do you know how well lightyear compares to them in terms of avg bandwidth usage? Might be interesting to have benchmarks for that too..

Apart from that 1) sounds like a massive pain to resolve and would probably require huge internal changes as far as i can tell 2) what is your definition of "easier" serialization?

[cBournhonesque]

1. Is actually much easier to do than what i do now. Right now I do some complicated stuff to make sure that EntityActions are split up instead of sending them all as a giant message. I think the first thing i'll try is actually to just send all EntityActions in one giant message where I serialize stuff directly inside.
2. Something that uses the Read/Write traits, and doesn't use bit-level stuff

[cBournhonesque]

As a simple first step, replicating everything as a single ReplicationGroup brings the time to around 780us, which is a huge improvement. Probably because we don't allocate new space in the hashmaps, and we the vec allocations are more efficient.

Also instead instead of sending lots of small messages we send one big message, which made be more efficient for channel internals.

The full trace with all log spans is 1.9ms, with:

• send_entity_spawn is 495us, but most of the time seems to be spent on logging; will reduce the inner trace
• send_component_update is 626us, but most of the time seems to be spent on logging; will reduce the inner trace
• buffer_replication_messages is 76us, with 8us in finalize and and 1.46us in buffer_send_with_priority
• send_packets_to_client is 90us, with 9us in build-packets

If I remove the prepare_entity_spawn and prepare_component_insert tracing, I get 1ms with:

• in PreUpdate, handle_replicating_add is 190us (system 40us, commands is 150us)
• send_entity_spawn is 113us (price of inserting in the hashmap and extending the vecs)
• send_component_update is 180us (same)
• buffer_replication_messages is 83us (finalize is 10us, most of the rest is in the serialization)
• send_packets_to_client is 82us, (12us in send packets, among which 4.3us for build_packets; is the rest of the time spent in ServerConnection::send()? Why is it taking so long?). Also replicon doesn't seem to benchmark the renet packet building part.

So should we get rid of ReplicationGroups?

• maybe it's ok to send all EntityActions in one big message, fragmentation will be rare except for bigger games? Idk
• Or maybe we can have one default group where all entities are.
• Avoiding HashMaps and Vec allocations is key
• if we're ok with removing replicationGroups, how to guarantee that some entities states are linked