Atomic Bomberman is lagging

[Ceotom]

We have 2 players with following setup: 1 player: Hosts DOSBox ipxnet server (using DOSBox-X) 2 player: Connects to 1 player's ipxnet server

Both running Win10, who hosts in-game doesn't matter. Players teleports continuously and doesn't properly sync some events, like bomb placement, power-up pickups. Bombs can appear at wrong places or too late, power-ups may re-appear at same place after being picked up and etc. Video from 1 player side (red in-game): https://github.com/solemnwarning/ipxwrapper/assets/36301121/816715b9-57d1-4ac1-a1c6-39fa43489b80 You can notice how 2 player walks jerky 17s: 1 player pickups disease that forces player to set bombs continuously as soon as possible, but at 19s it reappears below bomb 25s: For some reason 2 bomb unexpectedly appears behind 1 player which is impossible normally (he can set only 1 bomb)

Logs with debugging messages (we didn't wait for the game to start because it takes VERY LONG to start with debugging enabled and we quit on "Receiving level data"): 1player.log 2player.log

During test with VMware VMs with this setup:

Host machine: Hosts DOSBox ipxnet server on Win10 VM1 and 2: Connects to host's ipxnet server using host-only adapter, running WinXP due to performance reasons, 2 VMs with Win10 are too slow.

I'm found that those issues appears only if simulate packet loss, ping just add delays until you action appear on other side.

Pinging showed that we has ~38ms ping and 0% packet loss. Looks like ipxwrapper drops packets?

[solemnwarning]

Hello,

As you can probably tell from looking at the issues list on here, I've been struggling with some people encountering lag or general weirdness but never managed to pin down the cause.

In your testing with VMWare, are both VMs running on the same host, on the same network or across the Internet?

I'm found that those issues appears only if simulate packet loss, ping just add delays until you action appear on other side.

What are you using to simulate packet loss?

Thanks

[Ceotom]

In your testing with VMWare, are both VMs running on the same host, on the same network or across the Internet? Both VMs running on same host.

What are you using to simulate packet loss? VMware lets you to set fake packet loss, ping or bandwidth in network adapter settings.

[solemnwarning]

So, the logs you attached were from an actual play session, not the VMWare packet loss test, right? They're suggesting packet loss may be going on (1player transmitted 3,182 packets vs. 2player receiving 1,242 packets), but some of that could be down to them not quite lining up time wise (i.e. one game being exited before the other).

[Ceotom]

Yes, those logs from actual play session, but when recording those logs we didn't wait for actual gameplay because the synchronization (Receiving level data) taken more than ~20 minutes with debug messages enabled, so we closed the game at almost same time.

Player 2 was host in this session, do you mean that player 2 fails to receive packets?

[solemnwarning]

Player 1 sent 3,182 packets and Player 2 received 1,242 Player 2 sent 1,007 and Player 1 received 976

[Ceotom]

Tested in VM (without packet loss and etc) and got those results: VM1 (host) sent 2785 packets and VM2 received 2782 packets VM2 sent 2693 packets and VM1 (host) received 2666 packets Also enabled debug logging doesn't has such a big impact on performance, sync finished quickly.

Something clearly wrong in real game session Logs: vm1.log vm2.log

Any idea where to dig?

[solemnwarning]

I spent a bit of time fiddling earlier. Bomberman is sending lots of small broadcast packets which seem to come through okay with the native IPX driver on XP, swapping in IPXWrapper degrades performance slightly, but no obvious cause. Switching to DOSBox encapsulation degrades it badly.

Sending lots of UDP packets like this isn't a "done" thing, and as a result the IP stack of any involved machine could be dropping, throttling or awkwardly shaping the traffic.

Depending on how bomberman processes traffic, I might be able to introduce a feature to coalesce frequent-but-small packets into fewer batched ones, which might help.

First though, I'm finishing off a profiling feature I was working off before which will hopefully clear things up.

[Ceotom]

Switching to DOSBox encapsulation degrades it badly.

We can't test if IPXWrapper UDP/real IPX performs significantly better right now, but will post results here when possible. This is DOSBox IPX tunneling issue or with current implementation in IPXWrapper? Never had issues with it in DOS games.

Depending on how bomberman processes traffic, I might be able to introduce a feature to coalesce frequent-but-small packets into fewer batched ones, which might help.

Is just adding some padding to make packets weight more may solve issue? This looks easier to implement. Or this not how it work?

First though, I'm finishing off a profiling feature I was working off before which will hopefully clear things up.

Do your best. It's great that projects like this exist.

[solemnwarning]

The issue isn't the packets themselves being small, but the sheer number of them. Bomberman is sending 400+ (broadcast) packets every second when in a network game (most of them <50 bytes), which is resulting in it spending a lot of time sending/receiving packets. I don't have profiling data to back this up, but I can only assume the original IPX driver in Windows was a lot more simplistic than the IP stack in modern Windows.

It seems to be spending around 1s inside the IPXWrapper sendto() function out of every 10 seconds, a good chunk of which is inside the OS sendto() call.

I experimented a bit with using WSASendTo() to send the packets using overlapped I/O instead, which theoretically offers better performance since the thread can return and be ready to queue up another packet before the first one has been fully committed into the socket buffer, but that actually seems to make things even slower, probably because of how stupidly tiny and numerous these packets are.

I'll have to try and find more optimisations that can be made elsewhere.

[solemnwarning]

So there are still places within IPXWrapper where I might be able to squeeze a bit more performance out where certain APIs are being used in terrible ways by Atomic Bomberman, but I still think the root cause here is the sheer number of packets.

I've attached a prototype build of IPXWrapper which batches and merges packets when the application starts sending them at ridiculous rates (~90% packet count reduction during AB gameplay). Can you and your friend give it a try and report how it works?

There is now a profiling option in ipxconfig you can turn on to gather useful statistics for this which shouldn't slow the game down like the full debug log does.

ipxwrapper-git.zip

[Ceotom]

It's a little better now, but still players still move jittery (but smoother than before) and bombs still may do not appear on other side in time. Synchronizing at beginning of match are faster now too

Prof Player 1.log Debug Player 1.log Prof Player 2.log Debug Player 2.log

While recording log with debugging enabled were able to reach gameplay, but players doesn't move at all. (While profiling enabled everything is "fine")

[solemnwarning]

Well that raises more questions than it answers.

Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster.

The stats in the log back this up too:

[988646.95, thread 5088] IPX sockets sent 31939 packets (1089730 bytes)
[988646.98, thread 5088] IPX sockets received 3009 packets (104774 bytes)
[988647.03, thread 5088] UDP sockets sent 31939 packets (2047900 bytes)
[988647.06, thread 5088] UDP sockets received 3009 packets (195044 bytes)

If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above.

[Ceotom]

Placebo effect in action ¯_(ツ)_/¯

On Monday, November 27, 2023 20:37:04 (+05:00), Daniel Collins wrote:

Well that raises more questions than it answers.

Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster.

The stats in the log back this up too:

[988646.95, thread 5088] IPX sockets sent 31939 packets (1089730 bytes) [988646.98, thread 5088] IPX sockets received 3009 packets (104774 bytes) [988647.03, thread 5088] UDP sockets sent 31939 packets (2047900 bytes) [988647.06, thread 5088] UDP sockets received 3009 packets (195044 bytes)

If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above.

— Reply to this email directly, view it on GitHub https://github.com/solemnwarning/ipxwrapper/issues/14#issuecomment-1828071454 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIU6SQKV3JAS7S6CVHD2G4LYGSXSBAVCNFSM6AAAAAA7RQVG6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRYGA3TCNBVGQ . You are receiving this because you authored the thread.

[Ceotom]

It really slightly better now, but not due to changes in wrapper.

On Monday, November 27, 2023 20:37:04 (+05:00), Daniel Collins wrote:

Well that raises more questions than it answers.

Going by the Prof logs there, the packet coalescing I implemented isn't actually kicking in (there should be a message along the lines of "High send rate to xxx detected, coalescing future packets"), so nothing else in my build should be significantly faster.

The stats in the log back this up too:

[988646.95, thread 5088] IPX sockets sent 31939 packets (1089730 bytes) [988646.98, thread 5088] IPX sockets received 3009 packets (104774 bytes) [988647.03, thread 5088] UDP sockets sent 31939 packets (2047900 bytes) [988647.06, thread 5088] UDP sockets received 3009 packets (195044 bytes)

If the packet coalescing was working, the UDP socket stats would show far fewer packets than the IPX socket stats. The counters suggest there is a lot of packet loss going on, which isn't too surprising at the packet rates above.

— Reply to this email directly, view it on GitHub https://github.com/solemnwarning/ipxwrapper/issues/14#issuecomment-1828071454 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIU6SQKV3JAS7S6CVHD2G4LYGSXSBAVCNFSM6AAAAAA7RQVG6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRYGA3TCNBVGQ . You are receiving this because you authored the thread.

[solemnwarning]

I haven't been able to come up with any reasons why the packet coalescing might not kick in, so could you try this build out and send me the logs with profiling enabled? Hopefully something in there will point at why it isn't working.

Thanks

ipxwrapper-git.zip

[Ceotom]

It's definitely working now and makes game playable, but there's still issues: Walking animation doesn't play on another player (no direct gameplay impact) Looks like sometimes bombs appears with significant delay on other side, which causes them to explode later. (You may see how player, who set those bombs walks on the blast of his bombs without dying) Setting multiple bombs in a row is laggy Thrown bombs is lagging (no direct gameplay impact)

Perf Player 1.log Perf Player 2.log Video from player 1 side

[solemnwarning]

Unfortunately, the Atomic Bomberman network code just isn't very good. The best I can do is tweak IPXWrapper to reduce the chances of packets getting lost and improve the speed of the calls it is (over)using - as you've seen, when packets are lost it can go out of sync and makes no attempt at correcting the state between players. The walking animation not playing isn't even specific to IPXWrapper - that happens when using the native Windows IPX driver on XP too.

I've made a few more optimisations in this build which might help a little bit, let me know how you get on.

ipxwrapper-git.zip

[Ceotom]

The walking animation not playing isn't even specific to IPXWrapper - that happens when using the native Windows IPX driver on XP too.

I understand that issues may appear on software/hardware game designed for.

We didn't noticed any difference with this build

[Ceotom]

We also tested game with 10 bots and it's looks unplayable if there's more than 3 players (for obvious reason)

Player 1.log Player 2.log

[insellium]

Hi. I was testing Atomic Bomberman with Ceotom a month or two ago. How is the fixing process going?

[solemnwarning]

@insellium I've not got any further leads/improvements to follow at the moment, so no progress on the Atomic Bomberman front.

[jangatzke]

We had a LAN party with 10 players using 5 computers. The game got async using windows and the ipxwrapper version linked in this issue. We switched to Fedora + Wine + same wrapper version and the game was super stable and without lags.