Feature: Spectator delay

Add some sort of parameter that shows spectators the game delayed by some number of seconds. It's a useful feature that exists in many other games for things like tournaments. It's a bit MTR, but now the engine development is active and more approachable, perhaps someone will pick up the project. Having the idea written down somewhere seems good.

I don't know how exactly it would be configured. It could be in host configuration, or an engine option, or even something set locally when spectators join by unsynced luarules. The basic version of the feature itself seems simple enough given the way rejoin works ("just" stop "rejoining" spectators 30s behind the game, rather than when its live), but there are a bunch of annoying things to deal with to polish it. Chat doesn't work on a delay so the engine would need some sort of "unsynced chat" for spectators to use when delayed. There is also the question of what to do when a player becomes a spectator, or a spectator turns into a player.

This has been considered in the past when we had run our main tournaments. It would be a huge help. The main purpose of spectator delay is to prevent spectator cheating. For that to be robust, the client machine must not be fed the sim frame updates sooner than the spectator delay - so a client side solution is not acceptable from security stand point. If the data is on the client's machine, you have to assume you cannot guarantee to protect it from misuse.

Problems with any attempt on the current network stack:

Spectators will find any in-match spectator chat or drawing will be delayed by the spectator delay. All sim frames for the duration of the delay have already been locked in. This effectively means all the social aspects of being a spectator is removed when spectator delay is in effect.
Spectators are really players as far as the engine is concerned, so their connections have to obey the same rules - like reporting frame checksums for example. The spectators will appear to be lagging behind the game by the spec delay.. which is correct, but looks weird.

To solve these problems properly would require a new network stack because spectators go from being players to being something completely separate and they have to combine the normal game messages with any spectator-specific bits. This would also get complicated if spectators are supposed to be able to interact with players - such as training sessions for example. Not sure, that the mess and instability that would cause is worth it.

One solution that seemed sensible, would be to use a relay server. So another Room/Lobby would be able to get the match relayed to it. I've been informed that this approach used to be possible, if that's true that may be the best approach to achieving this feature. A feature that I agree would bring a huge benefit.

For that to be robust, the client machine must not be fed the sim frame updates sooner than the spectator delay - so a client side solution is not acceptable from security stand point.

I don't think such robustness is strictly necessary. There are a lot of ways to cheat if you are at the level of intercepting packets and processing them into a live game. For example, the player being fed information by the spectator could just disable LOS on their copy of spring. I think an implementation that has all the packets but just doesn't sim up to the present will solve essentially all cases. Hardcore cheating can be dealt with via moderation.

The social aspect of spectating seems like the main problem, and requiring a special server setup to do delays seems like a big hassle when the feature could theoretically be seamlessly integrated from a user perspective. Spectators don't have to go to being something "completely different" though. What if we "just" split chat and map markers out into their own network thing, one that sends to everyone no matter when they are in the game, rather than as part of the "synced" network processing.

As a side effect, players who are reconnecting would be able to chat with their team as they reconnect, which is useful to say how their reconnect is going, whether to pause, etc. Spectators and players talking at delay wouldn't be a problem because I assume spectators wouldn't be able to talk to players in games with spectator delay. There is an issue though: that player messages should be "synced" for their ingame chat to make sense to spectators (and to not be a spoiler). And replays need to contain chat. So another network stack is pretty tricky.....

Here you go, perhaps spectator chat could be sent to the present (as happens currently), but also read from the present? So a spectator that sends a message at minute 5 will actually synced-send it with the frame at minute 6, but that's fine, because the clients of other spectators are set to instantly display spectator chat that arrives in the present, even though they are otherwise seeing the game as it was a minute in the past. Then we just have to not display the spectator chat that is processed along with the frames that spectators can actually see.

As a programmer, without knowing the code, i can obviously only speculate.

Regarding robustness, i think it needs to be strict enough to avoid bypassing the delay via lua or other easily available inject methods. Dealing with professional cheat software writers is another issue.

I certainly think something has to change in the networking. But i propose something like a shim, before the usuall networking system, that can receive a different kind of stream data, decode it and feed that to the normal network system. Imagine each client records his own replay onto the drive. This replay , could be streamed , after the set delay, to a connected spectator client. The advantage of this is, replays are already recorded. Also every spectator connection can be spread out to the players to avoid overloading. Replays could also be stored as "blocks" , that are sent out a parts. Each block could also be getting a light encryption and the key is sent at the end of the block. That would allow sending a time block ahead to avoid stutter in spec mode. For spikey data connections. This might be an unnecessary complication of the system.

Regarding chat. Of course the player chat and spec chat needs to be seperate. Anything is pointless since they are seperated by
a time delay from the players anyway. Put the spectators in their own life chat. That is not recorded. Put the player chat into the replays, if thats not done yet.

Advantage of a peer to peer replay stream system is that i does not need server bandwith. But it can easily be tested by streaming replays from a stream source. Because its a server-client system after all. Just hat peers are randomly selected as a stream source, not the BAR servers. It seems fairly simple todo because it does nto interference with the gameplay networking of players. Also this can run on a seperate thread. Therefore not impacting the game, IF there is another unused thread on the PC.

Whats needed is a spec mode identifier for ppl connecting to the game. That puts them into a startup screen where the time delay for the stream is counting down. Afterwards the data is streamed to a client by a randomly selected player. Dont know how later joining spectators will handle this, but i assume they need a full catchup the way the engine is written right now.

I think this feature is quite needed. As stream spec cheating is very easy todo in this time. VPNs are easy to use, second PCs are easy to come by. Even an old laptop will do for this purpose. I bet its happending in every other 8v8 random game. It will skew balancing data too.

I think you need to track down a few more details about the networking. Eg bandwidth isn't a problem, and spectator delay would happen below the level that lua has to care about. Since midgame catchup already exists, I think it would be relatively easy to make a quick-and-dirty spectator delay which sets spectators to only "catch up" to five minutes behind the game. It wouldn't be be pretty, and it would have the inter-spec communication issues TK raises, but it would work as a proof of concept.

beyond-all-reason / spring

Feature: Spectator delay #706