Design a Speedrun Timer Synchronization Protocol (STSP?)

[CryZe]

In order to visualize LiveSplit One as a Twitch extension we need to synchronize the runner's timer to the browser. Additionally we want to synchronize information between different LiveSplit Ones when racing with each other. So we need to design a synchronization algorithm. We want to design it in such a way, that splits i/o is using it too for their racing feature. And they want it to be timer agnostic, so it needs to support "dumb clients" that don't necessarily track everything properly or can even parse any split formats. However proper timers such as LiveSplit should synchronize pretty much everything without any loss of information. So the protocol needs to be flexible in this regard where each client can specify their capabilities.

Here's some rough ideas of how the protocol could look like (ignoring encoding for now):

struct ClientCapabilities {
    can_parse_splits_of: Vec<String>,
    wants_splits_for_every_attempt: bool,
    wants_rough_info_for_every_attempt: bool,
}

enum Event {
    StartAttempt(Option<Splits>),
    DiscardAttempt,
    ResetAttempt,
    FinishAttempt,
    UpdateSplit(usize, Time),
}

enum Splits {
    Splits(Vec<u8>),
    RoughInfo(SplitsInfo),
}

struct SplitsInfo {
    game: String,
    category: String,
    attempts: usize, // This corresponds with the attempt count BEFORE the attempt started.
    segments: Vec<SegmentInfo>,
}

struct SegmentInfo {
    name: String,
    pb: Time,
}

struct Time {
    real_time: Option<TimeSpan>,
    game_time: Option<TimeSpan>,
}

We absolutely need to design this together with the splits i/o people and need to do this really soon as they are beginning to diverge too far.

The idea so far is that synchronizing a single attempt isn't too hard, we just need to update the times of individual splits until eventually the attempt is done. Then either the client is smart enough to update its information about the PB, possible history, etc. or we send an absolute snapshot of the current information again. Based on the client's ability of parsing the absolute snapshot, we either send some rough information or the full splits file. What may not be properly considered in this design are proxies, such as splits i/o that may not be "sufficiently smart clients", but need to serve the information back to clients that may be.

The initial design is mostly for observing timers, but you may also be interested in controlling the timer, so that's something that is probably going to be part of the protocol eventually too.

[glacials]

Cool! Some thoughts:

If the Splits.io exchange format is in good shape (or if we can get it there), we can get a lot of this stuff for free with something like JSON Patch. That way we won't need to maintain both a format for state and a protocol for events (the "events" would be defined by the state format). Plus clients won't need to know both how to read the state format and how to apply changes to it that come from events; they just constantly display their current state and blindly apply updates to it using a generic patching library.

Otherwise I would be very concerned about timers staying in sync if an event fails to process / implementation difference causes ripple effects in the run / etc.

The part about letting clients smartly tell you whether they're too dumb to update a run with partial information vs. needing full state seems over-engineered to me. The dev work to support a protocol that exchanges this information is probably about similar to the dev work to implement partial run updates. But that's assuming we have a generic way to apply the updates like I mentioned above, rather than dozens of event types needing custom code.

Also I assume that there will be a source of truth here, a place clients connect to receive these updates (much bigger conversation if not). Is that Splits.io? We can use our existing WebSockets connections to send run updates.

[wooferzfg]

We should consider the stuff mentioned in this issue as well: https://github.com/LiveSplit/livesplit-core/issues/90

[CryZe]

Yeah that would need to be implemented first on our side as otherwise we can‘t control real time at all.

[wooferzfg]

Summarizing what we discussed today (please correct me if any of this is wrong):

Each client will request the splits info in a specific format (LiveSplit, WSplit, Llanfair, Splits.io exchange format, etc.). If the server doesn't support that format, it should return the splits info in the Splits.io exchange format.

For synchronization, we'll mostly keep the event system in the proposal. In addition to that, we need to handle some other cases:

• In order to synchronize Game Time, we need to constantly send the current Game Time to the server and to other clients. If this doesn't end up being feasible, we'll need to come up with another solution.
• Splits.io's new "Split" button in races supports adding new segments in the middle of a run. Splits.io can handle this by still sending UpdateSplit for any new segments that are added and having a separate endpoint for adding a new segment during a run. LiveSplit can then ignore UpdateSplit events for any segments that don't exist in the initial split info.

[glacials]

👍 retracting my previous comment, as one of the core goals I wasn't aware of is to allow two timers to communicate extra information over the protocol that other timers, formats, etc. don't yet know about. e.g. if LiveSplit adds support for custom variables and releases a new version, they want LiveSplit <-> LiveSplit communication to work with new custom variables immediately without relying on the Splits.io exchange format updating to support them.

From a high level, here's the stuff that was not obvious to me when initially reading this thread.

Use case 1: PC game to PC timer (local computer only)

A PC game has native support for sending gametime to speedrun timers.

┌User's computer───────────────────┐
│ ┌──────────────────────────┐     │
│ │         PC game          │     │
│ └──────────────────────────┘     │
│               │                  │
│               │                  │
│               │Sync protocol.    │
│               │Game connects to  │
│               │localhost:PORT    │
│               │                  │
│               ▼                  │
│ ┌──────────────────────────┐     │
│ │        LiveSplit         │     │
│ └──────────────────────────┘     │
└──────────────────────────────────┘

Use case 2: PC to phone

Runner is streaming a fullscreen game with only one monitor. They have LiveSplit on their computer to autosplit, global hotkey split, and/or include it in their stream, but can't see it themselves. They use their phone as a timer display, optionally allowing it to be a control device as well.

┌User's computer───────────────────┐
│ ┌──────────────────────────┐     │
│ │        LiveSplit         │     │
│ └──────────────────────────┘     │
│               │                  │
│               │Sync protocol.    │
│               │LiveSplit         │
│               │connects to       │
│               │IP_ADDRESS:PORT   │
│               │after local wifi  │
│               │discovery of      │
│               │timers            │
│               │                  │
│               │                  │
└───────────────┼──────────────────┘
                │
                │
                │
                │
┌───User's phone┼──────────────────┐
│               │                  │
│               │                  │
│               │                  │
│               │                  │
│               ▼                  │
│ ┌──────────────────────────┐     │
│ │        LiveSplit         │     │
│ └──────────────────────────┘     │
│                                  │
│                                  │
│                                  │
│                                  │
│                                  │
└──────────────────────────────────┘

Use case 3: timer to timer

The runner is racing and wants to see comparisons from others in the same race.

┌User's computer───────────────────┐
│ ┌──────────────────────────┐     │
│ │        LiveSplit         │     │
│ └──────────────────────────┘     │
│               ▲                  │
│               │                  │
│               │Sync protocol.    │
│               │LiveSplit         │
│               │connects to       │
│               │splits.io:PORT    │
│               │                  │
│               │                  │
│               │                  │
│               │                  │
└───────────────┼──────────────────┘
                │
                │
                │
                │
┌──Splits.io────┼──────────────────┐
│               │                  │
│               │                  │
│               │                  │
│               │                  │
│               ▼                  │
│ ┌──────────────────────────┐     │
│ │       Timer server       │     │
│ └──────────────────────────┘     │
│               ▲                  │
│               │                  │
│               │                  │
│               │                  │
│       ┌───────┼───────┐          │
└───────┼───────┼───────┼──────────┘
        │┌──────┴──────┐│
        ││             ││
        ││             ││
        ▼│             │▼
  ┌Other │acer──┐ ┌Othe│ racer──┐
  │      │      │ │    │        │
  └──────▼──────┘ └────▼────────┘
  ┌Other racer──┐ ┌Other racer──┐
  │             │ │             │
  └─────────────┘ └─────────────┘

Use case 4: * timer to Twitch extension

The runner is streaming and wants a timer shown as a Twitch extension.

       ┌User's computer───────────────────┐
       │ ┌──────────────────────────┐     │
       │ │        LiveSplit         │     │
       │ └──────────────────────────┘     │
       │               ▲                  │
       │               │                  │
       │               │Sync protocol.    │
       │               │LiveSplit         │
       │               │connects to       │
       │               │splits.io:PORT    │
       │               │                  │
       │               │                  │
       │               │                  │
       │               │                  │
       └───────────────┼──────────────────┘
                       │
                       │
                       │
                       │
       ┌──Splits.io────┼──────────────────┐
       │               │                  │
       │               │                  │
       │               │                  │
       │               │                  │
       │               ▼                  │
       │ ┌──────────────────────────┐     │
       │ │       Timer server       │     │
       │ └──────────────────────────┘     │
       │               │                  │
       │               │                  │
       │               │                  │
       │               │                  │
       │    ┌──────────┴─────────────┐    │
       └────┼────────────────────────┼────┘
            │                        │
            │                        │
            │                        │
            ▼                        ▼
┌─Viewer's Twitch page─┐ ┌─Viewer's Twitch page─┐
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │Local observe-only│ │ │ │Local observe-only│ │
│ │timer             │ │ │ │timer             │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
└──────────────────────┘ └──────────────────────┘
┌─Viewer's Twitch page─┐ ┌─Viewer's Twitch page─┐
│ ┌──────────────────┐ │ │ ┌──────────────────┐ │
│ │Local observe-only│ │ │ │Local observe-only│ │
│ │timer             │ │ │ │timer             │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
└──────────────────────┘ └──────────────────────┘

[YaLTeR]

Hey, just came across this issue. We've actually used a kind of timer synchronization and control for our SourceRuns marathon stream timer for the past 3 marathons already, so I guess that information might be useful. Our stream setup is that the runner streams to one of our RTMP relays (or just Twitch if there are issues), our host grabs that feed as a video source into the main stream.

The problem we're solving: there should be a timer on the stream in the layout, and its starting and stopping should be synchronized to the runner's LiveSplit in such a way that neither they nor the host has to think about it during the run. When the runner's LiveSplit starts, it should start, when it stops (manually or via autostop) it should stop. Additionally, it should account for the stream latency and allow manual override in case the runner doesn't use LiveSplit or in case of technical issues.

To this end, I wrote a small command relay server https://github.com/YaLTeR/network-relay (warning, old Futures) and a LiveSplit component https://github.com/YaLTeR/LiveSplit.NetControlClient, and my friend made the web control UI for the host https://github.com/Matherunner/livesplitcontrol.

The relay server generates a unique password for the runner and displays it in the host UI. The runner uses the password to connect to the relay server through the LiveSplit component. The LiveSplit component then proceeds to send start, stop, pause, etc. events to the relay. The relay, uh, relays the events to all connected listeners (the host UI being one of them).

The web UI has a timer (using LSC!) that reacts to the events accordingly (starts, stops, etc.), used through the OBS browser source, and a control panel for the host with manual override buttons (the same start, stop, etc.). Additionally, the host can set the latency: the host asks the runner to show their LiveSplit on stream and start it, then hits the "OFFSET" button as soon as they see the timer start on the video source. After that, all commands from the relay are delayed by this time, resulting in close to perfect timer synchronization to runner's stream.

To sum up, what we're using (assuming we keep our relay server responsible for relaying purposes):

• Runner's LiveSplit connects with a password, then sends start, stop, etc. events.
• Server LiveSplit reacts to these events (with a configurable latency, which is its own event).
• Manual button presses from the host are sent as regular events into the relay so there needn't be any special support for those.

[DarkRTA]

A possible use case for this I had thought of would be having LiveSplit One's OBS plugin be more of a dumb client that connects to the actual timer and pulls the state of the current run from it.

This will allow for all sorts of different use cases including being able to use a completely different layout on stream. (In some cases the runner may want to see more info on their layout even if space on their stream is constrained)

[CryZe]

This is blocked by #807, but we should be able to work on this soon.