Documentation on delta for GoldSrc demo

[khanghugo]

Great work on this.

I am trying to write my own demo writer but now I am working through delta. I hope you won't mind writing documentation on it since apparently you are the best at writing docs for GoldSrc demo as of now.

Regardless, I am still plowing through it by reading your delta decoding snippet along with this and this.

[cgdangelo]

Sorry for the delayed reply! Thanks for the kind words. ❤️ I'm really glad you've found the documentation helpful.

Unfortunately I have to admit that the fundamentals behind delta compression or even the bit packing/unpacking underlying it are still pretty opaque to me. I adapted hlviewer's implementation, mostly just to get something working with fp-ts ADTs. I think that project is much more readable as a reference for the decoding side:

• https://github.com/skyrim/hlviewer.js/blob/master/src/Replay/readDelta.ts
• https://github.com/cgdangelo/talent/blob/main/libraries/parser-goldsrc/src/delta.ts

I'll take a stab at generalizing what I do know (or what I think I know, at least 😄), but I've only had to decode them relying on the hlviewer implementation. Encoding from within a client or just trying to write the structures to file might be a little more nuanced.

---

Delta compression in GoldSrc

• Delta compression roughly encompasses:
  • Serialization / compression strategy for representing a bunch of key-value pairs as a series of bits.
  • Communication protocol between client and server.
  • Extensible system for defining custom data structures? Mods and stuff.

Delta encoding

• A delta encoder is a data structure with a name, list of fields, and optional transformations (or possible side effects) to be performed while encoding.

• Individual delta encoders look like they're defined in <mod>/delta.lst, along with their fields.

  Not 100% sure on what the header implies:

  // none == no conditional encode routine
  // gamedll routine_name : before transmitting data, invoke the named function from the game .dll to reset fields as needed
  // clientdll routine_name : same as above, except the routine is called via the client.dll

  My guess is that a client would need these to know how to encode the deltas.

• Each field has:
  • Name: human-friendly identifier.
  • Data type flags (DT_*): describes the type of the value.
  • Bitsize: total number of bits needed for field value?
  • Default value maybe? Not sure on the last value.

Delta decoding

• Delta decoding is performed by clients to read the structured delta data from a server.
• Deltas are decoded in response to specific SVC_* messages from server.

Client/server communication

Big guesses to follow here...

1. Server loads delta encoders from <mod>/delta.lst.

• Presumably the server has to read this file to know how to encode.

2. Client connects to server.

• Client joins a server.

3. Server sends SVC_DELTADESCRIPTION messages.

• This provides a definition of all the necessary delta structures to clients.
• Shortly after connect, clients get a SVC_DELTADESCRIPTION message for each of the <mod>/delta.lst deltas.
• Clients parse the message: https://github.com/cgdangelo/talent/blob/main/libraries/parser-goldsrc/docs/demo-structure.md#svc_deltadescription-14.
  • The message contains a delta name, and each field's metadata represented as a list of individual delta_description_t deltas.
• Client stores the delta description so it can perform a lookup by delta name later when other SVC_* messages are received and will decode them according to the metadata in the description.

4. Server sends SVC_* messages containing specific deltas.

• Some SVC_* messages can contain 1 or more delta structures.
• Client reads these as they come in and updates its local game state.
• AlliedModders wiki has some extra context about when events are sent: https://wiki.alliedmods.net/Half-Life_1_Engine_Messages.

Message	When
SVC_CLIENTDATA	Every frame. Has client data, weapon data from the previous frame.	Message structure: SVC_CLIENTDATA.
SVC_EVENT	When some specific game events happen; can be queued / dropped.	Message structure: SVC_EVENT.
SVC_EVENT_RELIABLE	When some specific game events happen, but can't be queued / dropped.	Message structure: SVC_EVENT_RELIABLE.
SVC_SPAWNBASELINE	When entity spawns for the first time?	Message structure: SVC_SPAWNBASELINE.
SVC_PACKETENTITIES	Initial entity state?	Message structure: SVC_PACKETENTITIES.
SVC_DELTAPACKETENTITIES	Every frame? Same as SVC_PACKETENTITIES but this compares to a previous frame somehow.	Message structure: SVC_DELTAPACKETENTITIES.

"Reliable" terminology harkens back to Quake netcode stuff I believe: https://fabiensanglard.net/quakeSource/quakeSourceNetWork.php.

Compression algorithm

Bitpacking

There is bitwise magic afoot here, so I've never had a great understanding about this.

hlviewer is an excellent reference: https://github.com/skyrim/hlviewer.js/blob/master/src/BitReader.ts#L10-L48.

Roughly I believe it works like:

• You have x amount of bits in a byte.
• Using some mask value you can read n number of those bits to read a value.
• You have x - n bits left in the byte to store more data.

If a message is finished writing, usually it will fill up to the next byte boundary.

Reading a delta

• https://github.com/skyrim/hlviewer.js/blob/master/src/Replay/readDelta.ts#L5-L88

[khanghugo]

Hey, thanks for the detailed reply. Since then I have found some (enough) details to understand the delta. Good enough to implement my own, and also a delta writer.

https://github.com/khanghugo/demosuperimpose-goldsrc/blob/master/src/netmsg_doer/utils.rs