Fix replay generation

[tfausak]

I spent some time today putting all the pieces in place for replay generation. Everything is there now, but it doesn't work.

[tfausak]

The first problem is that I'm getting "Negative exponent" errors. There are only a couple places that use exponentiation: compressed words and vectors. So I changed anything of the form x ^ y to x ^ if y < 0 then trace "..." 0 else y. That revealed some troubling stuff.

putting int vector Vector {vectorX = 0, vectorY = 0, vectorZ = 0}
putting compressed word CompressedWord {compressedWordLimit = 1, compressedWordValue = 0}
putting compressed word CompressedWord {compressedWordLimit = 0, compressedWordValue = 9223372036854775890}

It's possible that those vectors are legit, but the compressed words almost certainly aren't.

[tfausak]

On the way in, there are a lot of vectors with bias = 2 and dx = dy = dz = 2. On the way out, I try to write them with bias = dx = dy = dz = 0, which doesn't work. I think the bias has to be at least 2.

For the other crazy high values, I'm guessing that I'm underflowing a Word somewhere.

[tfausak]

Yup. If an integer vector has any negative components, they cause underflow when I compute the deltas. fromIntegral got me good!

Main.hs: putIntVector
 vector     = Vector {vectorX = -53, vectorY = 23000, vectorZ = 82}
 fields     = [18446744073709551563,23000,82]
 least      = 82
 greatest   = 18446744073709551563
 difference = 18446744073709551481
 maxBits    = 64
 maxValue   = 0
 numBits    = 62
 bias       = 9223372036854775808
 deltas     = [9223372036854775755,9223372036854798808,9223372036854775890]

[tfausak]

Here's how that vector looks on the way in:

getIntVector
 numBits  = 14
 bias     = 32768
 maxBits  = 16
 maxValue = 65536
 dx       = 32715
 dy       = 55768
 dz       = 32850
 x        = -53
 y        = 23000
 z        = 82

After correcting for the underflow, this is what it looks like on the way out:

Main.hs: putIntVector
 vector     = Vector {vectorX = -53, vectorY = 23000, vectorZ = 82}
 fields     = [-53,23000,82]
 least      = -53
 greatest   = 23000
 difference = 23053
 maxBits    = 15
 maxValue   = 32768
 numBits    = 13
 bias       = 16384
 deltas     = [16331,39384,16466]

Note that dy is greater than maxValue, and maxBits and numBits are both too small by 1. So I did the math wrong somewhere.

[tfausak]

This is closer to working now. The next step is making sure that I can read the replay file I write. I currently can't do that. That means there are problems somewhere in the serialization process.

Once I can do that, I need to make sure the result from reading the replay I just wrote matches the original replay. If it doesn't, that means some values changed when they shouldn't have.

[tfausak]

I'm trying to write property-based tests for all the conversions, but it's getting annoying to encode the invariants. Stuff like making sure new replications don't have any values, existing replications don't have any initializations, and so on. Arguably this is a problem with my data types that they even allow it, but it hasn't really mattered so far. Nobody's building these up by hand.

[tfausak]

Here's a "real world" test I'm trying.

{-# LANGUAGE ScopedTypeVariables #-}

import qualified Data.Binary
import qualified Octane
import qualified Test.Hspec

main :: IO ()
main = do
  let input = "./tmp/input.replay"
  (replay :: Octane.Replay) <- Data.Binary.decodeFile input
  let output = "./tmp/output.replay"
  Data.Binary.encodeFile output replay
  (newReplay :: Octane.Replay) <- Data.Binary.decodeFile output
  Test.Hspec.hspec (Test.Hspec.it "" (Test.Hspec.shouldBe newReplay replay))

It fails, of course.

uncaught exception:
  ErrorCall (Data.Binary.Get.runGet at position 218:
    parsing previous frame probably failed. time: 5.77779e-34, delta: 8.077936e-28 ...)

Here's the file I tried with: input-replay.zip

[tfausak]

Once I can do that, I need to make sure the result from reading the replay I just wrote matches the original replay. If it doesn't, that means some values changed when they shouldn't have.

It's worth noting that this isn't true. In particular, optimizing the frames causes the output replay to be different than the input replay. If I ran it through again, it should be at a steady state. Other than that, the only real way to test is to load it in Rocket League and see if it looks the same. Heck, if it even loads that will be good.

[naphthalene]

You are a hero for figuring a lot of this stuff out. I'm going to read your code and see how I can help.

[tfausak]

Thanks! I can't take all the credit. Both @jjbott and @danielsamuels did a lot of the work.

Let me know if I can help explain any parts of the code.

[tfausak]

Unfortunately not actually fixed yet. I can round trip replays through my library, but Rocket League crashes when it tries to open replays I created.

[tfausak]

I recently learned that Rocket League keeps logs. How I've made it this far without knowing that is incredible. They end up in %UserProfile%\Documents\My Games\Rocket League\TAGame\Logs on Windows. Fortunately it logs even when it crashes, so I may be able to figure out how to fix replay generation. Here's what the logs say when I try to load one of my replays:

[0147.38] Replay: Playing replay, EngineVersion=868 LicenseeVersion=12 ReplayVersion=4 GameVersion=14
[0147.89] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class ProjectX.GRI_X
[0147.92] Log: UGFxDataStore_X::GetValue - accessing table 'TeamInfo' out of bounds [0/0]
[0147.92] Log: UGFxDataStore_X::GetValue - accessing table 'TeamInfo' out of bounds [1/0]
[0147.93] ScriptWarning: ScriptWarning, Accessed None 'MatchType'
    GameEvent_Soccar_TA Stadium_p.TheWorld:PersistentLevel.GameEvent_Soccar_TA_0
    Function TAGame.GameEvent_TA:GetMatchTypeName:0042
Script call stack:
    Function TAGame.GFxHUD_TA:PostBeginPlay
    Function TAGame.GFxHUD_TA:InitGFx
    Function TAGame.GFxHUD_TA:HandleReceivedPRI
    Function TAGame.GFxHUD_TA:SetOwnerPRI
    Function TAGame.GFxHUD_TA:HandleGameEventChanged
    Function TAGame.GFxHUD_Soccar_TA:SetGameEvent
    Function TAGame.GFxHUD_TA:SetGameEvent
    Function TAGame.GameEvent_TA:GetMatchTypeName

[0147.93] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.GameEvent_Team_TA
[0147.93] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.Team_TA
[0147.93] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class ProjectX.PRI_X
[0147.93] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class Engine.ReplicationInfo
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.RBActor_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.Vehicle_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.CarComponent_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.CarComponent_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.CarComponent_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.CarComponent_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class TAGame.CarComponent_TA
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class Engine.Actor
[0147.95] Log: UReplayPackageMap_TA::LoadClassNetCach - unable to find cached class Engine.Actor
[0147.95] Critical: appError called: Ran out of virtual memory. To prevent this condition, you must free up more space on your primary hard disk.
[0147.95] Critical: Windows GetLastError: The system cannot find the path specified. (3)
[0150.95] Log: === Critical error: ===
Ran out of virtual memory. To prevent this condition, you must free up more space on your primary hard disk.

As expected, it has to do with the class net cache. I try to write a minimal cache that only references things that show up in the replay. I think some of the classes are required to be there. Since the class net cache is such a small part of the replay file in terms of size, it probably makes sense to always write a maximal cache. In other words, one that includes every class and every property.

[tfausak]

I am not going to fix this. If you want to write replays, use Rattletrap.