Automated frame by frame comparison of Player with RPG_RT

[fmatthew5876]

The save data between RPG_RT and Player is supposed to be the same. We could construct an automated system to run both Player and RPG_RT in parallel. Each instance would send an LSD serialized full copy of its save data to a verifier every frame.

The verifier would compare the 2 save datas to ensure they are an exact match.

If we had this capability, we could quickly resolve the subtle timing bugs and get a nearly perfect emultion of RPG_RT behavior. The kind of bugs such a system could find may be nearly impossible to find by humans looking carefully.

I'm not sure if this is even possible. It sounds like a big project.

This setup would have 3 components all running at the same time. Player, RPG_RT, and a Verifier.

Some of the steps would include:

• Player for each frame
  • Hook the end of each frame
  • Serialize the save data using liblcf
  • Pause.
• RPG_RT for each frame
  • Inject some code at the end of the frame to hook.
  • Make RPG_RT save its save data
    • Writing to memory/socket/ipc is better than actually writing to disk.
• Verifier for each frame
  • Wait for save data from RPG_RT and Player
  • Bitwise compare the serialized streams.
    • Optionally, we can disable comparison for chunks we know are different and we don't want to fix.
  • If the save data matches:
    • Signal both Player and RPG_RT that they may continue to the next frame
  • Else
    • Pause everything and dump debug info. Optionally allow the user to advance to the next frame.

The Verifier could be built into Player. Features such as pausing on specific frames and advancing manually frame by frame are good general features to have.

The next step would be to script a playback of a game in both engines, and then use this framework to ensure they do everything identically.

[fmatthew5876]

Another possible and much simpler implementation:

Write a DynRPG plugin (or equivalent) for RPG_RT. Every frame, this plugin will dump the save data and which keyboard keys (if any) were reacted to that frame.

Play a game in DynRPG with this plugin for a while and generate some data.

Load the playback capture data into Player. Each frame, replay keyboard events as in the capture. At the end of the frame, compare the save data with the capture's save data. Break and dump information if any differences.

[CherryDT]

Yes, however you need to take into account that the relevant part here would be "after main loop completed" (because that's when the save menu would normally be able to step in), and that's also when the frame counter is increased, which is not every actual redrawing frame because during things like transitions, screen updating and "wait for 1/60 second" is triggered within still the same iteration of the main loop.

EDIT: Also your idea will break because of the RNG as soon as you have random movement, encounters and such things.

[CherryDT]

Hm, there would be an advanced way to ensure sync: Use a simple reproducible RNG algorithm which you also implement in the RPG_RT patch for this so that you know that the two are in sync.

EDIT: Damn, no, it wouldn't work, because your code is probably different enough from RM's that you don't call the random function in the same order when evaluating the different values thoughout processing in each frame. Unless you go so far that you ensure that the random function returns the same values all the same as long as the frame is the same, but that may have undesired effects...

[20kdc]

Maybe just have all randomness return 0? Then it's order independent.

[CherryDT]

I'd rather have it return a different value per frame (but a consistent one), that would make it reasonable random for most cases to still ensure that some games could be tested properly...

[fmatthew5876]

I was able to get a poor mans version of this working on linux. I'm running RPG_RT.exe using wine.

An event with this code:

Shake Screen: 9, 5, @1.0s
Loop
  Open Save Menu

And this bash script

FILE=SomeGame/Save01.lsd
 while inotifywait -e close_write "${FILE}"; do
         ${SOME_COMMAND} "${FILE}"
         I=$((I+1))
 done

With this you can run a command on the save game file each time it changes.

Then just play the game and keep saving. With this I was easily able to collect timing data on the shake screen algorithm. I did it by printing with LCF_DEBUG_TRACE and greping for shake chunks.

With that, I was able to generate output like this:

SSAVE SEARCH -e shake                                                                                                                                                 
===================
SSAVE UPDATE 0
===================
0x0b (size: 1, pos: 0x41): frame_count
  56
--
0x1f (size: 1, pos: 0xed): shake_strength
  9
0x20 (size: 1, pos: 0xf0): shake_speed
  5
0x23 (size: 1, pos: 0xf3): shake_time_left
  60
SSAVE UPDATE 1
===================
0x0b (size: 1, pos: 0x41): frame_count
  80
--
0x1f (size: 1, pos: 0xed): shake_strength
  9
0x20 (size: 1, pos: 0xf0): shake_speed
  5
0x21 (size: 5, pos: 0xf3): shake_position
  -18
0x23 (size: 1, pos: 0xfa): shake_time_left
  53
SSAVE UPDATE 2
===================
0x0b (size: 1, pos: 0x41): frame_count
  98
--
0x1f (size: 1, pos: 0xed): shake_strength
  9
0x20 (size: 1, pos: 0xf0): shake_speed
  5
0x21 (size: 1, pos: 0xf3): shake_position
  3
0x23 (size: 1, pos: 0xf6): shake_time_left
  46

EDIT: You don't need wait commands at all. Every call to OpenSaveMenu advances by 1 frame. So with this trick you can get frame by frame snapshots of LSD data. It works for both Player and RPG_RT.

[Ghabry]

Linking to #2297