Run sim by a single log

[LostTemple1990]

Someone may send me a single log when he got a failure after finished a mission It's a hard work to find out what's wrong with the log checking all the actions in log line by line.

The VP will run a sim after finishing a mission,and then generate a log for the mission So we just need to check what VP do for running a sim and generating a log,and then we can easily get this feature

function EV:GARRISON_MISSION_COMPLETE_RESPONSE in Collector.lua shows what VP does after completed a mission a log was generated by local st = serialize(cr) which cr may short for combat report? the sim was ran by two steps below

• Generate checkpoints by log local ok, checkpoints = generateCheckpoints(cr)
• Check differences between sim and real log local novel, nok, om, mismatchTable = isNovelLog(cr, checkpoints)

According to the steps above,we can easily get this feature by

• Create a UI for inputing a single log
• Deserialize the single log to a table

  local cr = deserialize(combatLog)

• Run a sim and match the checkpoints

      local ok, checkpoints = generateCheckpoints(cr)
  if not ok then
      EV("RUN_SIM_BY_LOG_COMPLETE","Generate Checkpoints fail")
      return
  end
  local novel, nok, om, mismatchTable = isNovelLog(cr, checkpoints)
  if not novel then
      EV("RUN_SIM_BY_LOG_COMPLETE","Prediction Correct!")
      return
  end
  EV("RUN_SIM_BY_LOG_COMPLETE",mismatchTable)

• Shows the result on UI you created
• Check what's wrong by yourself according to the result~

[zealvurte]

So here's how I'm handling this.

This first bit I think you already have: in checkpointMatch I added returns that contain the reason for the mismatch, to help know where to look for where the sim was wrong. This is also passed on and returned by checkSim and isNovelLog. This was purely to help when submitting reports, as they could now contain direction on where to look at a glance when using the visualiser, without having to put it in the game and run another comparison yourself.

When generating reports and checking if they're novel, it uses an oracle and a fork limit of 0, meaning forks are not only not (typically) generated if there is any random targeting, but and are never simmed regardless. This means it will only ever generate checkpoints for 1 possible outcome that it believes matches the oracle, so the comparison is only done between that and the log. Unfortunately it's possible for the sim to get the wrong/no target from the oracle (due to VP merging effects and not using the effectIndex as a criteria for matching where effects can have different targets), and instead will pick a random one if none was found (which could randomly produce a correct or incorrect sim result). It's also possible for the oracle to mask incorrect behaviour in the sim, so this isn't quite suitable for doing an analysis for verification.

I correct for this in 2 ways:

1. If an oracle is used, match the target with the effectIndex as well. To support this, I kept all effects in the data separate, and made sure any non-functional ones existed in the data still.
2. If an oracle is used, and if no match is found, then the sim should not pick a new random target, thus eliminating unnecessary forking when the sim has already failed, and it gives a consistent and accurate result.
3. When doing an analysis, I run it for both with and without an oracle to verify the sim.

To do the analysis in game, I added some new global functions (that for now I just call from a macro). Basically they take a index (or missionID) for a report stored in VP's report table, and run the analysis. They support specifying different levels of debug output, a flag for not using an oracle, and a trace filter function (more on that later).

It creates a new sim (with or without an oracle) from the report, with a fork limit set to 0 only if using an oracle. Depending on debug level specified, it also adds a function to output every event from the sim (be aware that forks fully run one after the other, not per event), by assigning it to sim.trace.

It then runs the sim (with a function to stop the sim early if the debug level allows for it), and if there was no oracle specified, it generates a single set of checkpoints that are a range for of all fork checkpoints (otherwise it just uses the main checkpoints already generated). It took a lot of work to get all the string parsing for the forked checkpoints right, and there may have been a better solution for that, but it's what I ended up doing.

With no debugging, it will just output the result with a reason, much like the one I do when first generating the reports. With low debug levels, it checks for matches with the number of checkpoints, and the unit health (and presence) within each checkpoint, and outputs the first mismatch for each, or the entire list of checkpoints.

At higher debug levels, the trace function is used instead, outputting events as they happen and checkpoints at the end of each round, which it does for every fork. Within that trace function, is where a provided trace filter function will be called on every event, and if that filter function returns false, it skips output for that event (including if that event would also output the checkpoints), so this can quickly help look at only the spells, rounds, units, etc. you care about. The trace also checks each checkpoint exists as it goes if using an oracle, which allows it to end the sim early as soon as it finds something wrong.

Within the trace I'm using the eventTypes as per the log, and cached info about units and their spells from the sim (this requires storing a new table of spell info when the sim is created), to enable writing output for each event. This meant adding in some additional trace calls within the sim in key places (and I've yet to cover all events, but all damage, heals, and most applications/removals are done), and changing up the parameters that are used for this.

Hope that helps you!