[GB/C] Match Emulation with Halt Bug Test Rom

[TiKevin83]

Gifvex has written a test rom which led to some discoveries about nuances with the halt bug. Current BizHawk's Gambatte does not match console output for it: https://github.com/gifvex/test-roms/blob/master/tests/halt_tests.s This is the expected output for the rom: https://imgur.com/a/4xFvg

Extratricky was able to figure out that these changes should get Bizhawk's Gambatte to sync for the purposes of the yellow TAS: 1) halt bug exists on CGB 2) check interrupts before halting 3) skip one interrupt check post-halt on DMG

The Blue NSC and 151 TASes and the Yellow Glitchless TAS can be treated as test roms for the purpose of making sure that any fixes do not break other nuances with emu accuracy.

[alyosha-tas]

Could you post a compiled version of those tests somewhere please? That would be much easier for me to work with.

[TiKevin83]

https://www.dropbox.com/s/xqc2pie8a3c9357/halt_tests.gb?dl=0

I was able to compile the test rom using the recommended WLA-DX assembler on linux

[TiKevin83]

I built Bizhawk from the latest commit to test where we're at now and got this output from the test rom: https://imgur.com/gallery/ivaHyou

[alyosha-tas]

Cool thanks, I'll look at it this week

[alyosha-tas]

For the expected results, are they for both GB and GBC? Which is which in the above screen shots?

This looks very similar to what is described here: https://www.reddit.com/r/EmuDev/comments/7206vh/sameboy_now_correctly_emulates_pinball_deluxe/

[TiKevin83]

In the first image of the expected result left is DMG and right is CGB.

These tests are for newly discovered nuances of those cpu bugs that you linked. The Yellow TAS desynced at 44 minutes due to the nuances which led to Gif making the more thorough test rom. SameBoy does not even yet match console for Gifvex's test ROM, Lior is aware of this and willing to help.

[alyosha-tas]

Ok some good news, I worked out a plausible implementation that gives the corect results in GBHawk. The basic idea is that a bug is triggered when the I flag is off on the first T cycle of a Halt but subsequently gets set on the first check (2 T cycles later.) When interrupts are enabled, this runs the Halt command twice, when interrupts are disabled, this runs an instance of the original halt bug (reading the next byte twice.)

I don't have even the slightest idea how this would be implemented in Gambatte though.

I haven't looked at the GBC variant yet either.

It's a start though. See 19c509e

EDIT: It's also worth noting that these changes broke a few other test ROMs, but I haven't tried fixing them yet and I believe it should be possible.

[alyosha-tas]

Making some progress here. I managed to re-work the GBHawk ppu so that it passes this test as well as all the other timing sensitive tests currently available (hblank_ly_scx_timing-GS, intr_2_mode0_timing_sprites, etc.)

Something still seems kind of off though. It seems as though maybe the timer should be delayed 4 cycles somewhere (since I had to shift the ppu back in time 4 cycles to make everything work.) Maybe I'll look into that next, but at least for now I can use this as a reference implementation.

[TiKevin83]

Are you accounting for the 4 extra cycles in the BIOS when in GBA as GBC mode in GBHawk? That wouldn't affect the DMG implementation though.

[TiKevin83]

Testing BizHawk after your commit to see if it gets what we expect for the existing Yellow TAS and ran into this problem:

TASMovie.CS line 58: throw new InvalidOperationException("Cannot create a TasMovie against a core that does not implement IStatable");

It ran into this trying to convert the BK2 to a new TASProject to use with the GBHawk core.

[alyosha-tas]

Those cycles are accounted for in that mode (since everything always runs 1 T-state at a time) but I haven't looked at CGB yet.

You said earlier that the pokemon yellow TAS de-syncs at 44 minutes, but I'm not clear on under what circumstances those are. Did you make some changes to gambatte to try to account for this and then it de-synced?

I'm not sure you can easily convert from a gambatte file to a GBHawk file. Probably the easiest thing to do is extract the input file from the BK2 and paste it into a BK2 that was generated natively in GBHawk. I'll try to do it if you have trouble with that, just let me know.

[TiKevin83]

We have a console verification pipeline for GB/GBC games that we thought would work for BizHawk 2.2.2+Gambatte core. It did work for the Blue NSC and Blue 151 TASes but desynced at 44 minutes for Yellow Glitchless, Gifvex did a bunch of black magic using a TAS romhack of yellow and narrowed the desync down to a specific frame and then to the halt bug nuance.

The pipeline involves creating a cycle-timestamped input file using a lua script Gifvex wrote and the new implementation of Total executed cycles (frames aren't accurate enough, we need cycles). We then use GBI's new movie playback feature to deliver the time-stamped inputs to the GBP. GBI keeps track of Total executed cycles using the frequency of the audio controller to know when to send inputs since it can't directly get that info from the GBP.

Yes that would be quite helpful for the Bk2, thanks

[alyosha-tas]

I started working on the GBC Variant. It took only a few modifications, which was encouraging.

I haven't re-worked the GBC ppu core yet though, I'll try to get to it tomorrow. Then pokemon yellow should be testable I hope.

[alyosha-tas]

I reworked the ppu in the gbc to work similarly to the gb version.

Good news is that Pokémon blue still syncs.

Bad news is that yellow version is desyncing very early in the run. It looks like the edge case halt bug is being hit on a STAT interrupt. I would need some more tests to fix STAT interrupt timing in GBC in order to fix this.

[alyosha-tas]

Actually that's not the case, it's desyncing at around frame 8400.

At this point the game is drawing with the window. In GBHawk there is an attempt at emulating penalties for this that is throwing off some code that waits for mode 0 each scanline.

Maybe I can disable this for now and see if it syncs up.

EDIT: Blue version doesn't do this in the same spot, so it's not effected.

EDIT2: Yup, that did the trick, I'll see how far I can get it to sync now.

[alyosha-tas]

I'm gettin stuck at around frame 38000. A read of Stat mode 3 lasts 2 T-states longer then it should and causes RNG to de-sync, I'm currently not sure how to fix it since everything i try so far throws off things earlier in the run

[TiKevin83]

I was able to get everything building and pull over the input log into a new project for GBHawk but TAStudio is crashing when I try to insert lines specifically in GBHawk, I'm not having the same issue with the Gambatte core in my build. If you have a working project file already that would be awesome.

[alyosha-tas]

Actually I am making progress again and am at the 20 minute mark of the movie.

http://tasvideos.org/userfiles/info/47010075012703261

Here is what I have so far. Works on the current dev build. Right now I have a bit of hack in ppu mode 3 timing when window is active to make it sync as in Gambatte. I'll Sort out the fine details later.

[TiKevin83]

Awesome, yep I'm seeing that same desync at around frame 74150 when the TAS does a zig-zag walk while the trainer on the left is leaving the screen. Great work! The playback looks a lot more smooth in GBHawk than Gambatte but it also seems to be a bit slower than Gambatte when the speed is not locked (not surprising if you're emulating down to the T-Cycle).

[alyosha-tas]

It desyncs there only because that is as far as I have adjusted the input for GBHawk's timing. Currently I'm at about frame 105000.

At what point exactly should I be looking for the desync?

[TiKevin83]

At frame 155833 the new halt bug happens, then in the next battle bubblebeam should fully desync and crit the cubone if it's accurate to console

[alyosha-tas]

http://tasvideos.org/userfiles/info/47024381789350657

Ok here is the file with inputs adjusted up to the desync. I do indeed get a halt bug hit exactly where expected, and the desync results in the bubble beam crit as described.

The emulator timings up to this point are very tight, they can't really be off by more then 1 T cycle for the behaviours that the game needs (ppu power up timing, mode 3 timing, timer, VBlank) so this was a very useful exercise up to this point.

What should the next steps be? @TiKevin83 will you try to continue the run from there?

[TiKevin83]

Awesome, I think I will rewrite from there to the end since that can be done in just a few days to verify that we don't run into anything else with accuracy. I won't be able to do that though with insert crashing in TAStudio when using GBHawk. Also I would leave this issue open until Gambatte syncs as well as that was the initial problem.

[alyosha-tas]

Insert works fine for me in TAStudio. Do you have re-pro steps for the crash? Does it still not work even with the new file?

[TiKevin83]

Inserting is working after importing your latest bk2. I'll let you know when I get a rewrite tested on console with Extrems.

[TiKevin83]

The LUA script we have working with a branch of Gambatte that implements totalexecutedcycles() is not working for me with GBHawk at the moment, would you be able to take a look at it and see if you can get it to output anything? I can't confirm yet that it's still working for Gambatte either because the master branch I'm on doesn't have the patches for it to implement totalexecutedcycles.

https://pastebin.com/MNqw6QpR

[alyosha-tas]

Do you have a timestamps.txt file I can use to test with? (That gives the expected output)

[TiKevin83]

The script was designed to create timestamps.txt as output as you play back a TAS, here's an example of what the output timestamps.txt should look like. Thanks for taking a look

https://pastebin.com/Y3iQzeHc

[alyosha-tas]

https://pastebin.com/CxW3jyw9

Ok try this.

Basically only 2 things changed. First, the 'input labels' string was adjusted to match GBHawk's input scheme (incidentally this is probably why you were getting crashes before since this needed to be adjusted in the bk2 as well.)

Second, GBHawk always ends on VBlank, so there is no need to check the PC reg (it will give the wrong answer anyway, since the PC reg doesn't get updated on the first T-state after a frame boundary to 0040.) When the screen is turned off, it just treats that as one extra long frame.

I did a brief test and it seemed to work, except the format was a bit off and it didn't create new lines. Let me know how it works out.

[TiKevin83]

That's awesome, there's some different behavior with it wanting timestamps.txt not to exist before starting and then it doesn't generate the output until you close the program, but it appears to be working (though I think it's outputting dividers instead of cycles or vice versa, but we have to do a transform on the count anyway to pass it to GBI so that part isn't an issue). We definitely need to figure out what's up with the spacing and newlines though, I can take a look at that a bit myself.

[TiKevin83]

I have very good news and I have bad news: Gifvex has published fixes to Gambatte for several bugs including the halt bug that desynced the yellow TAS: https://github.com/Dabomstew/gambatte-speedrun/commits/master

Bad news is I have no idea what I'm doing with LUA scripts and am still having trouble getting the newlines and spacing to work :)

[alyosha-tas]

file:write("\n")

It looks like you just need to add this line after the write, then the file will be output with line breaks.

Oh so gifvex already fixed it in Gambatte, glad he knows how to read that code because I wasn't going to be able to do it.

[TiKevin83]

The LUA output worked! That's weird that it needed a second file:write, but I think I see why now.

I ported gif's accuracy improvements to the TASVideos variable names but I have zero understanding of how to compile it or really anything in C, I've never written C or C++ before, so I have no way of testing it. Trying to build it in VS throws a bunch of errors even after installing the necessary stuff for C in VS.

https://github.com/TiKevin83/BizHawk/commit/82e119a9cee61d12313a3fd5b09fe71d0ff3567a

[alyosha-tas]

if (gbIsCgb || (!gbIsCgb && cycleCounter <= halttime + 4)))

^looks like you have one too many closing parentheses right here.

I ported over your changes and aside from that everything worked and the test ROM does indeed pass now.

Cool stuff!

I'll go ahead and commit that stuff, and then I think we can call this one closed!

[alyosha-tas]

I recommited everything again after including GifVex's improvements to callbacks and totalexecutedcycles, so @TiKevin83 you should be able to use Gambatte from BizHawk master now for your lua script.

[TiKevin83]

That is really, really cool. Thanks!