[Feature Request] Add support for frame buffer extension API

[gonetz]

Need to implement the following extension to Zilmar's specs:

/** Function: FrameBufferRead Purpose: This function is called to notify the dll that the frame buffer memory is beening read at the given address. DLL should copy content from its render buffer to the frame buffer in N64 RDRAM DLL is responsible to maintain its own frame buffer memory addr list DLL should copy 4KB block content back to RDRAM frame buffer. Emulator should not call this function again if other memory is read within the same 4KB range input: addr rdram address val val size 1 = wxUint8, 2 = wxUint16, 4 = wxUint32 output: none ***/ EXPORT void CALL FBRead(wxUint32 addr)

/** Function: FrameBufferWriteList Purpose: This function is called to notify the dll that the frame buffer has been modified by CPU at the given address. input: FrameBufferModifyEntry plist size = size of the plist, max = 1024 output: none **/ EXPORT void CALL FBWList(FrameBufferModifyEntry *plist, wxUint32 size)

/** Function: FrameBufferWrite Purpose: This function is called to notify the dll that the frame buffer has been modified by CPU at the given address. input: addr rdram address val val size 1 = wxUint8, 2 = wxUint16, 4 = wxUint32 output: none ***/ EXPORT void CALL FBWrite(wxUint32 addr, wxUint32 size)

/**** Function: FBGetFrameBufferInfo Purpose: This function is called by the emulator core to retrieve frame buffer information from the video plugin in order to be able to notify the video plugin about CPU frame buffer read/write operations

size: = 1 byte = 2 word (16 bit) <-- this is N64 default depth buffer format = 4 dword (32 bit)

when frame buffer information is not available yet, set all values in the FrameBufferInfo structure to 0

input: FrameBufferInfo pinfo[6] pinfo is pointed to a FrameBufferInfo structure which to be filled in by this function output: Values are return in the FrameBufferInfo structure Plugin can return up to 6 frame buffer info ****/ EXPORT void CALL FBGetFrameBufferInfo(void *p)

[AmbientMalice]

Gonetz, shouldn't this be posted on the PJ64 github page?

[gonetz]

Nope. These are gfx plugin API functions. Would be great to see support of them from PJ64 side.

[AmbientMalice]

I see your point. Does mupen64plus have all the API features you need, or will work have to be done there, too?

[gonetz]

I don't know about mupen64plus. Please test and let me know. I tested with Mupen64 only.

[gonetz]

WIP code pushed to FBinfo feature branch:

https://github.com/gonetz/GLideN64/tree/FBInfo

phew!

[purplemarshmallow]

I did some tests (Mupen 0.5.1) Zelda OOT and MM: sometimes coronas disappear behind walls but most of the time it does not work. Banjo Kazooie: crash at intro during jigsaw effect. The image looks weird Pokemon Snap: crash at title screen Bomberman 64: images during intro show up but they look strange

[Papermanzero]

You can use the newest builds from mupen64plus from here: https://bitbucket.org/ecsv/mupen64plus-mxe-daily/overview

[gonetz]

I know about issues, it is WIP. Tell me what I don't know, namely why it does not work? I had not enough time for debug.

[purplemarshmallow]

When I tested with 1964 it worked as good but really slow. I guess 1964 can call the same address multiple times. With a check for duplicate calls it should work with 1964 as well

[AmbientMalice]

So when using mupen with this branch, should FB to RDRAM and such be enabled or disabled? I assume disabled.

edit:

Unfortunately, I get a bunch of build errors, and can't test this. Using VS2013.

[gonetz]

FB to RDRAM and such should be disabled. I use VS2013 also.

[AmbientMalice]

Build errors fixed by https://github.com/gonetz/GLideN64/commit/004b88276ccb9a874e5186538df89865f09fe413 but nothing seems to work. Castlevania menu FB effect don't work, for example.

[gonetz]

This is WIP, alpha version of the feature. I published it for devs who want to dig the code and find what is wrong in it and how it can be fixed. I know about issues, no need to report about them atm.

[weinerschnitzel]

@gonetz I see that you are making some more commits. Could you vaguely point out internal issues you already know about that need attending? That would give the rest of us clues as to what to look for

[purplemarshmallow]

In Bomberman64 the bottom of the image is missing. This has been investigated and it happens because Mupen64 misses some fb reads. This implementation is very accurate and if the emulator provides wrong information the result will be wrong

I tested more games (Mupen 0.5.1). I need to find out is the problem on Mupen's side or in GLideN64

• Glover2: strange black stripes inside the image. bottom is missing like bomberman.
• Castlevania/Castlevania2: fully working
• Mario Kart: black square inside the monitor
• Banjo Kazooie jigsaw effect: The result is random. Sometimes the bottom of the image is missing like Bomberman. Sometimes the whole image is buggy and broken. In both cases the emulator crashes afterwards.
• Ocarina of Time: Sometimes coronas work correctly sometimes coronas are visible through walls.
• Jet force gemini: The top of the monitors is buggy.
• Majora's Mask: camera shows nothing.
• Conker/Mickey's speedway: pause menu background is completely missing
• Mario Tennis: fb read is often called when the pause menu is opened and causes a slowdown.
• Shadowman: pause menu takes a long time to open. Sometimes the whole image is missing sometimes the bottom is missing like Bomberman.
• 1080: freeze at intro, I can't get ingame.
• Pokemon Snap: freeze at title screen
• Mini Racers: fully working
• Clay Fighters sc: bottom of the image is missing like Bomberman
• bakushou jinsei 64: not working but strange dots appear inside the image
• Heiwa Pachinko World 64: bottom of the image is missing like Bomberman

There is a bug introduced in GLideN64. Buffers can be copied to the wrong destination. It's very noticeable if you enable copy auxiliary and test Pokemon Stadium games.

[LegendOfDragoon]

@purplemarshmallow what about 1964? Is it missing on that emulator as well?

[purplemarshmallow]

I did some testing with 1964. With 1964 there's a big slowdown during CPU reads. Does not happen with Mupen64. Mario Kart: Correct image but really slow Bomberman64: Correct image but really slow Jet force gemini: correct monitors but slow Banjo Kazooie: crash. Crashes with both 1964 and Mupen so there's likely a problem on GLideN64's side.

[gonetz]

Looks like 1964 has better support for FB extension. If it works, it works correct. Mupen64 misses many fb reads, thus resulted images have black areas.

[gonetz]

Banjo Kazooie: crash. Crashes with both 1964 and Mupen so there's likely a problem on GLideN64's side.

Alas, it's not GLideN64 bug. It is problem of FBInfo conception.

The problem happens when the game needs to read the current color buffer for the puzzle effect. At this moment only color buffer data is necessary. Depth buffer data is not used and CPU uses that memory area for its own needs. CPU writes something to that area and emulator sends bunch of FBWrite notifications to depth buffer area. Then CPU starts to read that data, and emulator sends FBRead notifications from depth buffer. The plugin reads the depth buffer from video card and writes it to RDRAM. Thus, the data written to depth buffer by CPU becomes broken and the game hangs. When I disable depth buffer writes, the game goes forward and the puzzle effect works (with a black line, but works).

I think that particular situation can be fixed: if game does writes and reads to the same buffer then do nothing.

[LegendOfDragoon]

Mario Kart: black square inside the monitor

It worked fine with Glide64 Final when I tested that a while back on Mupen64 0.5.

Mario Kart: Correct image but really slow

Slow as in < 60? If that's the case, then there's something wrong.

[gonetz]

It worked fine with Glide64 Final when I tested that a while back on Mupen64 0.5.

Forget about Glide64. I made very simple and rough support for FBInfo in it. GLideN64 works as described in API. Problems with image are most likely result of error on emulator side. At least Bomberman and Mario Kart are such cases.

< Slow as in < 60?

yes, sometimes it is really slow.

[purplemarshmallow]

Alas, it's not GLideN64 bug. It is problem of FBInfo conception.

I don't think so. I can think of a good solution. The gfx plugin controls it. The gfx plugin must check if it's safe to make a copy. And send the depth buffer area only to the emulator if it's safe. The plugin already knows if it's safe to make a copy. The current depth buffer emulation method does not crash. Should be enough to tell the emulator that depth buffer area is not used.

[gonetz]

You are wrong here. Re-read FBInfo specification and my explanation of the problem.

Our standard method of frame/depth buffer emulation copies the buffer during ProcessDList. At this moment RDP controls the workflow and it can safely write data to the buffers. That's why "the current depth buffer emulation method does not crash".

FBInfo works when CPU takes back the control. The assumption "The gfx plugin controls it" is wrong. Plugin controls nothing at this moment, it does what emulators commands to do. Plugin can't detect that CPU is going to re-use depth buffer memory for its own needs. CPU takes that decision. However, emulator knows that this memory area is used as a buffer because plugin informed about it via FBGetFrameBufferInfo callback. According to the FBInfo API specification emulator informs the plugin about writes to buffer area. The plugin now knows that CPU modified buffer content and thus the changes need to be shown on screen. Then emulator reads values from the buffer area. Again, it knows that it is a buffer and sends FBRead notifications. According to the specification: "This function is called to notify the dll that the frame buffer memory is beeing read at the given address. DLL should copy content from its render buffer to the frame buffer in N64 RDRAM". The buffer for which FB functions called is valid for the plugin, because it was set in latest DList. Plugin should read the data to the buffer in RDRAM. That data will overwrite CPU data and program will crash. Thus strict following to the API specification causes crash.

[purplemarshmallow]

Yes you are correct here. Strict following to the API specification causes crash.

1080 still crashes. But depth buffer writes are not the problem here. Color buffer writes cause the crash. And there's an additional problem. If I disable color buffer writes I can get ingame. But it takes a very long time. The fb write code slows things down. If i disable the fb write code I'm ingame immediately (Mupen).

[purplemarshmallow]

I noticed a regression. The top row of pixels is not copied. Maybe the bottom row is also incorrect. This can cause crashes if the plugin writes into game data. And copy from RDRAM does not seem to work anymore. If I try it the plugin just crashes.

[gonetz]

I found why 1964 works so slow. It does not follow FBRead specification: "Emulator should not call this function again if other memory is read within the same 4KB range". FBRead calls come from near addresses in no particular order: FBRead addr=8032d1b2 FBRead addr=8032d1b0 FBRead addr=8032d1b6 FBRead addr=8032d1b4 FBRead addr=8032d1ba

[gonetz]

I optimized FBRead calls. Number of buffer reads reduced from hundred to 6 in Mario Kart with 1964. However 1964 still works much slower. I think it is because 1964 calls plugin's FBRead too often. 1964 has slowdown near Mario Kart monitor even if FBRead does nothing.

[weinerschnitzel]

@LegendOfDragoon, we should look into this. Pinging you because I can't find time on IRC

[weinerschnitzel]

@gonetz, can you upload your optimized build? or email me at gmail.com... I would like to try a few things with 1964 to prevent calls when the FBRead addr is within 4KB of the last call and see if results help.

[gonetz]

https://drive.google.com/file/d/0B0YqMPjGo3B2Y0pYZnFzbk4tNnc/view?usp=sharing

pass: fbinfo

[weinerschnitzel]

Ok. I modified VIDEO_FrameBufferRead in 1964 to skip FBReads that are within 4KB of the last read address, and this fixes the slowdowns in MK64. I saw that FBReads per second would exceed 60000 with RiceVideo, although Rice wouldn't slowdown. Skipping FBReads that are "too close" brings the count below 200 with Rice or GLideN64.

I don't notice any visual degradation or things breaking, but I may not be keen enough to see a difference. I'll try more test cases.

[AmbientMalice]

@weinerschnitzel Would you mind uploading your 1964 for testing when it's acceptably working?

[weinerschnitzel]

https://www.dropbox.com/s/dp1pm782ez9do3i/1964%20test.7z?dl=0

Here is the little test build I've been working on. 1964.exe is built with the FBRead modification, and the VI/s counter is swapped with an FBRead/s counter. 1964_11.exe is vanilla 1964 1.1. I'll make a clean PR to @LegendOfDragoon's 1964 repository that we can scrutinize.

FBInfo seems to work really well. My only concern is if skipping FBReads makes the MK64 video monitor choppier. I can't tell.

[gonetz]

You may test your fix on full-screen fb effects. Banjo-Kazooie puzzle effect is a good example. If image has no missing (black) parts - the fix works correct.

[AmbientMalice]

So I've been doing some testing. Why is 1964's audio eating up 70% or more? Mario Kart is glitchy with black lines across the displays. Banjo Kazooie shows white puzzle pieces before freezing.

[weinerschnitzel]

1964audio can be expensive at times. I'm also not seeing any black squares when skipping FBReads. MK64 video monitors work fine, Banjo Kazooie puzzle effect works fine, and Bomberman64 into works fine. I'm on Intel, maybe your issue is vendor related @AmbientMalice? Thanks for the testing

[AmbientMalice]

I'm on Nvidia. GTX 750Ti. Windows 10.

With optimisation:

Without:

[weinerschnitzel]

Do you get the same issue with RiceVideo/1964video when you have Framebuffer with Emulator enabled?

[AmbientMalice]

Seems to work fine with 1964video. So it's obviously a GLideN64-side bug.

[weinerschnitzel]

...and just to be sure, can you confirm the black bars are there for you in GLideN64 + vanilla 1964 1.1?

[AmbientMalice]

No, they only appear with optimised build+GLideN64.

[weinerschnitzel]

I can't reproduce the black bars on my end. I hope more people can chime in on this so I can better tell if my fix is wrong and Intel is bugged but allows everything to show up properly or vice-versa. I don't think FB effects working in 1964video and not GLideN64 is enough to call it GLideN64's fault yet.

I did notice Banjo Kazooie crashes again when FB Read/Writes are enabled in ROM Properties, but works when it is not ticked. (Vanilla 1.1 and optimized) I did tweak my fix slightly and Banjo Kazooie works again. Having said that: I can't say if something else broke down the line for GLideN64, or my tweak made FBInfo behave more properly for 1964.

@AmbientMalice could you test Bomberman64 and Banjo Kazooie intros as well?

[LegendOfDragoon]

I tried 1964 video and Glide64 Final in Mario Kart. Didn't notice a difference in performance or visuals for the video monitor.

[AmbientMalice]

@LegendOfDragoon

Just a few points.

Did you test both 1964 exes provided?

Did you enable framebuffer notifications in glide64? It's disabled by default, in most versions.

Vanilla 1964 should slow to a crawl for most people with Mario Kart's TVs. The optimised build should suffer no slowdown.

Currently, only GLideN64 has the incorrect graphics bug. Since you can't run it, AFAIK, that presents a problem. We need someone who can.

[AmbientMalice]

Hey, @Lithium64 you can run GLideN64, right? Could you test these builds?

[theboy181]

Seems OK over here.. The optimized version is faster.. However, I think that Jabo's Jumbotron is way faster and better.. Why not implement the hack in to GLideN64? I prefer the Jumbotron display at 30fps.. and the performance issue is non-existent.

[AmbientMalice]

The sort of hack Jabo's uses is unacceptably dirty, in my view. GLideN64 moving away from game specific hacks wherever possible can only be a good thing. The Mario 64 display judders on a real N64, and in my view GLideN64 should reflect that.

[theboy181]

I respect your opinion. However what is the point of HW rendering with your views? When we are using HW the goal is to reach HiRes and better performance.

MK64 is a popular game.

try to think of any hack that improves the users experience as an enhancement not a hack.

[LegendOfDragoon]

Did you test both 1964 exes provided? Did you enable framebuffer notifications in glide64? It's disabled by default, in most versions.

Yes, I tested both. I set up the enable framebuffer as well.

I decided to test the build, Gonetz provided as well as public release 1.1. Even though the graphics are totally broken on my end, it was still worthwhile to test fb emulation performance. I guess Glide64 Final and Rice have code to prevent doing the same buffer repeatedly, while this version of GLideN64 does not. I'm impressed though, GlideN64 does have faster fb emulation than any other hardware rendering plugin. I guess I should look at the code, when I ever decide to work on frame buffer emulation..

[gonetz]

GLideN64 is the only plugin, which follows FBRead specification. The specification states that plugin must fill 4kb of RDRAM starting from the given address. Other plugins read the whole buffer and ignore the rest of FBRead commands for that buffer. GLideN64 fills exactly 4kb. This behavior is intentional. I needed to test, how well FBRead is implemented on emulator's side. That is, can we trust the emulator. The exact following to the specification has two consequences:

• plugin reads only those data, which was actually asked by the emulator. Thus, if emulator does not issue FBRead for some necessary data, that data will not be read and we will have black or garbage pixels. We can see it with Mupen64.
• plugin reads video cards buffer many times. 4kb is 3% of standard 320x240 buffer. Read of one 4kb chunk is faster than read of whole buffer, but not 30 times faster due to various overheads. 8x at maximum. Thus, reads of 2-3 chunks will be faster, than read of whole buffer at once, but read of whole buffer by 4kb chunks is much slower. Again, it is intentional behavior to reveal issues with FBRead support by emulators.

Now I know the issues:

• Mupen64 does not call FBRead for all necessary data. We can't trust it. Solution: read the whole buffer.
• 1964 does not follow FBRead specification and calls FBRead to near addresses. Solutions: a) fix 1964 b) add filter for near addresses in the plugin.