Mario Artist Polygon Studio - Cannot model / paint (Framebuffer, SetOtherMode Coverage Issue)

[LuigiBlood]

It is simply impossible to model in Polygon Studio. You cannot select models individually (all or nothing), doesn't detect vertexes and stuff.

From the main menu, select the Crown, then the Rocket, then the commands in the middle.

EDIT: As there are a lot of findings, I'll do a brief catch up in the first post of all the problems found:

Model Detection not working

• To access this: From the main menu, select the Crown, then the Rocket, then the commands in the middle. Select one from the five icons on the top right of the screen for a sample model.

• Game uses an auxiliary framebuffer at 80770800 (320x240) which is not written back to RDRAM because of the definition of an auxiliary framebuffer being solely based on the width.

• When rendering the model to that buffer, game sets SetOtherMode_L to CVG_DST_FULL and ALPHA_CVG_SEL, which forces alpha value to be set to 1.0 but is not emulated.

• Fix:

  • Change the definition of an auxiliary framebuffer (currently is any framebuffer that isn't full width), or do a game specific hack to treat them as auxiliary.
  • ~~Treat the special case of use Alpha = Coverage = 1. I did something like this in the shader: ssShader << " if (uAlphaCvgSel != 0 && uCvgDest == 2) fragColor.a = 1.0;" << std::endl; Coverage emulation is not needed here.~~ This is fixed.

Painting models

GLideN64 problem screenshot (after fixing the aux framebuffer problems):

• To access this: You need a model ready from the previous part, then to the rocket model menu and select the mosaic colored screen on the bottom left. Then just draw on the model to reproduce the issue.

• Game uses an auxiliary framebuffer at 80796000 for Painting models, GLideN64 replaces most of the texture with black, suggesting an alpha problem. It also suffers from the same problem as the model detection as in not written back to RDRAM.

• However I can't really understand the problem much on this part, other than the fact that pens don't work properly, but erasers does not work either but uses different combiner settings where the end result is different. But stamps are working perfectly fine.

• When I'm about to draw with the pen or eraser, the emulation freezes for a few seconds while it's loading the texture data from 80500000-805FFFFF. Not sure why.

• Fix:

  • Do the auxiliary framebuffer fix, but also unknown how to fix the paint bug.

Photo Mode

Angrylion-rdp-plus expected result:

• To access this: From the main menu, select the Crown, then the camera above the UFO. Press A to take a picture and check the result.

• Game uses an auxiliary framebuffer at 8074B000 for photos without HUD. It is not written to RDRAM for the same reasons as Model Detection not working and as a result does a black frame photo on GLideN64. Fixing this solves this problem.

• Fix:

  • Do the auxiliary framebuffer fix explained earlier.

Wireframe Effect

Angrylion-rdp-plus expected result:

Current GLideN64 output (as of master at 2021-01-14):

• To access this: From the main menu, select the Crown, then the UFO. Select the second to last menu from the left (glowing stars), then on the bottom right, go to Page 2, and the select the last effect (should be a wireframe box).

• This effect does not work, it only does a solid color screen behind the HUD.

• Fix:

  • Unknown (for me)

[olivieryuyu]

does it work with other plugins?

[LuigiBlood]

Works with angrylion.

[LuigiBlood]

Checkup: Tried on d8ac5a7 WIP build, still does not work.

[LuigiBlood]

Tried this on latest WIP build, still not able to model at all in either HLE or LLE modes.

[olivieryuyu]

can u give a savestate for PJ64 exactly at the point where it bugs?

[LuigiBlood]

As explained in the first post, no save state required you can just do this: From the main menu, select the Crown on the right, then the Rocket on the left, then select the handle in the middle, you're in the modeler. It does not take long to do honestly.

[olivieryuyu]

Got it

Despite a normal ucode name ( F3DEX fifo 2.08), there is a new command 0x08.

@gonetz

We are not done yet :(

[gonetz]

Can you decode it?

[olivieryuyu]

Will start the decoding next week.

[olivieryuyu]

:)

[olivieryuyu]

this command 0x08 is simply from L3DEX command. Got it wrong.

[LuigiBlood]

Did you figure it out then? Is it easy to implement?

[olivieryuyu]

Not at all

It is not a HLE issue because it is the same in LLE.

If you use Angrylion plugin and load a savestate with GlideN64 (HLE/LLE), then it works till u switch options. Super weird...

[LuigiBlood]

Info about how the game handles the modeling aspects from gamemasterplc:

The game seems to write vertex ids to an auxiliary buffer using some special combiner settings and primitive colour @LuigiBlood The game updates the buffer when you stop the camera or place a new object The auxiliary buffer for vertex picking is at 0x80770800 Even values mean an unpickable vertex It’s read directly by the cpu for deciding whether the cursor is over a polygon the check takes place in the function at 0x8043B748 And seems to be reused for all picking The special combiner settings convince it to render polygons entirely filled with the primitive color To an auxiliary buffer I wonder how GlideN64 will render to an auxiliary buffer at low res And it is a framebuffer

[gamemasterplc]

Even values actually mean an unpickable pixel. And the game doesn't update the auxillary framebuffer constantly. The auxillary framebuffer is used by the CPU only but should be written to by the RDP.

[LuigiBlood]

I compared the memory between angrylion and GLideN64, at 80770800, the auxiliary buffer isn't even rendered at all in GLideN64, it only has garbage.

[LuigiBlood]

Here's what we're supposed to have using angrylion (I put a cube and nothing else):

GLideN64:

I think by definition, it is not an auxiliary buffer since the width is actually 320.

bool FrameBuffer::isAuxiliary() const
{
    return m_width != VI.width;
}

EDIT3: To mention about the second picture, I got the same thing just before I go to the model editor in angrylion, meaning that absolutely nothing was rendered at this location in RAM at all in GLideN64.

EDIT4: VI origin change update does not fix this, neither does LLE. Same results.

[LuigiBlood]

Okay so I decided to compile GLideN64 (which thankfully the lazy way exists) and see for myself on the latest: I found it rewrites that framebuffer with nothing. That's better, I guess, but it's still nothing rendered to make it work.

[gamemasterplc]

If nothing is all zeroes, then that is the clear color.

[LuigiBlood]

It is all zeroes this time actually. But unfortunately that's the best it got for this game.

[LuigiBlood]

It seems the Display List is at 80355A20, it keeps adding +8 to the blue color (and sometimes more) via gDPSetPrimColor, while Alpha is always 0, for each triangle. (EDIT: Putting Alpha = 255 does not fix the problem) The only thing that seems to touch RAM is slightly before that Display List and it's gDPFillRectangle, where it zeroes out the RAM.

[LuigiBlood]

Okay so I managed to get it to work somewhat but this is DEFINITELY not a good solution, but here's the gist of it: The framebuffer is absolutely not written, and it is an alpha problem as well. Forcing both fixed the Polygon Studio modeling problem.

Here are the changes I made to give a clue: In gDP.cpp: line 249

void gDPSetPrimColor( u32 m, u32 l, u32 r, u32 g, u32 b, u32 a )
{
    gDP.primColor.m = _FIXED2FLOAT( m, 5 );
    gDP.primColor.l = _FIXED2FLOATCOLOR( l, 8 );
    gDP.primColor.r = _FIXED2FLOATCOLOR( r, 8 );
    gDP.primColor.g = _FIXED2FLOATCOLOR( g, 8 );
    gDP.primColor.b = _FIXED2FLOATCOLOR( b, 8 );
    gDP.primColor.a = _FIXED2FLOATCOLOR( a, 8 );
    //Force Alpha = 1.0 if 0
    if (gDP.primColor.a == 0.0f)
    {
        gDP.primColor.a = 1.0f;
    }
    DebugMsg( DEBUG_NORMAL, "gDPSetPrimColor( %i, %i, %i, %i, %i, %i );\n", m, l, r, g, b, a );
}

In FrameBuffer.cpp: line 689

if (m_pCurrent != nullptr &&
    config.frameBufferEmulation.copyAuxToRDRAM != 0 &&
    (config.generalEmulation.hacks & hack_Snap) == 0) {
    //if (m_pCurrent->isAuxiliary()) {
        FrameBuffer_CopyToRDRAM(m_pCurrent->m_startAddress, true);
        removeBuffer(m_pCurrent->m_startAddress);
    //}
}

I have intentionally broken features to make this work, to see what the problem was. I think it might hinge on the definition of an auxiliary buffer according to GLideN64 and also an alpha rendering accuracy problem. Doing this breaks the thick black edges in Polygon Studio, also.

EDIT: A temporary hack I could have done to fix other issues could be this but it's not really great, is it...

if (m_pCurrent != nullptr &&
    config.frameBufferEmulation.copyAuxToRDRAM != 0 &&
    (config.generalEmulation.hacks & hack_Snap) == 0) {
    if (m_pCurrent->isAuxiliary() || (m_pCurrent->m_startAddress == 0x770800)) {
        FrameBuffer_CopyToRDRAM(m_pCurrent->m_startAddress, true);
        removeBuffer(m_pCurrent->m_startAddress);
    }
}

EDIT2: One other thing though, this is still not fully playable since the Paint feature is still broken as well, even with this as a fix it still has issues; and so does the photo mode, and the model mode is a little bit broken with the background having blinking white dots. I also made a fork for posterity using a game specific hack bit. https://github.com/LuigiBlood/GLideN64/tree/polygonstudio_hack

[LuigiBlood]

Okay so the game uses FIVE full framebuffers at 80700000 & 80725800 (main double buffer), 8074B000 (Photo Mode), 80770800 (Model Triangle Detection), 80796000 (Model Paint Mode during drawing). There's also 80500000 which contains the textures used for Model Paint mode. It seems Paint mode stalls because GLideN64 is copying the texture data (which is BIG, it's 0x200000 in size right before the first framebuffer.) That said it only stalls for a moment and then it kinda works.

EDIT: I keep editing, as I'm finding out things: There are 80500000 and 80600000 for model paint textures, the other is a copy. It's still a copy of 0x100000 which happens to be slow and then it works again.

[LuigiBlood]

Here's the combiners.

//For Model Detection
FCFFFFFF FFFDF6FB

a0  = 1111  = F        (Color 'a', 1) G_CCMUX_0
c0  = 11111 = 1F       (Color 'c', 1) G_CCMUX_0
Aa0 = 111   = 7        (Alpha 'a', 1) G_ACMUX_0
Ac0 = 111   = 7        (Alpha 'c', 1) G_ACMUX_0
a1  = 1111  = F        (Color 'a', 2) G_CCMUX_0
c1  = 11111 = 1F       (Color 'c', 2) G_CCMUX_0

b0  = 1111  = F        (Color 'b', 1) G_CCMUX_0
b1  = 1111  = F        (Color 'b', 2) G_CCMUX_0
Aa1 = 111   = 7        (Alpha 'a', 2) G_ACMUX_0
Ac1 = 111   = 7        (Alpha 'c', 2) G_ACMUX_0
d0  = 011   = 3        (Color 'd', 1) G_CCMUX_PRIMITIVE
Ab0 = 111   = 7        (Alpha 'b', 1) G_ACMUX_0
Ad0 = 011   = 3        (Alpha 'd', 1) G_ACMUX_TEXEL1
d1  = 011   = 3        (Color 'd', 2) G_CCMUX_PRIMITIVE
Ab1 = 111   = 7        (Alpha 'b', 2) G_ACMUX_0
Ad1 = 011   = 3        (Alpha 'd', 2) G_ACMUX_TEXEL1

For whatever reason for this one, the Alpha channel is not 1.0. Replacing G_ACMUX_TEXEL1 with G_ACMUX_1 fixes the model buffer.

//For Paint Model
FC309661 552EFF7F

a0  = 0011  = 3        (Color 'a', 1) G_CCMUX_PRIMITIVE
c0  = 00001 = 01       (Color 'c', 1) G_CCMUX_TEXEL0
Aa0 = 001   = 1        (Alpha 'a', 1) G_ACMUX_TEXEL0
Ac0 = 011   = 3        (Alpha 'c', 1) G_ACMUX_PRIMITIVE
a1  = 0011  = 3        (Color 'a', 2) G_CCMUX_PRIMITIVE
c1  = 00001 = 01       (Color 'c', 2) G_CCMUX_TEXEL0

b0  = 0101  = 5        (Color 'b', 1) G_CCMUX_ENVIRONMENT
b1  = 0101  = 5        (Color 'b', 2) G_CCMUX_ENVIRONMENT
Aa1 = 001   = 1        (Alpha 'a', 2) G_ACMUX_TEXEL0
Ac1 = 011   = 3        (Alpha 'c', 2) G_ACMUX_PRIMITIVE
d0  = 101   = 5        (Color 'd', 1) G_CCMUX_ENVIRONMENT
Ab0 = 111   = 7        (Alpha 'b', 1) G_ACMUX_0
Ad0 = 111   = 7        (Alpha 'd', 1) G_ACMUX_0
d1  = 101   = 5        (Color 'd', 2) G_CCMUX_ENVIRONMENT
Ab1 = 111   = 7        (Alpha 'b', 2) G_ACMUX_0
Ad1 = 111   = 7        (Alpha 'd', 2) G_ACMUX_0

I haven't done anything to this one yet. EDIT: Turns out this was wrong.

This is the Combiner for the Paint Pen:

//For Paint Model Pen
FCFFB3FF FF65FEFF

a0  = 1111  = F        (Color 'a', 1) G_CCMUX_0
c0  = 11111 = 1F    (Color 'c', 1) G_CCMUX_0
Aa0 = 011   = 3        (Alpha 'a', 1) G_ACMUX_PRIMITIVE
Ac0 = 001   = 1        (Alpha 'c', 1) G_ACMUX_TEXEL0
a1  = 1111  = F        (Color 'a', 2) G_CCMUX_0
c1  = 11111 = 1F    (Color 'c', 2) G_CCMUX_0

b0  = 1111  = F        (Color 'b', 1) G_CCMUX_0
b1  = 1111  = F        (Color 'b', 2) G_CCMUX_0
Aa1 = 011   = 3        (Alpha 'a', 2) G_ACMUX_PRIMITIVE
Ac1 = 001   = 1        (Alpha 'c', 2) G_ACMUX_TEXEL0
d0  = 011   = 3        (Color 'd', 1) G_CCMUX_PRIMITIVE
Ab0 = 111   = F        (Alpha 'b', 1) G_ACMUX_0
Ad0 = 111   = F        (Alpha 'd', 1) G_ACMUX_0
d1  = 011   = 3        (Color 'd', 2) G_CCMUX_PRIMITIVE
Ab1 = 111   = 7        (Alpha 'b', 2) G_ACMUX_0
Ad1 = 111   = 7        (Alpha 'd', 2) G_ACMUX_0

C1 (G_CCMUX_0 - G_CCMUX_0) * G_CCMUX_0 + G_CCMUX_PRIMITIVE
C2 (G_CCMUX_0 - G_CCMUX_0) * G_CCMUX_0 + G_CCMUX_PRIMITIVE
A1 (G_ACMUX_PRIMITIVE - G_ACMUX_0) * G_ACMUX_TEXEL0 + G_ACMUX_0
A2 (G_ACMUX_PRIMITIVE - G_ACMUX_0) * G_ACMUX_TEXEL0 + G_ACMUX_0

[LuigiBlood]

Okay I looked at more of the problem, it turns out the real problem is the lack of Coverage support for the modeling (at least for 80770800, for the model detection). On SetOtherMode_L, the game sets the coverage to full (CVG_DST_FULL), but the game also sets alpha = coverage value (ALPHA_CVG_SEL), effectively forcing alpha = 1 at all times when rendering things.

Of course there's still the lack of framebuffer writing because of the definition of an aux framebuffer.

EDIT: When it comes to Paint mode, while I seem to see what's going on with the black color instead of transparency, I'm not sure what's causing it.

Also I handled the first issue like this in the end:

    // Deal with Alpha = Coverage = 1.0f
    ssShader << "  if (uAlphaCvgSel != 0 && uCvgDest == 2) fragColor.a = 1.0;" << std::endl;

https://github.com/LuigiBlood/GLideN64/blob/56c3d58b6a8c06d6a7451abf1001f3749f049f4f/src/Graphics/OpenGLContext/GLSL/glsl_CombinerProgramBuilder.cpp#L2484

Which is fixing that problem in particular without doing a really bad combiner hack.

[LuigiBlood]

I have updated the first post for a more comprehensive list of problems with Polygon Studio, including screenshots, ways to access the problematic parts of the game, and descriptions of the problems.

[olivieryuyu]

@gonetz

There is quite some good lead on how this specific framebuffer effects works. I would also guess this what is actually used Paper Mario, Zelda OOT and MM.

[gonetz]

There is quite some good lead on how this specific framebuffer effects works.

What do you mean?

[olivieryuyu]

_"When rendering the model to that buffer, game sets SetOtherMode_L to CVG_DST_FULL and ALPHA_CVGSEL, which forces alpha value to be set to 1.0 but is not emulated."

I could image that conditional write to framebuffer could be made according to alpha value. This should be checked in Zelda OOT menu, Zelda MM camera and Paper Mario menu.

[ghost]

@LuigiBlood What is the current situation of this issue? With current master, I seem to be able to place models (although hover color isn't showing). Painting, taking photos and wireframe mode don't work.

I tried all sort of hacks to pixel alpha but I can't get any of the effects (other than seeing the model) working. Is the auxiliary framebuffer hack necessary for them to work?

[LuigiBlood]

Hover color not working is precisely the main problem. You can place objects, but you cannot actually select one, or being able to modify the object's vertexes and faces itself.

The auxiliary framebuffer hack is necessary because else it's like it doesn't really exist and doesn't really do anything.

[LuigiBlood]

Is someone still looking into this?

[ghost]

Personally, I never dug too deep. I am not too familiar with code related to framebuffer effects and I don't know the details about the way the emulators and the plugin interact regarding the shared RDRAM.

I only checked whether the emulation of alpha blending helped. If dual source blending or framebuffer fetch are supported in your device, the CVG_DEST_FULL part of the issue should be gone. But this seems to be only a minor part of the whole issue.

[LuigiBlood]

For the frame buffers this is the only potential fix I could come up with: https://github.com/LuigiBlood/GLideN64/blob/polygonstudio_hack/src/FrameBuffer.cpp#L692

[LuigiBlood]

So I added the Polygon Studio Aux Buffer """fix""" to current master for testing and the coverage value stuff is indeed fixed as I didn't need to do changes for it. I have updated the first post accordingly. I also noticed the Wire effect "works" but isn't perfect. The Paint stuff still is the same as described.

EDIT: The model detection seems a bit off, maybe a bit shifted to the left or something? Nevermind just disable the dithering. Also Pixel Coverage Calculation does not need to be enabled.