MEGA65: VIC-IV "glyph_width" anomalies

[lgblgblgb]

In vic4.c we can see the following for glyph_width calculation:

Uint8 glyph_width_deduct = SXA_TRIM_RIGHT_BITS012(char_value) + (SXA_TRIM_RIGHT_BIT3(char_value) ? 8 : 0);
Uint8 glyph_width = (SXA_4BIT_PER_PIXEL(color_data) ? 16 : 8) - glyph_width_deduct;

There can be at least two problems caused by this, as far I can guess.

First problem: hard-coded glyph_width for FCM?

First, when calling vic4_render_fullcolor_char_row() the glyph_width argument of that function is set to 8 as a constant. I'm not very sure why this is the case, in FCM we can have only glyph_width as 8? But then the calculation part above is useless as never used in FCM mode at least. To check: Are we sure, that glyph_width must be hardcoded to 8 in FCM? Please note, that this FCM problem (?) is not connected to the second problem here (which is NCM) I've just noticed that it's hard coded in the source as 8. So these two problems are two separated things, though common in the sense "how glyph with should be handled" at least.

Second problem: Another World strange anomalies in NCM

The second case is even more interesting. In the fantastic MEGA65 of Another World we can see a very strange effect in Xemu. Another World (AW for short from now) uses NCM (nybble-colour-mode, that is 16 colours). It seems every other frame shown is "compressed" for whatever reason.

Example for a good frame:

Example for a bad ("compressed") frame:

Note, that on the "bad frame" there is even a strange rectangle missing at the side, that would be nice to solve as well, but let's forget it now here at least, and focus on main problem. Another problem here just for completeness, the discrete "breaks" in the image, for example see the diagonal line of the building at the right side.

The cause of the AW problem

According to my discoveries: AW uses two screen buffers and one seems to have using the feature of 16 colour palette "shifting" by setting the upper 4 bits of the palette. But it seems that is exactly what Xemu thinks having something to do with glyph_width! Now it seems glyph_width should be calculated differently at least in NCM mode?

To prove my idea, I temporary hardcoded a fixed value for glyph_with for NCM renderer, and the problem goes away (besides that odd rectangle still at the right side), really. No wonder, the renderer does what it was told, however the glyph_width seems to be garbage in NCM mode. Also the "discrete breaks" problem goes away (but again, that strange rectangle missing problem remains).

About MEGA65 port of Another World

As far as I know (for sure, I can be wrong, or the situation could change since then ...), AW is not "in the public" yet, thus I cannot include any testing material. Majikeyric (from Discord) kindly provided me a D81 image with asking me, not to distribute it, so of course I won't. If somebody want to test this with Xemu - I guess - Majikeyric should be asked for a D81 image to be able to do so.

My plan

I'm about to modify vic4.c to remove the glyph_width calculation before selecting the renderer, and implement the calculation when calling the selected renderer. This way:

• there is no useless calculation when it's not even used (in FCM) though it must be checked if it is not a mistake that we use a hard-coded value only for FCM!
• if there is difference (like in NCM?) the calculation of glyph_width can be solved another way there
• we also save some very minor processing power, since the current calculation of glyph_width has a condition which is not needed if calculation is done when render is already selected

Still, I have no idea how to solve the "AW problem". I can hard-code the "right" value not to have this problem, but I have the suspect it's not the correct solution. Also it's strange that for FCM, the value is hard coded.

Random wild thought

I even had the idea that maybe NCM is meant to have fixed glyph_width while FCM not, just by mistake it has been swapped in the source and now FCM is mixed and NCM has kind of broken implementation of it.

Bonus

Yet further (though maybe it's not scope of this issue): that rectangle at the right is strange ... It can be seed on the "bad frame" example. Even with my workaround to fix glyph_width there is the flashing rectangle there, since one frame misses that rectangle and one has it, no idea what it can be ... But with the workaround at least other parts of the screen is OK and not flashing like crazy between "normal" and "half-compressed-horizontally" states.

Summary

I'd love if @hernandp can check this situation as well ... :)

[lgblgblgb]

About the commit above (cc0268b771c47bfc6527ea7b83829a1df6dd68a2): what I talked about at section "My plans" above. Now it hard codes 16 for glyph_width for NCM which causes AW to works "quite well" (still the rectangle problem at the side). However possible this is just a workaround and can break other things. Also, the question why FCM uses a hard-coded value is not addressed by this commit at all.

Also, I've re-formatted the calls of the renderers themselves into multi-line versions, otherwise it would be a bit hard now to follow what's going on.

[hernandp]

Hey @lgblgblgb looking at this. According to the glyph_width calculation it must only apply to 16-bit character mode extended attributes (trim right plus 4bit per pixel). Following that, I could write a fast testing program and compare hardware vs. xemu. I guess that FCM hardcoded to 8 was my "assumption" to get things running, but actually should be affected by Xtended character attributes.

[lgblgblgb]

As I've stated in the PR, thank you, looks fine, I've also tested with Shallan's VIC4 test suite, I cannot see any negative side effects, though maybe those tests does not do anything about glyph_width, but still. Another World seems to work now as well, the strange "missing rectange at every two frames" problem still there, but I really think, it's must be another problem (hehe another world -> another problem). Therefore I close this issue for now.

[lgblgblgb]

Specific issue on the "flickering rectangle" problem: #304