Fix tutti-frutti for transparent textures

[plumsinus]

Since you've already fixed TF from short textures, and medusa from multi-patch textures, I think it would make sense to fix this too. Transparent textures used as an upper or lower texture, or on a 1-sided wall as a mid texture, produce visual glitches from undefined results.

The "usual" way that other ports handle this is to replace the transparency with black, though this isn't exactly standard - even PrBoom+ looks a bit glitchy in software mode.

example file: https://www.mediafire.com/?spsnd055tk59txm

[fabiangreffrath]

Phew, I already gave up wrapping my head around the original Medusa effect and decided to plainly steal the corresponding code from MBF instead. The downside is that the effect you describe is still present in MBF. :/

Which ports do have this fixed apart from PrBoom+?

[plumsinus]

Not sure offhand, except that GL ports usually seem to draw it OK, not that that is much help. Pretty sure ZDoom fixes it at least. I'll do a test of some more software ports tomorrow.

[plumsinus]

Software ports:

• ZDoom and Eternity both display the transparent area as black.
• PrBoom+ displays the transparent areas glitchy, but the non-transparent areas are drawn correctly.
• Legacy, Doom Retro, Boom, and Doom95 all display identical to vanilla.
• I did not test vavoom in software since the last version was OpenGL only. Will try it later...

All GL ports tested draw the transparent area as black, or as actually transparent in the case of 3DGE and Legacy.

[fabiangreffrath]

Thanks for checking! I just bisected PrBoom and the glitch was fixed between 2.4.4 and 2.4.5, which was exactly when "Unified software and opengl engine into one binary" and "Merged new internal patch (graphics) format from PrBoom 2.3". So this doesn't have very much to do with the Vanilla renderer anymore. :/

[fabiangreffrath]

And it wasn't fixed in SMMU 3.30 either.

[fabiangreffrath]

Let's just get that clear: What are we supposed to see through them, the sky?

The problem is that upper and lower textures are treated completely different than mid textures, i.e they follow different code paths. And they are very strictly not supposed to be transparent at all. :p

This will be an interesting fix -- if I ever get to it, that is. Especially because the handful of software renderes that already have it fixed have evolved a long way from the Vanilla renderer and their changes are not applicable to Crispy at all. The others are (or have merged) OpenGL renderes...

Is there any PWAD out there that exposes this glitch or is this purely academic?

[plumsinus]

Let's just get that clear: What are we supposed to see through them, the sky?

Ideally, nothing, just black where the transparent areas would normally be. Even "less glitchy than it is in vanilla" would be an improvement though.

Is there any PWAD out there that exposes this glitch or is this purely academic?

Yes, several, NOVA MAP13 has it and I have seen it elsewhere. I consider it a mapping error, but since all ZDoom-based ports and most OpenGL ports render the transparent area black, many people seem to treat that as the standard for limit-removing maps.

[plumsinus]

Example of how it should ideally look, taken with ZDoom:

[fabiangreffrath]

Gosh, I still have no idea where to even start hacking on this...

[fabiangreffrath]

So, I have investigated this issue a bit: What we are seeing in the areas that are supposed to be black in the textures is not a Tutti-Frutti effect, i.e. it is not random pixel values read from memory. Instead, they are the actual topdelta and length bytes of the posts that the individual columns of the patches are composed of. The reason why we see them is that Vanilla Doom renders upper and lower textures as well as middle textures on one-sided walls with the R_DrawColumn() function, whereas it only renders middle textures on two-sided walls with the R_DrawMaskedColumn() function. The former function, however, ignores the fact that a texture may contain transparent areas and may thus be composed of multiple posts -- instead, it dully draws them column by column. Only the latter function interprets the topdelta and length bytes of the posts correctly and and thus considers transparent areas. That is, these bytes are overridden when a texture is composed of multiple patches in R_DrawColumnInCache() which causes the Medusa-Effect, but this is already taken care of.

[fabiangreffrath]

Alright, I am closer.

The main problem is that the engine does not even consider single-patched transparent textures. For single-patched textures, R_GetColumn() returns the raw patch column, whereas for multi-patched textures (strictly: for multi-patched columns) it returns the composited column. The logic is: single patch -> opaque, multiple patches -> probably transparent.

What we need here, however, are composited columns even for a single-patched textures. Else, the raw patch column is passed over to R_DrawColumn() and we end up with the column->topdelta and column->length bytes of the (> 1) posts that make up this column, which we see as the odd pixels in the texture rendering.

So, I need to do multiple things: 1) Create composites even for single-patched textures 2) Tell R_GetColumn() to either return the raw patch column for single-patched mid-textures on 2-sided lines (only!) or the composed column for all other textures. 3) I will need to maintain separate column offsets for both cases

And then I will have to clean up this mess. ;)