[Feature request] Dithered palettisation and custom bitdepth

[theptoing]

I really like the new implementation of palettisation, looks very authentic. Though I was wondering if it would be possible to implement something like the "Balanced" dither that Ironwail has. It makes things look a lot more pleasant.

On a similar note, I really love the increased contrast/crunchiness that palettisation provides, but it does not work too well with modern maps, due to coloured lighting and fog. I think it would be neat if the bitdepth could be set manually, ideally per channel. Something like "vid_rgb 555" for example, which would clamp the colours to 15 bit RGB, which would give a good amount of more contrast but not mess things up with coloured lighting.

Depending on how you deal with colours, maybe even custom lut images could be cool. Is vkQuake actually reading out the colormap.lmp for the palettisation?

[Novum]

It's using the actual original Quake palette if that's what you are asking. Do mods having custom palettes? The generation of the lookup octree is pretty slow, I can't really do this at runtime.

[temx]

Is vkQuake actually reading out the colormap.lmp for the palettisation?

No, it uses palette.lmp, with a squared distance metric (the same that was used to generate the original colormap.lmp, but may differ due to precision / rounding). So it should be pretty close, but as Novum said it's hardcoded so it doesn't handle modifications.

but it does not work too well with modern maps,

A way to make this work better with colored lightmaps and fog is to do it independently per channel (I think FTE does this). That is, take the R channel of the lightmap and the fog, pretend it's gray (RRR), get result color, look it up in the palette, take the R channel of the palette color. Repeat for G and B. If you do this during rendering rather than post-process, you also get high bit-depth transparency/blending. Of course this ends up generating way more than 256 colors, but it's probably closer to the "intent".

The other big problem are RGB assets like skies: This is a content problem, they could be dithered down to 8 bits at load time for the best "authenticity".

"Balanced" dither that Ironwail has

IIRC Ironwail does texel space dithering in the balanced modes, switching to pixel space in the distance for fog and skies. You can see the dither noise added using r_lightmap 1. This alleviates both problems, but also does things like give different result colors for the same texture color + lightmap color pair. It's a good choice if you want to stick to 256 colors no matter what.

[Novum]

I did some hacky tests with bluenoise dithering

I think this could be improved by better measuring local variance, but then I need to first downsample the image in a separate pass. Right now it just measures variance by comparing the two best colors for each pixel.

[theptoing]

It's using the actual original Quake palette if that's what you are asking. Do mods having custom palettes? The generation of the lookup octree is pretty slow, I can't really do this at runtime.

There are some mods that use palettes, 2 I can think of are Malice, and Your Path of Destruction. YPOD in particular also has a new colormap which I think uses less fullbrights. This generally is broken in modern source ports since they do not read out the colormap.lmp. It would be neat if the engine would actually do that. It would also allow for fixes to the fullbrights, which should actually go brighter, just not darker.

[Novum]

Again, I can't recompute the lookup tables at runtime, so it won't happen.

[Novum]

ce065d8ee043f2efa43dad4ed0fa2888ab22a647