Wrong realtime lighting for vertex-lit surfaces

[slipher]

With vertex lighting, the wrong diffuse color is used for realtime lights. Instead of the basic diffuse texture, the result of multiplying the diffuse with the vertex color is used. This comes from the line diffuse *= var_Color; in lightMapping_fp.glsl. So dynamic lights are practically impossible to see on dimly lit vertex-lit surfaces such as parts of the station15 evergreen trees.

I guess it's structured this way because vertex lighting is a type of "rgbGen", at least in explicitly specified vertex lighting shaders where the syntax is rgbGen vertex. In principle, any rgbGen operation should be included in the diffuse. But obviously this is the wrong idea for vertex lighting, so we should try to detect the vertex lighting case and exclude that calculation from the diffuse used for lighting purposes. I guess the criteria for detecting vertex lighting can be rgbgen vertex + is bsp surface + no lightmap, in a lightmap stage.

[VReaperV]

Isn't vertex lighting supposed to be part of diffuse? If not, how is it expected to contribute to the final colour?

[slipher]

Diffuse color multiplied by vertex color. Vertex color is a light color serving the same function as lightmaps.