Improve lighting

[OFFTKP]

Uncomment lighting on Android

• Made lightingLUT's type GL_TEXTURE_2D and u_tex_lighting_lut to sampler2D
• Made lightingLUT's internalFormat GL_R32F, and converting the LUT data to floats before uploading. Internal format and format types need to match according to the spec: My guess is the conversion to float was happening implicitly in desktop GL at some point, because lutLookup returns a float. The type GL_R16 was probably interpreted as GL_R16F ? The alternative to this could be using GL_R16UI, but then we'd need to convert to float on the GPU
• Switched to using texelFetch instead of texture in the shader, since this texture is really treated as an array and we don't need filtering or interpolation
• Updated the gles.patch

Improve lighting implementation

• Fix light_vector not being properly updated for positional lighting
• Disabled factors were being set to 0 instead of 1 (see Lego Batman's fix)
• GPUREG_LIGHTING_LUTINPUT_ABS was not handled correctly (it's 0 == enabled, abs was on output instead of input, would use wrong bits)
• GPUREG_LIGHTING_LIGHT_PERMUTATION was not handled correctly (it was accessed every 3 bits when it's every 4 bits per enabled light)
• Fix spotlight attenuation, doesn't use LUT 2 but LUT's 8-15
• Fix environment 4 (SP), which uses GPUREG_LIGHTi_SPOTDIR, it's a fixed point number and not a floating point number as it was treated before
• Implement distance attenuation
• Implement geometric factors
• Implement specular clamping
• Clamp results at the end of calcLighting
• Removed unused v_tangent, v_bitangent, but might need them for bump lighting at some point (although they'll be treated differently)
• Removed pointless normalization of v_normal as it already comes in normalized from the vertex shader
• Probably other stuff I forgot

Known bugs

• Luigi's mansion:

  • Luigi Luigi's lighting is mostly fixed except when he opens the door he looks like he's seen a ghost:
  • The birds in the intro are too bright, edit: are they now? unsure
  • Spotlight just decides to stop working sometimes. It continues working when thunder strikes for a short while, then it stops working and the whole room goes brighter Fixed, GPUREG_LIGHTING_LIGHT_PERMUTATION was being accessed incorrectly
  • In the intro when thunder strikes twice the road isn't illuminated Fixed

• Cave Story 3D:

  • The evil doctor (and the rest of the scene) in the boot up of the game is wrongly lit. Game is better lit than before overall, was almost completely dark This should be fixed now post the Luigi spotlight fix

• Rabbids:

  • Positional lighting seems to work now, however there's some light flickering in the level selector near the castle Seems fixed after the Luigi spotlight fix, can't see any flickering

• Captain Toad Treasure Tracker:

  • Entire level is still too bright but better than before

• Pokedex 3D:

  • Pokemon not lit correctly, needs bump mapping

• Pushmo/Crashmo:

  • Seemingly same light issues as before on the characters

• Lego Batman:

  • There's supposed to be some glare on top of the legos, there isn't:

Improvements

• toon_shading.elf and fragment_lighting.elf seem to work fine
• Pokemon Omega Ruby is no longer too bright everywhere, lighting seems fine to me
• Pokemon SMD, same
• Super Mario 3D Land, same
• Cave Story 3D looks fine to me
• Luigi's mansion is better but there's still issues
• Captain Toad Treasure Tracker is better but still too bright
• Kirby Triple Deluxe seems less bright on his head and that looks closer to playthoughs I see online
• Lego Batman is better, should be some specular lighting missing on the lego heads

This patch makes it so that the shader editor shows the LUTs, useful for debugging luts.txt

[wheremyfoodat]

cc @skylersaleh

[wheremyfoodat]

Someone is cooking with this one 👀 (Stolen from Paris' discord messages)

[wheremyfoodat]

Closed this accidentally deleting the other branch

[OFFTKP]

Needs vertex shader to be changed in specialized shader PR

[OFFTKP]

TODO: bitfieldExtractCompat needs to sign extend for spotdir, or do something else