melonDS-emu / melonDS

DS emulator, sorta
https://melonds.kuribo64.net
GNU General Public License v3.0
3.18k stars 525 forks source link

Tweak to how polygon orientation is determined #2137

Open Jaklyy opened 1 month ago

Jaklyy commented 1 month ago

Makes small tweaks to clarify and improve the algorithm for determining polygon orientation. Main thing that sparked this dive was the mystery of how a polygon with a dot of 0 can be treated as either both facing directions or "neither" (always rendered) for poly face culling depending on whether the first/second vertex is overlapping the 3rd vertex. After essentially rewriting the entire algorithm and coming to almost exactly the same conclusions as the original algorithm i can now explain this by the fact that the initial vectors have a length of 0. Doesn't really explain why they decided to handle those vectors differently, but it at least gives us some excuse to go off of.

Current caveats: I have yet to figure out why we're doing normalization currently, but i kept it anyway. I'm assuming that i have simply yet to find some horrifying edge case. Update: i found the edge case, i... also found another edge case that isn't covered by it? i dunno what's going on there