From what I understand, Edge maintains invariant that p1.y < p2.y and in case p1.y == p2.y then p1.x < p2.x. However if the 1e10 tolerance is used for comparison of Y coordinates, this invariant is broken for the second edge and the subsequent code crashes later during triangulation.
I would therefore suggest rolling back that commit...
https://github.com/jhasse/poly2tri/commit/e0ba327ed83f3e32933cf6cc4f61fabc50191711 caused a regression - this simple polygon will now crash during triangulation:
0 0 -5 -3e-10 -10 -2e-10 -10 -4 0 -4
From what I understand,
Edge
maintains invariant that p1.y < p2.y and in case p1.y == p2.y then p1.x < p2.x. However if the 1e10 tolerance is used for comparison of Y coordinates, this invariant is broken for the second edge and the subsequent code crashes later during triangulation.I would therefore suggest rolling back that commit...