I tested a little your library and I created a case which is incorrectly tessellated. I attached a test file - it contains the problematic contour. First line contains number of vertices and then followed vertices (x, y, z values split by spaces).
contour3.txt
I use the LibTess2 library with doubles instead of floats (TESSreal type). The problematic intersection is between edge (2-1) and edge (75-76) (numbers are the indices of vertices). I tried also to debug the the code in sweep.c but I didn't find exactly where the problem is. If all vertices are slightly rotated around z-axis (like by 5 degrees) than the tessellation is correct.
note: option TESS_CONSTRAINED_DELAUNAY_TRIANGULATION changes the output triangles, but it produces same filled area
I tested a little your library and I created a case which is incorrectly tessellated. I attached a test file - it contains the problematic contour. First line contains number of vertices and then followed vertices (x, y, z values split by spaces). contour3.txt
I use the LibTess2 library with doubles instead of floats (TESSreal type). The problematic intersection is between edge (2-1) and edge (75-76) (numbers are the indices of vertices). I tried also to debug the the code in sweep.c but I didn't find exactly where the problem is. If all vertices are slightly rotated around z-axis (like by 5 degrees) than the tessellation is correct.
note: option TESS_CONSTRAINED_DELAUNAY_TRIANGULATION changes the output triangles, but it produces same filled area
The wrong tessellation: