Fix floating point precision failure caused by FMA on ppc64le

twpayne / go-geom

Package geom implements efficient geometry types for geospatial applications.

BSD 2-Clause "Simplified" License

839 stars 104 forks source link

Fix floating point precision failure caused by FMA on ppc64le #214

Closed prashantkhoje closed 2 years ago

prashantkhoje commented 2 years ago

This was exposed while testing CockroachDB. The geomfn/TestShortestLineString fails due to floating point precision differences caused by FMA on ppc64le. See https://go.dev/ref/spec#Floating_point_operators for more details.

twpayne commented 2 years ago

Same comments as for #213.

twpayne commented 2 years ago

Also, please can you rebase this on the latest master to pick up the CI fixes. Thanks!