search

yjh0502 / rust-s2

S2 geometry library in Rust
Apache License 2.0
77 stars 20 forks source link

Precise vector test #4

Closed haynes1 closed 5 years ago

haynes1 commented 5 years ago

All but one test pass.

The test that doesn't pass is test_precise_round_trip(). This test converts a f64 vector to a precise vector, then converts it back to an f64 vector and normalizes it. The result should be equivalent to just normalizing the original f64 vector but it isn't.

The problem is that after conversion, the post-conversion normalized vector is accurate to one more decimal place than the pre-conversion normalized vector.

pre: 0.2672612419124244 post: 0.26726124191242434

Potential Research: rust-lang/rust#24557 https://stackoverflow.com/questions/50361151/how-to-deal-with-inexact-floating-point-arithmetic-results-in-rust

yjh0502 commented 5 years ago

@haynes1 Any feedback so far? May I merge the PR and apply feedbacks by myself? I'm a bit worried that it might break your code as it changes API.