search

CQCL / hugr-llvm

http://crates.io/crates/hugr-llvm
Apache License 2.0
5 stars 2 forks source link

Investigate weird int -> float conversions #103

Open croyzor opened 1 month ago

croyzor commented 1 month ago

We have a test case for int -> float -> int roundtrips which displays some confusing behaviour. See comment in code highlighted here.

_Originally posted by @mark-koch in https://github.com/CQCL/hugr-llvm/pull/94#discussion_r1750525367_

doug-q commented 2 days ago

The issue here is that fptoui (floating-point-to-unsigned-int) and fptosi(floating-point-to-signed-int) return poison when their (floating point) argument "does not fit"(quoting lang ref) in the return type.

our logic is at present:

fn tunc_u(f: f64) -> Option<u64> {
  if (0u64 as f64) <= f && f <= (u64::MAX as f64) { Some(f as f64) } else { None }
}

If I change the <=s to < then the test behaves sensibly. I conclude that u64::MAX as f64 is too large to fit in a u64 because rounding, and the strange results we are seeing are poison.

The lower-bound check should surely be less-than-equal for the unsigned case,unclear for the signed case. A bit more investigation needed to pin this down exactly.