Closed thestinger closed 4 years ago
Nominating
accepted for P-high, same reasoning as #10183
I don't think this is backwards incompatible at a language level. It will not cause code that was working OK to stop working. Nominating.
changing to P-high, same reasoning as #10183
How do we propose to solve this and #10185? Since whether behaviour is defined or not depends on the dynamic value of the number being cast, it seems the only solution is to insert dynamic checks. We seem to agree we do not want to do that for arithmetic overflow, are we happy to do it for cast overflow?
We could add an intrinsic to LLVM that performs a "safe conversion". @zwarich may have other ideas.
AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.
Oh, that's easy enough then.
ping @pnkfelix is this covered by the new overflow checking stuff?
These casts are not checked by rustc with debug assertions.
I'm happy to handle this, but I need a concrete solution. I personally think that it should be checked along with overflowing integer arithmetic, as it's a very similar issue. I don't really mind what we do though.
Note that this issue is currently causing an ICE when used in certain constant expressions.
This allows violating memory safety in safe rust, example from this forum post:
Undefs, huh? Undefs are fun. They tend to propagate. After a few minutes of wrangling..
#[inline(never)]
pub fn f(ary: &[u8; 5]) -> &[u8] {
let idx = 1e100f64 as usize;
&ary[idx..]
}
fn main() {
println!("{}", f(&[1; 5])[0xdeadbeef]);
}
segfaults on my system (latest nightly) with -O.
Marking with I-unsound given the violation of memory safety in safe rust.
@bluss , this does not segfualt for me, just gives an assertion error. untagging since i was the one who added it
Sigh, I forgot the -O, re-tagging.
re-nominating for P-high. Apparently this was at some point P-high but got lower over time. This seems pretty important for correctness.
EDIT: didn’t react to triage comment, adding label manually.
It seems like the precedent from the overflow stuff (e.g. for shifting) is to just settle on some behavior. Java seems to produce the result modulo the range, which seems not unreasonable; I'm not sure just what kind of LLVM code we'd need to handle that.
According to https://docs.oracle.com/javase/specs/jls/se7/html/jls-5.html#jls-5.1.3 Java also guarantees that NaN
values are mapped to 0 and infinities to the minimum/maximum representable integer. Moreover the Java rule for the conversion is more complex than just wrapping, it can be a combination of saturation (for the conversion to int
or long
) and wrapping (for the conversion to smaller integral types, if needed). Replicating the whole conversion algorithm from Java is certainly possible, but it would require a fair amount of operations for every cast. In particular, in order to ensure that the result of a fpto[us]i
operation in LLVM does not exhibit undefined behaviour, a range check would be needed.
As an alternative, I would suggest that float->int casts are guaranteed to only be valid if the truncation of the original value can be represented as a value of the destination type (or maybe as [iu]size
?) and to have assertions on debug builds that trigger a panic when the value has not been represented faithfully.
The main advantages of the Java approach are that the conversion function is total, but this also means that unexpected behaviour might creep in: it would prevent undefined behaviour, but it would be easy to be tricked into not checking if the cast actually made any sense (this is unfortunately true also for the other casts :worried: ).
The other approach matches the one currently used for arithmetic operations: simple & efficient implementation in release, panics triggered by range checking in debug. Unfortunately unlike other as
casts, this would make such conversion checked, which can be surprising to the user (although maybe the analogy to arithmetic operations can help here). This would also break some code, but AFAICT it should only happen for code which is currently relying on undefined behaviour (i.e. it would replace the undefined behaviour "let's return any integer, you obviously don't care which" with a panic).
The problem isn't "let's return any integer, you obviously don't care which", it is that it causes an undef which isn't a random value but rather a nasal demon value and LLVM is allowed to assume the undef never occurs enabling optimizations that do horrible incorrect things. If it was a random value, but crucially not undef, then that would be enough to fix the soundness issues. We don't need to define how unrepresentable values are represented, we just need to prevent undef.
Discussed in @rust-lang/compiler meeting. The most consistent course of action remains:
The main problem is that we need a concrete suggestion for option 2.
triage: P-medium
@nikomatsakis Does as
ever currently panic in debug builds? If it doesn't, for consistency and predictability it seems preferable to keep it that way. (I think it should have, just like arithmetic, but that's a separate, and past, debate.)
otherwise, we need a fallback behavior, it should be something which has minimal (ideally, zero) runtime cost for valid values, but the precise behavior is not that important, so long as it is not LLVM undef.
Concrete suggestion: extract digits and exponent as u64
and bitshift digits by exponent.
fn f64_as_u64(f: f64) -> u64 {
let (mantissa, exponent, _sign) = f.integer_decode();
mantissa >> ((-exponent) & 63)
}
Yes it's not zero cost, but it's somewhat optimizable (would be better if we marked integer_decode inline
) and at least deterministic. A future MIR-pass that expands a float->int cast could probably analyze whether the float is guaranteed to be ok to cast and skip this heavy conversion.
Does LLVM not have platform intrinsics for the conversion functions?
EDIT: @zwarich said (a long time ago):
AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.
Why even bother panicking? AFAIK, @glaebhoerl is correct, as
is supposed to truncate/extend, not check the operands.
On Sat, Mar 05, 2016 at 03:47:55AM -0800, Gábor Lehel wrote:
@nikomatsakis Does
as
ever currently panic in debug builds? If it doesn't, for consistency and predictability it seems preferable to keep it that way. (I think it should have, just like arithmetic, but that's a separate, and past, debate.)
True. I find that persuasive.
On Wed, Mar 09, 2016 at 02:31:05AM -0800, Eduard-Mihai Burtescu wrote:
Does LLVM not have platform intrinsics for the conversion functions?
EDIT:
AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.
Why even bother panicking? AFAIK, @glaebhoerl is correct,
as
is supposed to truncate/extend, not check the operands.
Yes, I think I was mistaken before. as
is the "unchecked truncation"
operator, for better or worse, and it seems best to stay consistent
with that philosophy. Using target-specific intrinsics may be a perfectly
fine solution though?
@nikomatsakis: it seems the behavior hasn't been defined yet? Can you give an update about the planning regarding that?
Just ran into this with much smaller numbers
let x: f64 = -1.0;
x as u8
Results in 0, 16, etc. depending on optimizations, I was hoping it would be defined as 255 so I don't have to write x as i16 as u8
.
@gmorenz Did you try !0u8
?
In context that wouldn't make sense, I was getting the f64 from a transformation on data sent over the network, with a range of [-255, 255]. I was hoping it would wrap nicely (in the exact way that <i32> as u8
wraps).
Here's a recent LLVM proposal to "kill undef" http://lists.llvm.org/pipermail/llvm-dev/2016-October/106182.html , though I'm hardly knowledgeable enough to know whether or not this would automagically resolve this issue.
They're replacing undef with poison, the semantics being slightly different. It's not going to make int -> float casts defined behavior.
We probably should provide some explicit way to do a saturating cast? I’ve wanted that exact behaviour just now.
Seems like this should be marked I-crash, given https://github.com/rust-lang/rust/issues/10184#issuecomment-139858153 .
We had a question about this in #rust-beginners
today, someone ran into it in the wild.
The book I'm writing with @jimblandy, Programming Rust, mentions this bug.
Several kinds of casts are permitted.
Numbers may be cast from any of the built-in numeric types to any other.
(...)
However, as of this writing, casting a large floating-point value to an integer type that is too small to represent it can lead to undefined behavior. This can cause crashes even in safe Rust. It is a bug in the compiler, github.com/rust-lang/rust/issues/10184.
Our deadline for this chapter is May 19. I'd love to delete that last paragraph, but I feel like we should at least have some kind of plan here first.
Apparently current JavaScriptCore uses an interesting hack on x86. They use the CVTTSD2SI instruction, then fall back on some hairy C++ if the value is out of range. Since out-of-range values currently explode, using that instruction (with no fallback!) would be an improvement on what we have now, albeit only for one architecture.
Honestly I think we should deprecate numeric casts with as
and use From
and TryFrom
or something like the conv crate instead.
Maybe so, but that seems orthogonal to me.
OK, I've just re-read this whole conversation. I think there is agreement that this operation should not panic (for general consistency with as
). There are two leading contenders for what the behavior ought to be:
It's not clear to me if there is a clear precedent for what the result ought to be in the first case?
After having written that out, my preference would be to maintain a deterministic result. I feel like every place that we can hold the line on determinism is a win. I am not really sure what the result ought to be though.
I like saturation because I can understand it and it seems useful, but it seems somehow incongruent with the way that u64 as u32
does truncation. So perhaps some sort of result based on truncation makes sense, which I guess is probably what @oli-obk proposed -- I don't fully understand what that code is intended to do. =)
My code gives the correct value for things in the range 0..2^64 and deterministic but bogus values for everything else.
floats are represented by mantissa ^ exponent, e.g. 1.0
is (2 << 52) ^ -52
and since bitshifts and exponents are the same thing in binary, we can just reverse the shift (thus the negation of the exponent and the right shift).
+1 for determinism.
I see two semantics that make good sense for humans, and I think we should pick whichever one is faster for values that are in range, when the compiler can't optimize away any of the computation. (When the compiler knows that a value is in range, both options give the same results, so they are equally optimize-able.)
IntType::max_value()
/min_value()
)The table below is meant to specify both options fully. T
is any machine integer type. Tmin and Tmax are T::min_value()
and T::max_value()
. RTZ(v) means take the mathematical value of v and Round Toward Zero to get a mathematical integer.
v |
v as T (saturation) |
v as T (modulo) |
---|---|---|
in range (Tmin <= v <= Tmax) | RTZ(v) | RTZ(v) |
negative zero | 0 | 0 |
NaN | 0 | 0 |
Infinity | Tmax | 0 |
-Infinity | Tmin | 0 |
v > Tmax | Tmax | RTZ(v) truncated to fit T |
v < Tmin | Tmin | RTZ(v) truncated to fit T |
The ECMAScript standard specifies operations ToInt32, ToUint32, ToInt16, ToUint16, ToInt8, ToUint8, and my intent with the "modulo" option above is to match those operations in every case.
ECMAScript also specifies ToInt8Clamp which does not match either case above: it does "round half to even" rounding on fractional values rather than "round to zero".
@oli-obk's suggestion is a third way, worth considering if it's faster to compute, for values that are in range.
@oli-obk What about signed integer types?
Throwing another proposal into the mix: Mark u128 casts to floats as unsafe and force folks to explicitly choose a way of handling it. u128 is pretty rare currently.
@Manishearth I’d hope for similar semantics integers → floats as floats → integers. Since both are UB-ful, and we cannot make float→integer unsafe anymore, we should probably avoid making integer→float unsafe as well.
For float→integer saturating will be faster AFAICT (resulting in a sequence of and
, test+jump float comparison and jump, all for 0.66 or 0.5 2-3 cycles on modern arches). I personally couldn’t care less for what exact behaviour we decide on as long as the in-range values are as fast as they possibly could be.
Wouldn't it make sense to make it behave like overflow? So in a debug build it would panic if you do a cast with undefined behaviour. Then you could have methods for specifying the casting behaviour like 1.04E+17.saturating_cast::<u8>()
, unsafe { 1.04E+17.unsafe_cast::<u8>() }
and potentially others.
Oh, I thought the issue was only for u128, and we can make that unsafe both ways.
@cryze UB should not exist even in release mode in safe code. The overflow stuff is still defined behavior.
That said, panic on debug, and
This affects:
f32 -> u8, u16, u32, u64, u128, usize
(-1f32 as _
for all, f32::MAX as _
for all but u128) f32 -> i8, i16, i32, i64, i128, isize
(f32::MAX as _
for all)f64
-> all ints (f64::MAX as _
for all)f32::INFINITY as u128
is also UB
Status as of 2020-04-18
We intend to stabilize the saturating-float-casts behavior for
as
, and have stabilized unsafe library functions that handle the previous behavior. See #71269 for the latest discussion on that stabilization process.Status as of 2018-11-05
A flag has been implemented in the compiler,
-Zsaturating-float-casts
, which will cause all float to integer casts have "saturating" behavior where if it's out of bounds it's clamped to the nearest bound. A call for benchmarking of this change went out awhile ago. Results, while positive in many projects, are quite negative for some projects and indicates that we're not done here.The next steps are figuring out how to recover performance for these cases:
as
cast behavior (which is UB in some cases) and addunsafe
functions for the relevant types and such.freeze
concept which means that we get a garbage bit pattern, but it's at least not UBOld status
UPDATE (by @nikomatsakis): After much discussion, we've got the rudiments of a plan for how to address this problem. But we need some help with actually investigating the performance impact and working out the final details!
ORIGINAL ISSUE FOLLOWS: