floating point to integer casts can cause undefined behaviour

thestinger commented 10 years ago

Status as of 2020-04-18

We intend to stabilize the saturating-float-casts behavior for as, and have stabilized unsafe library functions that handle the previous behavior. See #71269 for the latest discussion on that stabilization process.

Status as of 2018-11-05

A flag has been implemented in the compiler, -Zsaturating-float-casts, which will cause all float to integer casts have "saturating" behavior where if it's out of bounds it's clamped to the nearest bound. A call for benchmarking of this change went out awhile ago. Results, while positive in many projects, are quite negative for some projects and indicates that we're not done here.

The next steps are figuring out how to recover performance for these cases:

One option is to take today's as cast behavior (which is UB in some cases) and add unsafe functions for the relevant types and such.
Another is to wait for LLVM to add a freeze concept which means that we get a garbage bit pattern, but it's at least not UB
Another is to implement casts via inline assembly in LLVM IR, as the current codegen is not heavily optimized.

Old status

UPDATE (by @nikomatsakis): After much discussion, we've got the rudiments of a plan for how to address this problem. But we need some help with actually investigating the performance impact and working out the final details!

ORIGINAL ISSUE FOLLOWS:

If the value cannot fit in ty2, the results are undefined.

1.04E+17 as u8

brson commented 10 years ago

Nominating

pnkfelix commented 10 years ago

accepted for P-high, same reasoning as #10183

pcwalton commented 10 years ago

I don't think this is backwards incompatible at a language level. It will not cause code that was working OK to stop working. Nominating.

pnkfelix commented 10 years ago

changing to P-high, same reasoning as #10183

nrc commented 9 years ago

How do we propose to solve this and #10185? Since whether behaviour is defined or not depends on the dynamic value of the number being cast, it seems the only solution is to insert dynamic checks. We seem to agree we do not want to do that for arithmetic overflow, are we happy to do it for cast overflow?

pcwalton commented 9 years ago

We could add an intrinsic to LLVM that performs a "safe conversion". @zwarich may have other ideas.

zwarich commented 9 years ago

AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.

pcwalton commented 9 years ago

Oh, that's easy enough then.

nrc commented 9 years ago

ping @pnkfelix is this covered by the new overflow checking stuff?

bluss commented 9 years ago

These casts are not checked by rustc with debug assertions.

Aatch commented 8 years ago

I'm happy to handle this, but I need a concrete solution. I personally think that it should be checked along with overflowing integer arithmetic, as it's a very similar issue. I don't really mind what we do though.

Note that this issue is currently causing an ICE when used in certain constant expressions.

bluss commented 8 years ago

This allows violating memory safety in safe rust, example from this forum post:

Undefs, huh? Undefs are fun. They tend to propagate. After a few minutes of wrangling..

#[inline(never)]
pub fn f(ary: &[u8; 5]) -> &[u8] {
    let idx = 1e100f64 as usize;
    &ary[idx..]
}

fn main() {
    println!("{}", f(&[1; 5])[0xdeadbeef]);
}

segfaults on my system (latest nightly) with -O.

steveklabnik commented 8 years ago

Marking with I-unsound given the violation of memory safety in safe rust.

steveklabnik commented 8 years ago

@bluss , this does not segfualt for me, just gives an assertion error. untagging since i was the one who added it

steveklabnik commented 8 years ago

Sigh, I forgot the -O, re-tagging.

nagisa commented 8 years ago

re-nominating for P-high. Apparently this was at some point P-high but got lower over time. This seems pretty important for correctness.

EDIT: didn’t react to triage comment, adding label manually.

nikomatsakis commented 8 years ago

It seems like the precedent from the overflow stuff (e.g. for shifting) is to just settle on some behavior. Java seems to produce the result modulo the range, which seems not unreasonable; I'm not sure just what kind of LLVM code we'd need to handle that.

ranma42 commented 8 years ago

According to https://docs.oracle.com/javase/specs/jls/se7/html/jls-5.html#jls-5.1.3 Java also guarantees that NaN values are mapped to 0 and infinities to the minimum/maximum representable integer. Moreover the Java rule for the conversion is more complex than just wrapping, it can be a combination of saturation (for the conversion to int or long) and wrapping (for the conversion to smaller integral types, if needed). Replicating the whole conversion algorithm from Java is certainly possible, but it would require a fair amount of operations for every cast. In particular, in order to ensure that the result of a fpto[us]i operation in LLVM does not exhibit undefined behaviour, a range check would be needed.

As an alternative, I would suggest that float->int casts are guaranteed to only be valid if the truncation of the original value can be represented as a value of the destination type (or maybe as [iu]size?) and to have assertions on debug builds that trigger a panic when the value has not been represented faithfully.

The main advantages of the Java approach are that the conversion function is total, but this also means that unexpected behaviour might creep in: it would prevent undefined behaviour, but it would be easy to be tricked into not checking if the cast actually made any sense (this is unfortunately true also for the other casts :worried: ).

The other approach matches the one currently used for arithmetic operations: simple & efficient implementation in release, panics triggered by range checking in debug. Unfortunately unlike other as casts, this would make such conversion checked, which can be surprising to the user (although maybe the analogy to arithmetic operations can help here). This would also break some code, but AFAICT it should only happen for code which is currently relying on undefined behaviour (i.e. it would replace the undefined behaviour "let's return any integer, you obviously don't care which" with a panic).

retep998 commented 8 years ago

The problem isn't "let's return any integer, you obviously don't care which", it is that it causes an undef which isn't a random value but rather a nasal demon value and LLVM is allowed to assume the undef never occurs enabling optimizations that do horrible incorrect things. If it was a random value, but crucially not undef, then that would be enough to fix the soundness issues. We don't need to define how unrepresentable values are represented, we just need to prevent undef.

nikomatsakis commented 8 years ago

Discussed in @rust-lang/compiler meeting. The most consistent course of action remains:

when overflow checks are enabled, check for illegal casts and panic.
otherwise, we need a fallback behavior, it should be something which has minimal (ideally, zero) runtime cost for valid values, but the precise behavior is not that important, so long as it is not LLVM undef.

The main problem is that we need a concrete suggestion for option 2.

nikomatsakis commented 8 years ago

triage: P-medium

glaebhoerl commented 8 years ago

@nikomatsakis Does as ever currently panic in debug builds? If it doesn't, for consistency and predictability it seems preferable to keep it that way. (I think it should have, just like arithmetic, but that's a separate, and past, debate.)

oli-obk commented 8 years ago

otherwise, we need a fallback behavior, it should be something which has minimal (ideally, zero) runtime cost for valid values, but the precise behavior is not that important, so long as it is not LLVM undef.

Concrete suggestion: extract digits and exponent as u64 and bitshift digits by exponent.

fn f64_as_u64(f: f64) -> u64 {
    let (mantissa, exponent, _sign) = f.integer_decode();
    mantissa >> ((-exponent) & 63)
}

Yes it's not zero cost, but it's somewhat optimizable (would be better if we marked integer_decode inline) and at least deterministic. A future MIR-pass that expands a float->int cast could probably analyze whether the float is guaranteed to be ok to cast and skip this heavy conversion.

eddyb commented 8 years ago

Does LLVM not have platform intrinsics for the conversion functions?

EDIT: @zwarich said (a long time ago):

AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.

Why even bother panicking? AFAIK, @glaebhoerl is correct, as is supposed to truncate/extend, not check the operands.

nikomatsakis commented 8 years ago

On Sat, Mar 05, 2016 at 03:47:55AM -0800, Gábor Lehel wrote:

@nikomatsakis Does as ever currently panic in debug builds? If it doesn't, for consistency and predictability it seems preferable to keep it that way. (I think it should have, just like arithmetic, but that's a separate, and past, debate.)

True. I find that persuasive.

nikomatsakis commented 8 years ago

On Wed, Mar 09, 2016 at 02:31:05AM -0800, Eduard-Mihai Burtescu wrote:

Does LLVM not have platform intrinsics for the conversion functions?

EDIT:

AFAIK the only solution at the moment is to use the target-specific intrinsics. That's what JavaScriptCore does, at least according to someone I asked.

Why even bother panicking? AFAIK, @glaebhoerl is correct, as is supposed to truncate/extend, not check the operands.

Yes, I think I was mistaken before. as is the "unchecked truncation" operator, for better or worse, and it seems best to stay consistent with that philosophy. Using target-specific intrinsics may be a perfectly fine solution though?

sanmai-NL commented 8 years ago

@nikomatsakis: it seems the behavior hasn't been defined yet? Can you give an update about the planning regarding that?

gmorenz commented 8 years ago

Just ran into this with much smaller numbers

    let x: f64 = -1.0;
    x as u8

Results in 0, 16, etc. depending on optimizations, I was hoping it would be defined as 255 so I don't have to write x as i16 as u8.

vks commented 8 years ago

@gmorenz Did you try !0u8?

gmorenz commented 8 years ago

In context that wouldn't make sense, I was getting the f64 from a transformation on data sent over the network, with a range of [-255, 255]. I was hoping it would wrap nicely (in the exact way that <i32> as u8 wraps).

bstrie commented 7 years ago

Here's a recent LLVM proposal to "kill undef" http://lists.llvm.org/pipermail/llvm-dev/2016-October/106182.html , though I'm hardly knowledgeable enough to know whether or not this would automagically resolve this issue.

notriddle commented 7 years ago

They're replacing undef with poison, the semantics being slightly different. It's not going to make int -> float casts defined behavior.

nagisa commented 7 years ago

We probably should provide some explicit way to do a saturating cast? I’ve wanted that exact behaviour just now.

jorendorff commented 7 years ago

Seems like this should be marked I-crash, given https://github.com/rust-lang/rust/issues/10184#issuecomment-139858153 .

steveklabnik commented 7 years ago

We had a question about this in #rust-beginners today, someone ran into it in the wild.

jorendorff commented 7 years ago

The book I'm writing with @jimblandy, Programming Rust, mentions this bug.

Several kinds of casts are permitted.

Numbers may be cast from any of the built-in numeric types to any other.

(...)

However, as of this writing, casting a large floating-point value to an integer type that is too small to represent it can lead to undefined behavior. This can cause crashes even in safe Rust. It is a bug in the compiler, github.com/rust-lang/rust/issues/10184.

Our deadline for this chapter is May 19. I'd love to delete that last paragraph, but I feel like we should at least have some kind of plan here first.

Apparently current JavaScriptCore uses an interesting hack on x86. They use the CVTTSD2SI instruction, then fall back on some hairy C++ if the value is out of range. Since out-of-range values currently explode, using that instruction (with no fallback!) would be an improvement on what we have now, albeit only for one architecture.

vks commented 7 years ago

Honestly I think we should deprecate numeric casts with as and use From and TryFrom or something like the conv crate instead.

jorendorff commented 7 years ago

Maybe so, but that seems orthogonal to me.

nikomatsakis commented 7 years ago

OK, I've just re-read this whole conversation. I think there is agreement that this operation should not panic (for general consistency with as). There are two leading contenders for what the behavior ought to be:

Some sort of defined result
- Pro: This I think is maximally consistent with our general philosophy thus far.
- Con: There doesn't seem to be a truly portable way to produce any particular defined result in this case. This implies that we would be using platform-specific intrinsics with some sort of fallback for out of range values (for example, falling back to saturation, this function that @oli-obk proposed, to Java's definition, or to whatever "hairy C++" JSC uses.
- At worst, we can just insert some ifs for the "out of range" cases.
An undefined value (not undefined behavior)
- Pro: This lets us just use the platform-specific intrinsics that are available on each platform.
- Con: It's a portability hazard. In general, I feel like we've not made use of undefined results very often, at least in the language (I'm sure we do in the libs in various places).

It's not clear to me if there is a clear precedent for what the result ought to be in the first case?

nikomatsakis commented 7 years ago

After having written that out, my preference would be to maintain a deterministic result. I feel like every place that we can hold the line on determinism is a win. I am not really sure what the result ought to be though.

I like saturation because I can understand it and it seems useful, but it seems somehow incongruent with the way that u64 as u32 does truncation. So perhaps some sort of result based on truncation makes sense, which I guess is probably what @oli-obk proposed -- I don't fully understand what that code is intended to do. =)

oli-obk commented 7 years ago

My code gives the correct value for things in the range 0..2^64 and deterministic but bogus values for everything else.

floats are represented by mantissa ^ exponent, e.g. 1.0 is (2 << 52) ^ -52 and since bitshifts and exponents are the same thing in binary, we can just reverse the shift (thus the negation of the exponent and the right shift).

jorendorff commented 7 years ago

+1 for determinism.

I see two semantics that make good sense for humans, and I think we should pick whichever one is faster for values that are in range, when the compiler can't optimize away any of the computation. (When the compiler knows that a value is in range, both options give the same results, so they are equally optimize-able.)

Saturation (out-of-range values become IntType::max_value()/min_value())
Modulo (out-of-range values are treated as if by converting to a bigint first, then truncating)

The table below is meant to specify both options fully. T is any machine integer type. Tmin and Tmax are T::min_value() and T::max_value(). RTZ(v) means take the mathematical value of v and Round Toward Zero to get a mathematical integer.

`v`	`v as T` (saturation)	`v as T` (modulo)
in range (Tmin <= v <= Tmax)	RTZ(v)	RTZ(v)
negative zero	0	0
NaN	0	0
Infinity	Tmax	0
-Infinity	Tmin	0
v > Tmax	Tmax	RTZ(v) truncated to fit T
v < Tmin	Tmin	RTZ(v) truncated to fit T

The ECMAScript standard specifies operations ToInt32, ToUint32, ToInt16, ToUint16, ToInt8, ToUint8, and my intent with the "modulo" option above is to match those operations in every case.

ECMAScript also specifies ToInt8Clamp which does not match either case above: it does "round half to even" rounding on fractional values rather than "round to zero".

@oli-obk's suggestion is a third way, worth considering if it's faster to compute, for values that are in range.

jorendorff commented 7 years ago

@oli-obk What about signed integer types?

est31 commented 7 years ago

Manishearth commented 7 years ago

Throwing another proposal into the mix: Mark u128 casts to floats as unsafe and force folks to explicitly choose a way of handling it. u128 is pretty rare currently.

nagisa commented 7 years ago

@Manishearth I’d hope for similar semantics integers → floats as floats → integers. Since both are UB-ful, and we cannot make float→integer unsafe anymore, we should probably avoid making integer→float unsafe as well.

For float→integer saturating will be faster AFAICT (resulting in a sequence of and, ~~test+jump~~ float comparison and jump, all for ~~0.66 or 0.5~~ 2-3 cycles on modern arches). I personally couldn’t care less for what exact behaviour we decide on as long as the in-range values are as fast as they possibly could be.

CryZe commented 7 years ago

Wouldn't it make sense to make it behave like overflow? So in a debug build it would panic if you do a cast with undefined behaviour. Then you could have methods for specifying the casting behaviour like 1.04E+17.saturating_cast::<u8>(), unsafe { 1.04E+17.unsafe_cast::<u8>() } and potentially others.

Manishearth commented 7 years ago

Oh, I thought the issue was only for u128, and we can make that unsafe both ways.

@cryze UB should not exist even in release mode in safe code. The overflow stuff is still defined behavior.

That said, panic on debug, and on release would be great.

Gankra commented 7 years ago

This affects:

f32 -> u8, u16, u32, u64, u128, usize (-1f32 as _ for all, f32::MAX as _ for all but u128)
f32 -> i8, i16, i32, i64, i128, isize (f32::MAX as _ for all)
f64 -> all ints (f64::MAX as _ for all)

nagisa commented 7 years ago

f32::INFINITY as u128 is also UB

rust-lang / rust