CAD97
commented
4 months ago This is a bit of an essay, sorry. I marked the most important takeaway, with the TL;DR being that I'm in support of the OP example being defined behavior.
There is a weird semantic space where mutating a non-mut let binding would live — behavior considered defined by the compiler, but still heavily discouraged, borderline still disallowed in any reasonable code.
To @workingjubilee's point, as I understand it:
let place: i32 = 0;
macro_from_upstream_library!(place);
assert_eq!(place, 0);Answering the issue title's question as no and declaring the OP's example as defined behavior would mean that this assertion could panic in valid executions. However, if it did, the macro would be unsound. Macros are absolutely allowed to expand to internally encapsulated unsafe, but the unsoundness of the macro which makes this assertion panic isn't even a difficult to spot technicality; the counterexample is as simple as just writing macro_from_upstream_library!(*&place).
When reasoning about what other code (outside the safety boundary, whatever you've locally defined that to be) can do, the question isn't about language validity and what is or isn't UB, it's about safety invariants, which are much stricter than simple validity.
(Aside: I think I want to write (another) blog post on the idea of and separation between the concepts of validity and safety. It comes up and I've re-explained it frequently enough that I'd like to have a stable reference that I can point to that covers just that difference specifically.)
I am treating the author of the macro as potentially adversarial to the author of the code.
There is some level of cooperation which you have to assume, and that cooperation is that upstream is sound. If you can't assume that, upstream could just unreachable_unchecked and eat your laundry for free. Or just write the mutation through addr_of! anyway despite it being invalid instead of just unsound.
The correct model that Rust helps with is explicitly not an adversarial upstream. An ignorant and/or negligent upstream, sure, but upstream is always presumed to be competent, cognizant, and cooperative enough to maintain the soundness of the interface.
In fact, allowing mutation helps with the case of negligent upstream, because now if they use addr_of! instead of addr_of_mut! incorrectly when the value could potentially be written to, the result is merely surprising behavior instead of deleting your code (or worse, seeming like it works). In the specific case where it gets run under Miri, the perfect result might be to point at the errant write. But since pointers in public API are involved, it's much more likely that we can't use Miri because there's FFI involved, and in that case I'd much prefer that my debugger works and shows me that the value changed, instead of presenting some arbitrarily wrong picture of the world that's been corrupted by UB.
That is the primary issue with UB — you can't reliably debug it when it happens. The main problem with how we're understanding your position is that you're apparently claiming that UB is preferable to a defined but semantically wrong result. This is a very confusing position to hold, because while you can debug a wrong result, you fundamentally can't debug a UB.
Perhaps the convenience of Miri as a sanitizer works against us here. UB is not equivalent to diagnosed by Miri. UB should always be thought of as perhaps the exact opposite — daemonic nondeterminism that chooses between time traveling and changing previous control flow choices, the most sensitive data accessible by the machine, the "correct" result, or a crash, whichever is worst for you at the moment, non-exhaustively.
"Forbid it, diagnose it as forbidden whenever possible, but as a mitigation permit it when it can't be prevented" is a defensible position. "Eat my laundry if I dare make a mistake" less so.
To the second form of the point mentioned both by Jubilee and Calvin, between
lib_call(&place); // or
lib_call(addr_of!(place)); // where
// extern fn lib_call(_: *const Type);I would still recommend using &place and reference-to-pointer decay in most cases, and in all cases place is an initialized let without mut. And if you want to be explicit that conversion to pointer is happening, ptr::from_ref(&place). The entire point of choosing to "always prefer addr_of!" when creating a pointer is that addr_of! has weaker semantics and permits more operations as valid; if the semantics (and optimization) you want are that of a reference, use a reference.
If a developer chooses to use addr_of!(place) and not ptr::from_ref(&place), they're explicitly requesting weaker semantics that don't include freezing place from potential mutation. Entirely ignoring that signal because the place was declared with let — potentially because the compiler told them the mut was unnecessary — seems needlessly adversarial of the compiler.
In fairness, I absolutely could see Miri in a conservative mode (e.g. under cargo-careful) choosing to warn when let-bound places are mutated. It's the same reasoning I gave behind considering shallow memory initialization requirements for reference retagging — it's "nicer" to "blame" the cause rather than the symptom.
If we had a formal classification for "forbidden but still with predictably defined behavior of the 'obvious' semantics or a crash," it could make sense to place this there. Exposed provenance is mostly in that bucket, I believe. But it isn't necessary to formally have that bucket in order to diagnose it. Miri isn't UBSAN which (in theory) accepts all valid programs and rejects most invalid; Miri (in theory) rejects all invalid programs and accepts a majority of valid ones.
This might be "erroneous behavior." It might be "safe misbehavior." Maybe we should try to build such a catorization as we stabilize the strict provenance APIs and start moving towards supporting CHERI as an experimental target. But making it UB will never make Jubilee's million-line maze easier to manage, and can only make it more difficult to debug.
I will even agree that any code written that deliberately relies on this being DB should be immediately fixed with priority equal to that of latent UB. Code which relies on the "wide" (or "infectious") model for shared mutability instead of wrapping things in UnsafeCell, which relies on the !Unpin hack instead of using UnsafePinned, or on moving containers without MaybeDangling not invalidating interior references is all in the same boat — should be fixed to not rely on these hidden details of the Rust AM as soon as possible.
None of these things are DB because we endorse doing them. They're DB because all else being equal, we prefer more things being DB. Just because you can doesn't mean you should, and we consistently make the nicer and more straightforward way to accomplish these things available at the same time as or before we canonicalize that the circuitous way is actually guaranteed to work instead of just working by happenstance.
To @Calvin304's primary point about initialization tracking:
I don't know at what point uninitialized bindings are currently diagnosed. However, with the shifts to use THIR unsafeck and MIR borrowck, the trend is to push these analyses later in the compilation pipeline.
MIR does actually know a difference between let and let mut; you can see this when dumping MIR built from surface Rust. But on the other hand, parameters in dumped MIR aren't ever marked mut, and the mir! macro just uses let and doesn't even parse let mut.
I intuitively expect that if/when the language is extended to support piecewise initialization of let, this analysis will happen on THIR; it must happen after types are known and name resolution has occurred in order to ensure no deref coercion happens on a partially uninitialized place, independently of whether we allow piecewise initialization to completely initialize a place.
Thus, once that analysis is done, it basically just becomes an implementation choice whether that information is validated in THIR then discarded or if it's passed along into MIR. We already have StorageLive and StorageDead, so a "StorageFreeze" for non-mut initialization of let is far from unreasonable.
The benefits of immutable-after-init let do exist (and aren't novel, since C++ compilers exploit them for const locals), but it's worth explicitly noting that these benefits would still apply to most let even if we say let can still validly be mutated. A non-mut place can only be mutated if it is used by a raw reference-of expression (addr_of! / &raw const), so it can still be optimized in the absence of that.
For such to exist in the TB model, the "root" tag for let allocation is still "Active", but any place mention (excluding direct place = assignment) doesn't use that tag directly, but instead a child tag, which is frozen for any non-mut bindings.
So, with that in mind, reducing the question of the OP to absurdity, it really does boil down to:
If, given some let place, a developer chooses to write addr_of!(place) instead of the stricter ptr::from_ref(&place) (maybe using coercion), does it still make sense to punish a mistaken write with arbitrarily disastrous UB?
My personal position and answer to the question is the following three-phase conditional:
- If we ever allow
addr_of!(place)for an uninitialized place, then we should permit writing through a such-derived pointer for the same reason we allow=assignment to an uninitialized place. - If we never allow
addr_of!(place)for an uninitialized place (thus requiring the use ofMaybeUninitforunsafepiecewise value initialization behind a pointer), then freezingletplaces is justifiable. - However,
addr_of!(place)is a request for weaker semantics than&place, so it makes more sense to respect that, especially since any and all optimization benefits can be exactly recovered by using&placeinstead, without any[^6] subtle pointer validity differences.
[^6]: The strongest argument for not using &place would be that the resulting pointer's pseudo-lifetime outlives the enforced borrow lifetime. (However, if the execution is valid, note that no lifetime extension occurs; the borrow lifetime and validity always die together for a shared borrow of a let binding… in the absence of any mutation as allowed by this, anyway.) Any project pedantic enough to care about this should probably be banning reference to pointer coercion and appreciate the syntactic noise (and semantic signal) of using ptr::from_ref.
The points/arguments I make elsewhere throughout this comment are why I agree with Ralf here that the example in the OP should be defined behavior. The most relevant point is that addr_of! is a non-mut place mention that nonetheless doesn't freeze the mentioned place (like a non-raw reference-of would), which I expand on a bit more below.
Separately, I also think an initial intuition can reasonably expect a binding's place mention to behave like *&mut binding (if a mut mention) or *&binding (if a non-mut mention). This doesn't work because of place projection (partial borrows), but up until that point integrates well with how auto(de)ref and deref/index place mutability inference work[^5]. If a developer understands that the purpose of addr_of! is to avoid creating a reference, that same understanding covers why addr_of! shouldn't freeze the mentioned place.
[^5]: Example: the place v[i] is explained as sugar for *Index::index(&v, i) (or the mut version), which after mental inlining turns into essentially *&*<_>::as_ptr(&v).add(i). Sugared place mentions have *& semantics, so it's understandable to expect something similar when mentioning a let binding as a place.
While I absolutely agree with the sentiment of #257[^0], I do wonder whether the perception of addr_of!(place) and &raw const place might differ, since the latter has const in the syntax. Additionally, addr_of! is enough syntactic salt that it's IMHO a strong signal for "I want the weaker semantics," but &raw const place is so convenient that it loses some of that signal[^4] and is easier than writing ptr::from_ref(&place) even when the stronger semantics are wanted.
[^4]: Similar to using raw pointers even when they're known non-null. If the hypothetical "One True Pointer Type" doesn't distinguish "being" mut, then the OTPT version of &raw mut would exclusively determine whether the place mention is mut, making it easier to argue for read-only (i.e. if you want write permissions, use the mut syntax, which despite applying to raw pointers because of coercion, is a bit of a weaker argument when the types differ).
[^0]: It remains extremely unfortunate that (potentially coerced) &mut place as *const _ produces a pointer that's illegal to write through imo, especially since there's no "unused mut" style lint emitted to say it could just be &place instead. The coercion "desugars" into &raw const (*&mut place), so IIUC SB errors and TB allows mutation, as SB uses a "raw" tag kind for &raw const but TB just uses the source's tag.
Drawing from a Rust background, I[^1] don't expect to be able to modify through an addr_of!(binding) pointer[^2], personally. Drawing from a C++ background, on the other hand, my understanding of when it's valid to use const_cast can suggest that addr_of!(binding).cast_mut().write(…) could be valid. But on the other side of that hand, const in C++ applied to the type of the storage place does actually make the place UB to mutate[^3].
[^1]: This is about me personally, not a theoretical I. And probably I'm far from a typical user w.r.t. what I intuitively expect, given my involvement with T-opsem and that Rust's ownership model immediately made intuitive sense to me, allowing me to mostly skip the "fighting the borrow checker" phase when I was first learning Rust.
[^2]: Although, for full clarity, my "first guess" semantic also doesn't have addr_of! freeze the location, either; it's still weaker than ptr::from_ref(&place) and the place could be mutated through a separate pointer to the place. Additionally, for addr_of!(*mut_ptr), my "first guess" heuristic is entirely unsure whether to treat this differently from mut_ptr.cast_const().
[^3]: In the absence of mutable fields, essentially the C++ version of Rust's UnsafeCell.
So in (not particularly useful) C++ terms, the question is whether let place = init; in Rust has C++ semantics more like auto place = init; auto const&& place = (auto&&)place; (actually mutable) or more like auto const place = init; (actually immutable).
Perhaps more usefully (despite being less strictly correspondent), it depends on whether the C++ developer learning Rust sees let place: T and let mut place: T as the moral equivalents of const T place and T place (actually immutable) or of T place and mutable T place (actually mutable). That mut isn't permitted in struct fields should hopefully indicate that Rust mut is closer to the absence of C++ const than the presence of C++ mutable.
As one last wrinkle, though, I've only really considered addr_of!(binding) above, as that's what's relevant to the OP topic. Indirectly relevant, however, is computed places, e.g. addr_of!((*mut_ptr).field) or even addr_of!(*impl_deref). That gets unambiguously into #257 territory, so I won't touch on it further except to say that addr_of! regularly producing valid-for-writes pointers despite being treated (e.g. for borrowck and deref/index mutability inference) as a non-mut place mention makes the concept of addr_of!(let_binding) being valid-for-writes a lot easier to conceptualize. Under this model, mut is truly "just a lint" and only controls whether a mut place mention is allowed, and since addr_of! is a non-mut place mention that doesn't freeze (make immutable) the mentioned place, the writing through the pointer is still permitted unless a different operation has frozen the place.
IMPORTANT TAKEAWAY BEGIN
So the question finally becomes, after defining addr_of!: does assignment to a let binding without mut freeze the assigned-to place (until the place is dropped)? I fail to see any reason for it to (the locality of reasoning for the developer is maintained by safety, not by validity) and a very clear reason for it not to (an errant write still being debuggable behavior) so my vote's that the OP example should be DB.
If you want errors to be detectable, you want them to not be UB. Being DB does not mean you as a developer have to tolerate it; it means the compiler has to tolerate it. Being UB doesn't mean a rushed developer is any less prone to making mistakes; it means that determining that and how they did is miles more difficult.
IMPORTANT TAKEAWAY END
Post-script response to @chorman0773's point: (it took >6h to write this 🙃)
It's already fundamentally the case throughout Rust's semantics that simple inorder translation isn't sufficient. I don't think making one small part of the translation easier should inform any surface language visible decisions. E.g. type inference is a full-function process, so I don't see how capture elision could happen without having done full-function analysis already. Also, it seems like your eager SSA form lowering
Modifying addr_of! to get "immutable after initialization" requires one of two things — either addr_of! always generates a pointer without write permission (what SB does), or addr_of! needs to behave differently based on whether it's referencing a non-mut binding (or pure place projection thereof) or some other place (i.e. a computed place/value or a place that names a (projection from) mut binding). IMHO that seems much worse than the options of either:
- Making non-moving binding place mention use a non-root borrow tag, which gets tagged
Frozenforletbindings (ofFreezetypes); or - Making place assignments behave differently if a place isn't previously initialized, which is already the case because whether the old value is dropped depends on initialization.
The "proper" solution is for lccc to have an xlang optimization pass that replaces alloca with SSA form where possible and desirable. Given that needs to exist anyway if you do any optimization on the xlang form, it seems like the benefit of deferring alloca when lowering MIR to xlang is quite minimal. Especially with how eagerly surface Rust creates references.
The only real potential benefit I see is in capture elision, turning by-reference capture of let bindings into by-value (and by-constant-value) capture … except since the capture is by-ref, the by-ref capture freezes the place, so mutation of the place is excluded by the capture semantics, and doesn't need to be excluded by the place semantics.
Final afterthought: the above prompted me to think about how closure capture is dependent on mut annotation. This is just another case of addr_of! being a non-mut place mention, but capture rules result in that even if addr_of!(x) gets write-capable provenance, even if x is bound with mut, (|| addr_of!(x))() does not, because that's semantically equivalent to ({ let ref_x = &x; move || addr_of!(*ref_x))().
I don't think there's a fair alternative which still captures a borrow lifetime. That we do precise field capture for closures limits the impact (i.e. capturing for addr_of!(place.field) won't ever impact pointers to the rest of place), but it does feel kind of footgun-y. (FWIW, a pure place mention (e.g. _ = x) doesn't capture x at all.)
chorman0773
workingjubilee
Monadic-Cat
zachs18
This came up in https://github.com/rust-lang/rust/issues/111502 and overlaps with https://github.com/rust-lang/unsafe-code-guidelines/issues/257: should it be allowed to mutate
let-bound variables, via code like this?Tree Borrows currently accepts this code since
addr_ofnever generates a new tag, it just creates an alias to the original pointer. Stacked Borrows rejects it due to #257. There are other ways to reject this code while making*constand*mutequivalent, hence I opened this as a separate issue.Personally I favor the Tree Borrows behavior here and don't think it is worth the extra effort to make these variables read-only.