What about: Targets where NULL is a valid pointer

[RalfJung]

This recently came up in a discussion. To my knowledge, LLVM has a pretty hard-coded assumption that for address space 0, NULL is never inbounds (let alone dereferencable), so we cannot actually support such targets with the LLVM backend. But there may well be tricks I am not aware of.

[gnzlbg]

Relevant: https://stackoverflow.com/questions/27714377/does-standard-define-null-pointer-constant-to-have-all-bits-set-to-zero

C is very explicit that a null pointer must be a constant expression:

An integer constant expression with the value 0, or such an expression cast to type void*, is called a null pointer constant. [...] NULL expands to an implementation-defined null pointer constant.

This allows the compiler to replace 0 with whatever the null pointer value is for the target (e.g. there were some targets where this was the case https://stackoverflow.com/a/2597232/1422197). That is, reading zero from stdin into an integer, and casting that integer into a pointer, does not necessarily make that integer a null pointer.

[RalfJung]

That is just a matter of how NULL is represented on the machine, though? Even in C, there still is a sentinel value of a pointer that can never be valid (inbounds or dereferencable), and that sentinel value is called NULL.

---

In terms of terminology I am using here, an address is inbounds of an allocation if its offset within that allocation is no larger than the size of the allocation. It is dereferencable if the offset is less than the size. The key difference is that the first byte after the end of an allocation is inbounds, but not dereferencable. In C (and LLVM GEP inbounds), pointer arithmetic is restricted to inbounds pointers; loads/stores are further restricted to dereferencable pointers.

In LLVM, a pointer is "inbounds" even when it pointers to (within the bounds of) a dead (deallocated) allocation, that's fine for GEP inbounds. Naturally, for "dereferencable", we only consider allocations that still exist. C doesn't let you do anything with pointers to dead allocations so the question does not come up.

[gnzlbg]

That is just a matter of how NULL is represented on the machine, though?

In the C virtual machine NULL is just an address that never points to a valid object and compares unequal to any address that does. Since this address cannot be dereferenced by a valid C program, the C standard doesn't care about what happens when a program does so.

Hardware, OTOH, does not need to have an invalid address (why would it). When an OS allocates virtual memory pages for a process, starting at a virtual address 0x0, what most OSes do is mark the 0x0 address as an invalid address, for example, by protecting access to it from user space, or not mapping it to any physical memory, such that the CPU raises an exception when the address is dereferenced (e.g. on POSIX you would get a SIGSEV, Windows use SEH, etc.).

That is, on most OSes, and on most hardware, the OS and the hardware cooperate to let the users know that their C programs are broken.

What happens when the hardware does not support memory protection, virtual addresses, etc., and the OS cannot let the user know about the errors in the programs (if there is an OS at all) ? From C point-of-view, it doesn't matter, those programs are illegal, undefined behavior is undefined, etc.

The real question is what happens when the user wants to write to the 0x0 address for "reasons". On Linux, you can easily disable memory protection for the 0 address if you really need to do that by modifying /proc/sys/vm/mmap_min_addr. The C standard does not require 0x0 to be a sentinel address, it requires one such address to exist, and that address is NULL. Using C, a user could set the sentinel address to 0xffff and make use of 0x0 as a valid address. On Linux, accessing 0xffff would then still properly raise a SIGSEV and catch null pointer dereferences.

The question is, do we want to allow Rust to be used on applications that have to write to address 0x0 ? If we hardcode 0x0 as the NULL address, then this becomes impossible. If we say that such an address must exist, this address is ptr::null(), integer representations of it can be obtained by casting that address to isize/usize (and have a {usize,isize}::null_ptr() integer constants for it), and ptr.is_null() is the only way to properly check whether a pointer is null, then this would remain possible (as in, one could improve the rust toolchain to be able to do that without ending up with a language that is not Rust).

[hanna-kruppe]

Using C, a user could set the sentinel address to 0xffff and make use of 0x0 as a valid address. On Linux, accessing 0xffff would then still properly raise a SIGSEV and catch null pointer dereferences.

Though keep in mind that this requires cooperation from the C toolchain (compiler, libraries, debuggers, etc. – AFAIK neither GNU nor LLVM make any effort to support this in the least) and is an ABI breaking change, so the odds of being able to actually doing this in Linux or any other established environment are practically nil.

[gnzlbg]

Though keep in mind that this requires cooperation from the C toolchain

Yes, since the C std states that NULL is a 0 integer constant expression, the translation to 0xffff has to happen in the toolchain.

I personally think that how it would work out in C is very weird. For example, one cannot create a ptr to 0x0 by just writing char* ptr = 0; (that would create a null pointer). So the only appropriate way to crete a pointer to 0x0 would be to do so from a call to malloc that returns it or similar. Weirder is that char ptr* = 0xfff wouldn't be a NULL pointer because that's not a NULL constant expression.

---

I think Rust could do much better here using ptr::null() / ptr::null_mut() and ptr.is_null() as the only ways to create null pointers and test for null-pointer-ness.

[RalfJung]

That is, on most OSes, and on most hardware, the OS and the hardware cooperate to let the users know that their C programs are broken.

Just to state the obvious, this is far from reliable -- compilers can still exploit NULL-deref in any way that want.

LLVM will optimize

pub fn foo() {
    unsafe {
        *(0usize as *mut i32) = 42;
    }
}

to "unreachable", but will compile

pub fn foo() {
    unsafe {
        *(8usize as *mut i32) = 42;
    }
}

to a store at address 8. If we want to support targets with other representations of the NULL pointer in Rust, we first have to make some fundamental changes to LLVM.

[hanna-kruppe]

If we want to support targets with other representations of the NULL pointer in Rust, we first have to make some fundamental changes to LLVM.

I'm not so sure about that. Recent llvm-dev discussion about supporting the GCC option -fno-delete-null-pointer-checks has brought up the idea of using a non-0 address space for all pointers. This exploits the property that address 0 is only assumed to be invalid in that address space, not in others, so it's automatically correct (modulo existing bugs in handling of other address spaces).

[gnzlbg]

Just to state the obvious, this is far from reliable -- compilers can still exploit NULL-deref in any way that want.

Note that I was talking here about what happens only if machine code dereferences a null pointer independently of what optimizations Rust and LLVM might do.

we first have to make some fundamental changes to LLVM.

FWIW I think that it would be perfectly fine for Rust with the LLVM backend to not support targets in which a null pointer is not 0x0 until LLVM supports them, if ever.

What I am unsure of is whether we want to make supporting those targets impossible for a sufficiently motivated party. For example, mrustc can compile Rust to C, which could be used with a C toolchain that supports non-0x0 null pointers for some target to target such a system. Somebody that has to target such a system is going to already be in a very bad place to begin with, and that we should do everything we can to make their lives easier as long as that doesn't impacts the 99.99% of users.

I would perfectly fine with just saying that the only valid way to check whether a pointer is null is to call ptr.is_null(). And if you want to check whether an isize value corresponds to a null pointer, you would have to cast it into a pointer and then call ptr.is_null() on it. We could also guarantee that casting ptr::null()/ptr::null_mut() into an isize/usize and then casting that isize/usize back to a pointer and calling ptr.is_null() always returns true.

I think that would be a reasonable way of handling null pointers in unsafe rust for the 99%, while still allowing interested parties to target weird platforms using Rust instead of just giving up and using C instead. We also have enough tooling already in the form of clippy, rustc warnings, and miri, to be able to warn and detect null pointer checks that are not "portable" (e.g. ptr as isize == 0 => "use ptr.is_null() instead"). We could always extend this "minimal null ptr" proposal to allow ptr as isize == 0 as a valid null pointer check later on if we end up needing that.

[RalfJung]

We also have enough tooling already in the form of clippy, rustc warnings, and miri, to be able to warn and detect null pointer checks that are not "portable"

I am not sure sure about that...

But I think this discussion is very hard to have in the abstract, it'd be easier if there was a backend for Rust or even C that properly supports this and that can be looked at for how they handle this. But from what I can see, the C version of this is "add some compiler flags and hope for the best".

---

TBH, I'd find it much more sane to support targets with no sentinel value on pointers than targets where that value is non-0. That would "just" remove a whole bunch of assumptions from the optimizer, miri and maybe other places, but it wouldn't have to mess with the definition of ptr::null() (those functions should likely panic or so on such targets). And the examples we have seen -- real-mode x86 and other similarly low-level use-cases -- AFAIK have no sentinel value, so we'd not even doing them a service by supporting 0xFF as sentinel.

[gnzlbg]

But from what I can see, the C version of this is "add some compiler flags and hope for the best".

I don't think compiler flags are needed. The reliable way to create a pointer in C to a fixed address is to just not use a constant expression:

// null pointer:
char* null = 0; 
// also a null pointer:
char* null2 = NULL; 
// not a null pointer:
int value = 0;
char* not_null = (char*)value;

Here, null is a null pointer, because it was constructed from a 0 integer constant expression. null2 is also a null pointer, because NULL expands to a 0 integer constant expression. But not_null is just a pointer to address 0x0.

[RalfJung]

I understand what the standard says about this, but given that the IRs on which compilers do their optimizations are so loosely specified, I am doubtful that this distinction can even be meaningfully made there -- and that all optimizations follow it correctly.

As far as LLVM is concerned, I very strongly doubt it makes any difference between your examples. Godbolt agrees.

[alercah]

I believe that the correct reading of the C standard is that one or more bit representations of pointer types are considered to be null pointer values, and imposes restrictions on them accordingly. The exact representation is implementation-defined or unspecified (the language in the standard is a tad fuzzy on this point, but I don't think it matters). So on a platform where the null pointer's representation is indeed all-zeroes, not_null being a null pointer is the correct behaviour (and reliable, if the implementation specifies that representation).

Everything Rust does currently is, I think, theoretically compatible with a platform where it is not 0; the only questions are:

1. Whether we want to consider the potential of supporting platforms with non-zero null at some point.
2. Whether we want to force everyone to work their code around that assumption (one way might be by having CTFE use other null values, for instance, so that we can trap on people who break this)

[parched]

TBH, I'd find it much more sane to support targets with no sentinel value on pointers than targets where that value is non-0.

I tend to agree and I think this is basically what GCC does with -fno-delete-null-pointer-checks.

[parched]

BTW, not sure how I missed this before, but LLVM supports this since version 7 https://reviews.llvm.org/rL336613, so it would be quite simple to add a similar null-pointer-is-valid field to the target spec and pass this through. The big part would be conditionalizing all null assumptions/optimisations in rustc.

If there are any non-zero null targets, they could just use null-pointer-is-valid too, it would just be a pessimization.

[skade]

Here's two code examples to boil this down to. Are the following three examples guaranteed to be true, currently?

let p: *const () = 0_usize as _;
assert!(p.is_null());
assert!(p == std::ptr::null());

Also, is this guaranteed to work?

assert_eq!(x, 0_usize);
let x: Option<&T> = unsafe { transmute(x) };
assert!(x.is_none());

Seeing a lot of the documentation trying to always refer to null instead of zero and a lot of API built around the hide the knowledge that the null pointer is defined as 0 in the ptr module makes me think that there was indeed some amount of thinking that went into the issue at hand.

[gnzlbg]

Seeing a lot of the documentation trying to always refer to null instead of zero

@skade The API docs of NonNull<T> explicitly say:

*mut T but non-zero and covariant.

(emphasis mine)

I'm not sure if this is by design or whether that's an oversight that should say non-null instead. That wording was introduced here.

AFAICT, the reference doesn't say what NULL is, but @gankro's post says:

For Rust to support a platform at all, its standard C dialect must: [...] Have NULL be 0

[Lokathor]

In current rust null must be zero for NonNull to work. Internally it declares the base value of the type to be 1 instead of 0, and that's all it does. So the niche to get the null-pointer Option optimization must be 0. Then, for Option<&T> and Option<&mut T> to be bitwise compatible they also must use zero as the null value.

It might be possible to change that in the future, but it would be a relatively massive change that would require rechecking huge amounts of code to fix up places where people's old assumptions need to be updated.

[Lokathor]

Also apparently the C spec is a wacky "the null pointer must always compare equal to the integer expression 0, and only equal to the pointer expression NULL, but doesn't have to literally be the all-zero bit pattern."

[skade]

@Lokathor this is all clear, but the question is if that's just the current implementation or if people can rely on it. A lot of the things documented here somehow work, but are actually undefined.

AFAICT, the reference doesn't say what NULL is, but @gankro's post says:

For Rust to support a platform at all, its standard C dialect must: [...] Have NULL be 0

This is circular, as this discussion has sparked from precisely that statement. ;)

I mean, the option to make that a language guarantee is fine, but I can't find any source for this.

[elichai]

I don't think relying on it is a good idea. As @Lokathor said, C doesn't require NULL to literally be 0.

[Lokathor]

@skade Currently it is in what most would call the "implementation defined" area.

Null is absolutely and definitely 0, no worries of UB. However, a new release of the compiler could potentially change that (with sufficient work done), so I would not call it an eternal guarantee.

[gnzlbg]

Null is absolutely and definitely 0

Citation needed. Many comments in this thread argue otherwise.

[Lokathor]

Sorry, in rust null cannot be any value other than 0.

In C it can be anything it chooses, and if the local C picks a non-zero value then Rust will just not have the same null.

[comex]

For what it's worth, I don't know of any C environments where NULL is not 0 that are not obsolete architectures from the 70's. There are environments where 0 is a valid pointer at the assembly level, and the contents of memory there are important: for example, on old ARM systems, the CPU starts executing code at address 0 when it's powered on. But 0 simultaneously serves as a null pointer for C programs written in those environments. This works because there's typically no need to access that address from C anyway.

[alercah]

The only thing approaching an example that I am aware of is pointers to member functions of virtual classes in C++ under Microsoft's ABI. But this is not C nor it is it a plain pointer.

[gnzlbg]

There are environments where 0 is a valid pointer at the assembly level,

That's the only use case I know as well. Maybe we should try to collect more use cases? For a restricted set of use cases, the best solution might be very different from "allowing NULL to be an implementation-defined address". For example, for putting code at address 0, we could maybe support an option on global_asm!. If for whatever reason users need to reuse that address, we could add intrinsics that allow reading / writing data from this address without invoking undefined behavior. Etc.

[Lokathor]

If people are wanting to put code at 0, that sounds like linker script work.

for reading and writing 0, i believe that inline asm can already do this?

[skade]

Null is absolutely and definitely 0, no worries of UB. However, a new release of the compiler could potentially change that (with sufficient work done), so I would not call it an eternal guarantee.

Just to be clear, I wrote "undefined", not "undefined behaviour". I'm perfectly aware that this does not cause UB, but that still means that the result of 0 == ptr::null() is undefined by the language, but yes, I should have said "implementation defined".

I'm not opposing or even arguing against making NULL 0. I think the discussion revolves too much around finding cases or C semantics of fringe platforms.

I agree with @comex that if no desirable platform can be found where this is the case, it's reasonable to assume it and write that down. It's a pragmatic solution. I think the question of 0x0 being accessible is more the question whether dereferencing 0x0 is undefined behaviour.

---

Thinking about it further, I think Option<&T> has to be even considered a separate case. It's similar to a nullable pointer, but - correct me if I'm wrong - no one should rely on the interface anyways. IMHO, such a type should not be produced by transmuting, but instead through API. Rust does not guarantee the layout of Option (or any structure that is not #[repr(C)].

[Ixrec]

Rust does not guarantee the layout of Option (or any structure that is not #[repr(C)].

I thought the "null pointer optimization" has been a hard guarantee since at least Rust 1.0. Or is that an FFI-only thing, and it's theoretically permitted for Rust to layout Option<&T> differently and do runtime conversions on FFI calls?

(the Nomicon describes this optimization, and the Reference doesn't appear to mention it, but iiuc neither is normative; although we don't seem to have anything that is normative, so I'm not sure what counts as evidence of anything in these discussions)

Though I agree that "is Option<&T>::None null?" is at least logically a separate question from whether null is 0 or legal to dereference or access.

[skade]

@Ixrec To my reading, it's safe to assume that rustc > 1.0 performs this optimisation. The Nomicon is a practical document though, not a spec.

But when taking the Nomicon as a reference, it also clearly says:

Transmuting between non-repr(C) types is UB

And Option is not repr(C). (https://doc.rust-lang.org/nomicon/transmutes.html)

On the other hand, also:

If T is an FFI-safe non-nullable pointer type, Option is guaranteed to have the same layout and ABI as T and is therefore also FFI-safe. As of this writing, this covers &, &mut, and function pointers, all of which can never be null.

as of writing.

[hanna-kruppe]

Note that this statement:

Transmuting between non-repr(C) types is UB

is definitely an incorrect over-simplification in many respects unrelated to the Option<&T> question (e.g. it ignores repr(transparent) and repr(Int) on enums) so I would not take it literally even if we were to treat the nomicon as more normative than it is.

[RalfJung]

@skade see https://rust-lang.github.io/unsafe-code-guidelines/layout/enums.html for our current thinking on enum layout guarantees. in particular this section.

[deliciouslytyped]

disclaimer: I have no deep knowledge of compilers and I think C is irrelevant to what rusts semantics here should be, past being something to inform things.

My two cents are that it would be a shame if "arbitrary" restrictions precluded using rust in obscure low level use cases which violate common conventions (that are not technical necessities). Or is that out of scope for rust?

I'd have to do a deeper review - so I don't know exactly what happened - , but I believe Christopher Domas made 0x0 a valid memory address as part of the techniques he developed for Sandsifter ( https://github.com/xoreaxeaxeax/sandsifter ) and the accompanying research; https://www.blackhat.com/docs/us-17/thursday/us-17-Domas-Breaking-The-x86-Instruction-Set-wp.pdf , it would be nice if things like this were possible in rust.

Specifically, the process state is corrupted if a generated instruction writes into the
fuzzer’s address  space. This is overcome by initializing all registers to 0 and mapping
the NULL pointer into the fuzzing process’s memory. 

[Lokathor]

You could still use assembly to read and write 0 in the odd case where it's important to be able to.

[deliciouslytyped]

To clarify a little, what I'm trying to say is should specific memory addresses really be treated as special by necessity?

[Lokathor]

It's a small optimization but it comes up so often that it's worth it

[devsnek]

Hopefully not repeating anything, but in WASM, there are no restrictions/special behaviour around address 0. Additionally, WASM uses a Harvard architecture, so there will be a function address (or more accurately, index) 0. I believe this is handled in llvm by emitting a dummy function at the 0th index function, but it is still worth mentioning.

[enbyted]

Just passing by with a real-world system with valid address 0 that actually has a reason to be written to. Microcontrollers in STM32H75x/H74x (ARM Cortex-M7 CPU) series have quite an interesting memory map with 8 different RAM memories. One of them is called ITCM, it spans 64kB from address 0 to 0xFFFF. This memory is dedicated to instructions as it's the only memory that can be as fast as the CPU (for reading instructions). As it is RAM memory it needs to be initialized. Not only that but in some applications I actually want to override all/parts of it when entering certain program states.

Fortunately this is memory region targeted towards instruction and not data, so pointers to that region are quite rare.

[phil-levis]

@enbyted Another real-world system that has a reason to write to address 0x0: bare metal code on ARMv6. Or more generally, bare metal code which doesn't use virtual memory on ARM systems that don't have a VTOR or VBAR register to specify the location of the interrupt vector table. The vector table is either at address 0x0 or 0xFFFF0000; if you have virtual memory it's easy to map 0xFFFF0000 to the page where the table is, but if you are operating with only physical memory you need to put the table at 0x0.

One example of where/when you do this: bare metal code on the Raspberry Pi (1). The Pi places the kernel image at 0x8000, and your code needs to copy the interrupt table to 0x0. This is what we have students do in a class I teach.

[chorman0773]

I actually designed what I think to be the least problematic solution. In many cases, address 0, when accessible, has more meaning than just "a place you can store something" (though not always). As this is generally going to be limited to embedded/freestanding, here (which is in the stage where I am ready to bring it as a full RFC, but I'm still figuring out the wording), I proposed that the volatile access functions conditionally-support accessing the null pointer, and that the results are implementation-defined. In hosted and wherever it's not actually valid, rustc (and likely anything else) would document that it's unsupported (thus leaving it UB in these cases), and when it is valid, it would likewise document the behaviour as accessing whatever happens to exist at that address. In all cases, it can be done w/o llvm helpfully removing your undefined code paths (no_delete_null_pointer_checks), and the question would be if it can be done w/o trapping and w/o accessing another allocated object (though anything allocated by a rust compiler at address 0 would be rather useless, as explored in the above Pre-RFC).

Whether or not rustc supports it would be on a target-by-target basis.

For completeness, conditionally-supported means that a particular implementation may support it, and documents when it does not, and implementation-defined means the implementation chooses how to (it is a parameterized part of the abstract machine), and documents how it makes the particular choice.

[RalfJung]

In terms of NULL accesses, it certainly makes sense to treat them the same as OOB accesses (Cc https://github.com/rust-lang/unsafe-code-guidelines/issues/2). This also recently came up on Zulip, and people were in favor of permitting volatile accesses (and only volatile accesses) to do that -- but it is unclear if that will be possible with LLVM.

[chorman0773]

As mentioned (and I believe you brought up originally on the Pre-RFC), there is an llvm attribute for this (though it is null_pointer_is_valid rather than the no_delete_null_pointer_checks which I think I got from the compiler flag). On these targets, the volatile intrinsics could emit code that has this flag, like this (in pseudo-rust, because I don't really want to write llir by hand):

#[llvm_attribute(null_pointer_is_valid)]
pub fn read_volatile<T>(ptr: *const T) -> T{
    <load volatile>(ptr)
}

where llvm_attribute is a placeholder for emitting a particular llvm function attribute. In fact, the definition is core::ptr could be conditionally replaced with that code, calling the unstable core::intrinsics verison, and rustc could introduce an unstable attribute which allows directly emiting llir attributes.

[RalfJung]

Ah right, LLVM has that for NULL specifically... maybe they would be open to adding another attribute that removes all inbounds assumptions (not just NULL).

In fact, the definition is core::ptr could be conditionally replaced with that code, calling the unstable core::intrinsics verison, and rustc could introduce an unstable attribute which allows directly emiting llir attributes.

Or we could just adjust how the intrinsic is codegen'd -- that seems simpler?

[chorman0773]

That would be the same yes. The difference is that either, the validity of null leaks to the surrounding code (so now non-volatile accesses wouldn't be optimized for null pointers), or that the intrinsic would have to emit (or be emitted as) a trampoline. This can obviously be done conditionally, chosing the most optimal one, or letting llvm decide if it wants to inline and extend the scope of the attribute.

[RalfJung]

Oh, this is a per-function attribute in LLVM?

One more reason to propose a change -- something that works per-access seems more useful here.

[chorman0773]

Yes, it's per function as far as I know (this was brought up in the Pre-RFC).

[Serentty]

On the Commander X16 (whose 65C02 CPU is currently not targeted by upstream Rust, but can still be targeted fairly easily by adding a custom target), addresses 0 and 1 control banking for RAM and ROM, and since the RAM window in the address space is fairly small, you would expect to change the bank a lot even from Rust code.

Avoiding inline assembly for this would be useful because it makes sure that the bank number can be an immediate, and opens it up to optimizations.

[bjorn3]

I don't think you can avoid inline asm. You need something with sideeffects such that the compiler doesn't assume that no banks have been switched. A regular or volatile store is not enough even if the address wouldn't be zero. You need something that can have arbitrary sideeffects in the eye of the optimizer, including reading and writing all memory that is bank switched. That would either be a regular function defined in a different codegen unit so the optimizer can't peek inside it to calculate which sideeffects it has (#[inline(never)] doesn't prevent this) or assembly, be it inline or externally.

[RalfJung]

Also, all mutable and shared references would be invalidated across such a bank switch.

[Serentty]

Also, all mutable and shared references would be invalidated across such a bank switch.

I don't think all would be, necessarily—just those pointing to the high RAM area which is banked. You could use raw pointers for that region of memory, or a custom pointer type which switches the banks when dereferencing, while reserving normal references for low RAM. But yes, better support for non-flat address spaces would be nice to see in Rust. But perhaps that's getting too off-topic from writing to address zero being useful.