Closed joshtriplett closed 2 years ago
Thanks for the summary @ChrisDenton!
As a minor correction:
However, the stable std::io::Stdout::as_raw_handle always returns a value even if it's null or invalid.
After https://github.com/rust-lang/rust/pull/93263, Stdout::as_raw_handle
does not return invalid values other than null. You'll either get a valid open handle, or null meaning stdout is not available.
For symmetry with this,
std::io::Stdout::as_handle
does likewise. This means you can't assume thatBorrowedHandle
is a valid handle. If this matters then it will need to be defensively checked before use, which is made harder by the interaction with the first bullet point (i.e. is -1 intended to mean "invalid handle" or "the current process").
-1
in a BorrowedHandle
always means the current process handle. To test a BorrowedHandle
or OwnedHandle
for validity, .is_null()
on the raw handle is sufficient. The documentation isn't currently clear on this, so I've now filed https://github.com/rust-lang/rust/pull/96932 to clarify it.
Thank you for the correction. To avoid adding confusion to this issue, I've taken the step of updating my summary. The previous version can be seen in the edit history.
In the RFC issue we discussed the possibility of having an BorrowedFd<'a>::as_filelike_view::<T>()
for where T: From<OwnedFd> + Into<OwnedFd>
, which basically constructs a FdView<'a, T>
wrapper equivalent to ManuallyDrop<T> + 'a
.
I think it would make sense to at least prototype this before stabilization so that if it makes sense to have this we can clearly document that if one implements both of these then the type can be used in a way to create temporary views.
This doesn't have to block stabilization though, to be on the safe side we could later introduce a marker for types to indicate that they support being used in a FdView
I was considering this in the context of exposing the descriptor from ReadDir
to enable openat
-like usesages. Today this wouldn't be safe to allow arbitrary views because POSIX says doing certain things with an fd while it's owned by a DIR
is undefined behavior and at least to me this became more obvious when considering arbitrary FdView
-like uses.
@the8472 I have an .as_filelike_view::<T>()
implementation in the io-lifetimes crate.
"Filelike" things are portability abstractions over "Fd" on Unix and "Handle" on Windows. IntoFilelike
and FromFilelike
are traits which are essentially Into<OwnedFd>
and From<OwnedFd>
(and their Handle counterparts on Windows) but which allow the crate to define impls for third-party types.
Yeah, that's pretty much what I had in mind. The thing is that it likely is surprising to users because to do it you have to use unsafe under the hood to turn a Borrowed*
into an Owned*
which then gets turned back into a borrow via the View
. It weakens the distinction between ownership and borrows. Which is ok for most fd-wrappers because files are essentially interior-mutable data types, but for some wrappers this isn't true.
I think we should discuss those semantics before stabilizing.
@the8472 Do you have any questions that I can answer?
My questions are more towards the libs team.
Do we want a FdView
in std?
If yes which trait bounds should it depend on? A a yet-unnamed new trait, From<OwnedFd> + Into<OwnedFd>
or From<OwnedFd> + Into<OwnedFd> + AsFd
. The choice determines for which types implementing those traits makes sense and in the latter two cases that should be documented.
@the8472 Could I ask you to sketch some wording for what the documentation might say about this, if we pick one of the latter two? I think that would help make it clear what needs to be decided here.
I'd add/change something like the following in the unix::io module description:
Types implementing
AsFd
orFrom<OwnedFd>
should not assume they have exclusive ownership of the underlying file description because the file descriptor could bedup
ed any time.Types implementing
From<OwnedFd> + Into<OwnedFd>
should not assume they have exclusive ownership of the file descriptor because theOwnedFd
may be constructed from aBorrowedFd
orRawFd
owned by another type as long as the lifetime constraints are manually upheld, e.g. by destructuring the wrapper viaInto<OwnedFd>
before the borrow ends.
The first paragraph has always been true.
The second one would somewhat weaken the meaning of Fd ownership to only mean that the file descriptor doesn't get closed under one's nose but not exclusive access. This does not necessarily fall out of existing APIs because the conversion from BorrowedFd
or RawFd
requires unsafe
and we could clarify the safety requirements of those unsafe functions that they should only be used in conjunction with known-safe wrappers. If we choose the latter then we'd need a new marker trait to let types opt into being temporary/shared-access wrappers.
My questions are more towards the libs team. Do we want a
FdView
in std?
I don't think we want one that operates on any possible object that holds a file descriptor, pretending a borrowed descriptor is an owned one. Not without some kind of unsafe operation involved.
I can imagine having ways to read and write a BorrowedFd, but not turning a BorrowedFd into an OwnedFd and passing it to arbitrary code.
reading and writing aren't always the most interesting things though. e.g. if you have a directory fd you want to pass it temporarily to cap-std's Dir
. If you have an unknown fd you might want to temporarily turn it into a File
to call metadata()
to figure out what kind of fd it is and things like that.
Anyway, if we don't want to allow it by default then we can do it through a separate trait which doesn't block stabilization.
I don't think we should allow it by default.
I think we should allow calling metadata on things other than a File.
We should talk about possible designs for directory file descriptors, but I don't think those should be a blocker for this work.
Looking into this, I noticed that the io-lifetimes crate's view mechanism is actually unsound. There's no guarantee that the Into<OwnedFd>
for a type return the same fd as the From<OwnedFd>
, so if the fd can be reassigned to a different fd, it can cause double closes of the original fd. Follow this PR for details.
So if a similar view mechanism is ever added to std, it will need an FdView
-like marker trait.
@rfcbot concern block-on-97122
I filed https://github.com/rust-lang/rust/issues/97122 to track the separate lang FCP for usage of rustc_nonnull_optimization_guaranteed
.
Current status:
BorrowedFd
, &OwnedFd
, &mut OwnedFd
, dup
, dup2
, and a possible future view API.
dup
and adds a way to do dup
on a BorrowedFd
.dup2
API@rfcbot resolved block-on-97122 (Now in FCP.)
@rfcbot resolve OwnedHandle's TryFrom impls
:bell: This is now entering its final comment period, as per the review above. :bell:
From the RFC:
if one part of a program holds a raw handle privately, other parts cannot access it
Does it make sense to model BorrowMut
instead of Borrow
to capture the above-mentioned uniqueness requirement (as this uniqueness requirement goes beyond mere interior mutability model)? Users would likely be surprised to find a uniqueness requirement stemming a borrow
.
From the RFC:
if one part of a program holds a raw handle privately, other parts cannot access it
Does it make sense to model
BorrowMut
instead ofBorrow
to capture the above-mentioned uniqueness requirement (as this uniqueness requirement goes beyond mere interior mutability model)? Users would likely be surprised to find a uniqueness requirement stemming aborrow
.
The above phrase isn't intended to be a uniqueness requirement; it's intended to be about types being able to maintain uniqueness if they want to. If safe Rust permitted things like File::from_raw_fd(23)
, it would grant access to file descriptor 23 regardless of how it's being used within the process.
The final comment period, with a disposition to merge, as per the review above, is now complete.
As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.
This will be merged soon.
What are the safety implications of Into<OwnedFd>
and/or From<OwnedFd>
and are they documented somewhere?
For example, do I understand correctly that wrappers using ioctl
for working with hardware shall never implement From<OwnedFd>
or, alternatively, shall verify that the OwnedFd they're created from actually belongs to a device of that type? Are there some other APIs or groups of APIs that should stop you from implementing From<OwnedFd>
? Are there cases where even Into<OwnedFd>
shall not be implemented?
What are the safety implications of
Into<OwnedFd>
and/orFrom<OwnedFd>
and are they documented somewhere?
There is documentation here, and more coming with https://github.com/rust-lang/rust/pull/97178.
For example, do I understand correctly that wrappers using
ioctl
for working with hardware shall never implementFrom<OwnedFd>
or, alternatively, shall verify that the OwnedFd they're created from actually belongs to a device of that type?
OwnedFd
is just about who calls close
. If you have ioctl
wrapper functions that take a file descriptor operand and leave the file descriptor open, they should not use From<OwnedFd>
. If you have a wrapper type meant to close the file descriptor when it's dropped, it can use From<OwnedFd>
.
There is no need to verify that the file descriptors have a specific type, and there's nothing special about ioctl
here.
Are there some other APIs or groups of APIs that should stop you from implementing
From<OwnedFd>
?
I'm not aware of any. From<OwnedFd>
just says (a) it accepts a file descriptor, and (b) it will call close
on that file descriptor (or transfer the responsibility to someone else).
Are there cases where even
Into<OwnedFd>
shall not be implemented?
Into<OwnedFd>
just says (a) it provides a file descriptor, and (b) whoever is receiving the file descriptor is responsible for making sure close
is called on it.
If my understanding is correct, there's one more component that you're not mentioning here: implementing From<OwnedFd>
for a type means that this type can safely work with an OwnedFd
created from any type implementing Into<OwnedFd>
.
Sure, for read
and write
that should be quite obvious except for some degenerate cases. However, what if a user safely does the following:
OwnedFd
via Into<OwnedFd>
From<OwnedFd>
ioctl
that would make sense for that device...As this would be a random RAM write, that shall not be possible without unsafe
. I don't know any OS-level protections against that, so one of the steps above should be impossible without unsafe
. My question is which one.
For read
and write
you have pretty obvious limitations on their effect: they (should) only affect the buffer in RAM you've explicitly passed to them, and write
does not modify that buffer. The problem is that you don't have such bounds for ioctl
- AFAIU, calling some ioctl
on an FD different from what you're expecting can cause immediate UB.
Ah, yes. I'm not familiar with the V4L2 APIs myself, but this sounds similar to the case with mmap
. APIs exposing such functions must be unsafe
, unless they fully encapsulate the ability to write to arbitrary memory addresses. This isn't specific to OwnedFd
, and should already be the case in Rust today.
For example, rustix::io::read
is a wrapper around the read
system call. The system call takes a raw pointer, so it could write to any address. The API encapsulates the unsafety by only allowing users to pass in a Rust slice, rather than an arbitrary pointer value, so it can be a safe API.
rustix::io::mmap
on the other hand doesn't fully encapsulate the ability to work with raw memory, so it is an unsafe API.
Both of these functions use AsFd
to accept their file descriptors. That says is "the file descriptor is not closed, and not held after the call". From<OwnedFd>
and Into<OwnedFd>
are similar; they encapsulate the "who's responsible for calling close
" and "make sure no one uses the fd after the close
" aspects of safety. However, some system calls have additional unrelated safety aspects that must be considered when deciding whether an API wrapping them must be unsafe
.
I'd say this is a bit of a grey area. You can already do extremely dangerous (including memory safety) things just by writing to things like /proc/self/mem
or (if you have root) /dev/sdx
. We usually ignore those ambient footguns because otherwise even opening files could be unsafe (after all merely opening something on a FUSE filesystem could trigger anything).
It leaves the judgment call how airtight you want your wrappers to be to the developers. E.g. for a mmap
wrapper you can either implement From<OwnedFd>
on the grounds that anyone can dup the fd via /proc/self/fd/n
anyway or you can not do that on the ground that - at least inside rust - you really want exclusive ownership (which means we can't trust safe code to not have dup
ed the fd beforehand) of the file so you can guarantee mutable-exclusive access to the memory. I lean towards the latter case, constructing the mmap wrapper should be unsafe
and the safety invariant should be that one asserts exclusive access to the file description.
Into<OwnedFd>
is more straight-forward I think. If you got still things in flight relying on the underlying file that could experience safety violations from concurrent access then it's better not to hand it out.
I think the shared file description thing is a bit of a problem in general. E.g. you could write a thing that very very carefully pokes around on the block device underlying a mounted filesystem (e.g. only fiddling around with a boot sector which isn't used at runtime) and if everything goes right your system won't blow up. But if there's another thing accessing the same file description through a duped file descriptor and only does random seeks all the time (which in itself would appear to be perfectly harmless) then this will go horribly horribly wrong.
Well, maybe this isn't the best example. In this case it's the file descriptor (to the block device) itself which is hazardous and needs to be treated as if it were one giant unsafe
block, not any of the other operations.
The mmap
scenario probably is a better one.
I think we've addressed these gray areas at this point.
/proc/self/mem
is considered to read and write to memory from the outside, as if by a debugger that is outside of Rust's control. This is the distinction effectively made by std
itself and throughout the ecosystem. It is a popular question though, so I'll submit a PR to add documentation answering it.
The only time Rust code can assume it has exclusive access to a file description is if it created a file descriptor for itself and fully encapsulated it, including not implementing AsFd
or using From<OwnedFd>
. This has always been true of raw file descriptors, and continues to be true with OwnedFd
/BorrowedFd
. https://github.com/rust-lang/rust/pull/97178 adds documentation about this.
I've now opened https://github.com/rust-lang/rust/pull/97837 to add documentation about /proc/self/mem
.
This is specific to /proc/self/mem
, it's just an easy example. There are many many accessible OS features that can subvert safety, doubly so if you're root.
Edit: I see the PR already tries to generalize beyond that
Regarding things that take AsFd
, I don't think people will defend against shared file descriptions in practice because it means that seek positions can change under their nose and being defensive about that gets really tedious and taking AsFd
is more of a tool to take either of StdinLock
or File
which on the surface do seem like you do get exclusive-under-sane-conditions access to the file.
I'd still suggest stating somewhere in the documentation that if safety of your type depends on some properties (filesystem, device driver, etc.) of the underlying file descriptor that can not hold for a reasonable FD (where "reasonable" means "not /proc/self/mem
or something like it"), then implementing From<OwnedFd>
is not appropriate and, if needed, you should make an unsafe associated function for that conversion instead.
On Unix-family platforms, std
's types don't statically require specific types of file descriptors. Even specialized types like std::fs::File
may safely hold non-file things like sockets or directories or timer fds or process fds or devices or many other things. Before OwnedFd
, people have been known to deliberately use File
in this way, as a makeshift OwnedFd
. It's safe to do this, because maybe you'll get EISDIR
, ESPIPE
, or other errors when you try to use it, but you'll never get undefined behavior.
All open file descriptors are suitable for From<OwnedFd>
. It's even valid to have a file descriptor for /proc/self/mem
or even /dev/mem
. If I call a library that accepts From<OwnedFd>
and I pass it such a file descriptor, and Undefined Behavior ensues when the library tries to do I/O with it, the bug is in my code, even though it was the library that did the I/O which caused the Undefined Behavior. That's surprising, but arguably, it's surprising that a popular general-purpose OS would have a file in the default global namespace for all processes which behaves like this.
I see several overlapping cases
/dev/mem
and the block devices underpinning mounted filesystems are such examples. Passing those around too liberally is indeed a fault of the callermmap
or reading bytes and transmuting them into repr(C)
. It's probably the library's fault for exposing From<OwnedFd>
in the first place or not checking all its preconditions properlyfdopendir
. Some unsafe
code operating on a file might be designed to be safe in the absence adversarial seek
calls. Messing with fcntl
s on a file, e.g. unlocking it, could lead to undefined behavior due to IPC mechanisms ending up in an illegal stateI'm having difficulty translating these into specific areas in the documentation that could be modified, and specific points to make. Would anyone be able to suggest wording here?
A library that does very specific things that are either harmless or correct 99.9% of the time but could lead to corruption in some unfortunate combination of file descriptor and syscall. I think that's mostly ioctls.
I'm not aware of any system calls which behave like this. Does anyone have specific examples of this?
https://lore.kernel.org/lkml/1276525077-26347-1-git-send-email-tytso@mit.edu/
The ioctl 2 has different meanings for regular files and some character device drivers. That was about a kernel bug where the kernel misinterpreted it, but they fixed that which still means that it can cause different behavior (or error out because the ioctl is not supported) depending on what file descriptor you call it on.
APIs for ioctl
s that have multiple meanings, where it's possible to get Undefined Behavior when the wrong meaning is invoked, should either use unsafe
or fstat
to ensure that they only operate on the types of descriptors they expect.
From<OwnedFd>
does not put any conditions on the types of file descriptors it can accept. To be sound, users of From<OwnedFd>
must not permit undefined behavior for any possible file descriptor type.
The 2 remaining items in the Steps / History at the top are completed:
The 2 Unresolved Questions are questions are questions for the future, and not questions that I expect need to be answered for this feature to proceed.
Thanks to @RalfJung for helping me understand this space and @joshtriplett for mentoring and reviews, and many others for discussions and comments that helped shape this.
For the next chapter in the I/O safety story: once this feature reaches stable, the next step is to begin adding trait implementations for AsFd
etc. for types in popular crates, to pave the way for a gradual migration. I've filed a planning issue here with the full plan:
Feature gate:
#![feature(io_safety)]
This is a tracking issue for RFC 3128: I/O Safety.
Raw OS handles such as
RawFd
andRawHandle
have hazards similar to raw pointers; they may be bogus or may dangle, leading to broken encapsulation boundaries and code whose behavior is impossible to bound in general.Introduce a concept of I/O safety, and introduce a new set of types and traits, led by
OwnedFd
andBorrowedFd
, to support it.Public API
The public API on UNIX platforms consists of the types
OwnedFd
andBorrowedFd
, the traitAsFd
, and implementations ofAsFd
,Into<OwnedFd>
, andFrom<OwnedFd>
for various types (such as files and sockets).The public API on Windows platforms consists of two sets of parallel types and traits and impls for
OwnedHandle
,OwnedSocket
,BorrowedHandle
,BorrowedSocket
, etc.Steps / History
Unresolved Questions