Open ijackson opened 3 years ago
I would like to propose that we stabilize these. Based on the feedback above, these are very useful to have in stable, and there seem to be no real risks. Concretely, this would mean marking the following as stable:
io::ErrorKind::HostUnreachable
io::ErrorKind::NetworkUnreachable
io::ErrorKind::NetworkDown
io::ErrorKind::NotADirectory
io::ErrorKind::IsADirectory
io::ErrorKind::DirectoryNotEmpty
io::ErrorKind::ReadOnlyFilesystem
io::ErrorKind::FilesystemLoop
io::ErrorKind::StaleNetworkFileHandle
io::ErrorKind::StorageFull
io::ErrorKind::NotSeekable
io::ErrorKind::FilesystemQuotaExceeded
io::ErrorKind::FileTooLarge
io::ErrorKind::ResourceBusy
io::ErrorKind::ExecutableFileBusy
io::ErrorKind::Deadlock
io::ErrorKind::CrossesDevices
io::ErrorKind::TooManyLinks
io::ErrorKind::InvalidFilename
io::ErrorKind::ArgumentListTooLong
The issues around io::ErrorKind::Other
and the unexpected failures they caused have been resolved with the introduction of io::ErrorKind::Uncategorized
. This also eliminates the potential breakage from exhaustive matching. As such, stabilization itself should not be a breaking change anymore.
As to the concern raised by @m-ou-se, I have gone over all of these errors, and almost all of them seem to have compatible OS error conditions on both windows and any posix compliant system. As such, platform specific issues should only occur on unusual platforms outside of either the current tier 1 or tier 2 lists. I think we should be OK treating any cases where these errors are not generated currently as bugs. A further search of the current open issue list suggests no such issues are currently known.
There are two exceptions:
First, io::ErrorKind::FilesystemQuotaExceeded
and io::ErrorKind::FileTooLarge
do not seem to correspond to posix standardised errors. I think for these in particular it is acceptable if they fall back to io::ErrorKind::StorageFull
, which seems to be what the posix standard suggest would happen under those conditions.
Second, io::ErrorKind::ExecutableFileBusy
and io::ErrorKind::Deadlock
seem to be best effort, and have limited system support, but this seems to me well enough documented.
io::ErrorKind::FileTooLarge
corresponds to EFBIG
, which is in POSIX, see here. This error is distinct from io::ErrorKind::StorageFull
or ENOSPC
.
As this problem is still around, I wonder if it would be a good idea to think about how to handle such changes.
Concretely, this is NOT a breaking change but a functional change, which is as important but not covered the same by Rust's guarantees.
The problems I see here are that:
unsafe
or nightly feature, but one that has been available for a long time.std
and core
could be affected by this problem. This makes writing reliable and future-proof code a lot harder and require to read every single changelog when upgrading to a newer Rust compiler.I don't know what we could about these, but I think clarifying at least Rust's guarantees for functional changes could help (maybe it's already specified somewhere but I didn't find it). And officially endorsing that such breaking changes will ALWAYS be highlighted in Rust's release notes (like it has been for the 1.55's blog post).
Sorry, I don't follow how this wasn't a breaking change. It was acknowledged as such much earlier on in the discussion. Code that did compile stopped compiling.
Sorry, I don't follow how this wasn't a breaking change. It was acknowledged as such much earlier on in the discussion. Code that did compile stopped compiling.
Sorry it wasn't very clear in my message, I mean that it didn't change nor remove any entity (variable, function, trait, module) or their signature.
This is a functional change in the sense that the ErrorKind
variants returned by the IO functions aren't the same as before, no variant was removed or renamed, it's just that these functions don't return the same values anymore.
I think, at this point, we would provide the best experience by stabilizing these, so that we don't return errors people can't match.
@rfcbot merge
Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:
No concerns currently listed.
Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!
See this document for info about what commands tagged team members can give me.
I think, at this point, we would provide the best experience by stabilizing these
I very much agree that we should stabilise these.
Having said that, I can understand why it has taken a while.
The conversations in this area have been very long and detailed. The change from Other
to Unrecognized
, which has already been done, and is not part of what we're proposing to stabilise, has (perhaps inevitablh) become entangled in the conversations (especially since it was prompted by the proposal to add new errorkinds).
Also, this is an area that many people interact with on a daily basis but which also has quite surprising depths in a number of diifferent directions. So people tend to come with questions. I think we have answered all the questions satisfactorily - but it is not always to find the answers in these long threads.
If necessary (eg, if we don't have consensus for the FCP here) maybe I could write up a stabilisation report or something maybe.
I'm expecting Unrecognized
to remain perma-unstable, by way of avoiding the problem that occurred with Other
.
And yes, I think there have been a lot of questions, but I do think they've been answered. Anyone is free to call out questions that haven't been, though.
Shouldn't this be a libs-api FCP?
Shouldn't this be a libs-api FCP?
Probably, yes, as I understand it.
Anyone is free to call out questions that haven't been, though.
Absolutely.
I'm expecting
Unrecognized
to remain perma-unstable, by way of avoiding the problem that occurred withOther
.
Yes.
Going to cancel the fcp in progress because it was started under the wrong team. I haven't refreshed myself on the current state of the thread yet so I don't want to start the FCP since it will permanently check my box, @joshtriplett please restart the FCP under libs-api whenever you have a chance.
@rfcbot cancel
@yaahc proposal cancelled.
@rfcbot merge
Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:
No concerns currently listed.
Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!
See this document for info about what commands tagged team members can give me.
Sorry, I don't follow how this wasn't a breaking change. It was acknowledged as such much earlier on in the discussion. Code that did compile stopped compiling.
Sorry it wasn't very clear in my message, I mean that it didn't change nor remove any entity (variable, function, trait, module) or their signature.
The breaking change was the change in possible match values ErrorKind from exhaustive, to non-exhaustive. Thats part of the signature is it not?
This is a functional change in the sense that the
ErrorKind
variants returned by the IO functions aren't the same as before, no variant was removed or renamed, it's just that these functions don't return the same values anymore.
It is true that Other wasn't removed, but even with no change to return values, this would be breaking, because non-exhaustive is an attribute of the enum type.
I'm pulling on this thread because I think its important that this be classified as a breaking change, not as a 'functionally breaking change'.
A change in what variant is returned under some circumstances is a change that is only detected at runtime. So yes, that would be a functional breaking change.
This was detected at compile time. => a breaking change.
@rbtcollins it sounds like you've said a couple times you think std::io::ErrorKind
got changed from being exhaustively matchable to not being exhaustively matchable. That is not the case. ErrorKind
has never been exhaustively matchable in stable code in any existing release of Rust.
@dtolnay ahha. I had misremembered how the change happened. Thank you for helping me get clear on it. Sorry @ClementNerma for the unneeded contention.
I would like to clarify once again that the Other
->Unrecognized
change, which did in practice cause things to go wrong for a significant number of people, regardless of what kind of breakness we decree it to have had, has already happened and is not part of this FCP.
What we are proposing to stabilise now is just the ability to name the additional ErrorKind
s (IsADirectory
etc.) which are already being generated, currently-stably-unmatchably, So if this FCP passes it will not make any breakage or trouble worse in any way. Indeed, ability to match these variants is likely to be practically useful to (belatedly) properly fix some of the downstream test suites that suffered regressions. I appreciate that the introduction of Unrecognized
was confusing and controversial and awkward, but it is not to do with this FCP (nor indeed to do with this MR - it was done in some other MR).
I think the primary questions that ought to be asked apropos this FCP are:
ErrorKinds
the right ones? Specifically:ErrorKind
s are fine if Rust programmers need to be able to match them in portable code.I think the answers to these questions are "yes". We spent the bulk of the discussions in this MR about the details of individual errors, with a smattering of more general discussions about principles. The MR had a lot of attention from Unix and Windows experts and there has been a year for folks from other systems to chime in.
But, of course, we might have missed something, which is what the FCP is for.
What we are proposing to stabilise now is precisely the ErrorKind
s listed in this table.
I have included the Unix and Windows errno values that map to these kinds. We are stabilising these mappings, but if we have missed some errors that ought to be categorised into these kinds and are currently Uncategorized
, I think we could add those mappings later and it would be at worst a minor functional breakage.
ErrorKind | Unix | Windows |
---|---|---|
ArgumentListTooLong | E2BIG | |
CrossesDevices | EXDEV | ERROR_NOT_SAME_DEVICE |
Deadlock | EDEADLK | ERROR_POSSIBLE_DEADLOCK |
DirectoryNotEmpty | ENOTEMPTY | ERROR_DIR_NOT_EMPTY |
ExecutableFileBusy | ETXTBSY | |
FileTooLarge | EFBIG | ERROR_FILE_TOO_LARGE |
FilesystemLoop | ELOOP | |
FilesystemQuotaExceeded | EDQUOT | ERROR_DISK_QUOTA_EXCEEDED |
HostUnreachable | EHOSTUNREACH | ERROR_HOST_UNREACHABLE WSAEHOSTUNREACH |
InvalidFilename | ENAMETOOLONG | ERROR_FILENAME_EXCED_RANGE ERROR_INVALID_NAME |
IsADirectory | EISDIR | ERROR_DIRECTORY_NOT_SUPPORTED |
NetworkDown | ENETDOWN | WSAENETDOWN |
NetworkUnreachable | ENETUNREACH | ERROR_NETWORK_UNREACHABLE WSAENETUNREACH |
NotADirectory | ENOTDIR | ERROR_DIRECTORY |
NotSeekable | ESPIPE | ERROR_SEEK_ON_DEVICE |
ReadOnlyFilesystem | EROFS | ERROR_WRITE_PROTECT |
ResourceBusy | EBUSY | ERROR_BUSY |
StaleNetworkFileHandle | ESTALE | |
StorageFull | ENOSPC | ERROR_DISK_FULL ERROR_HANDLE_DISK_FULL |
TooManyLinks | EMLINK | ERROR_TOO_MANY_LINKS |
@ijackson for ELOOP, Windows appears to give winapi::shared::winerror::ERROR_CANT_RESOLVE_FILENAME in similar situations (e.g. symlink loops). Could we add that in, or perhaps generalise FileSystemLoop to the slightly more general case of being unable to resolve, before stabilisation?
I'm marking @yaahc's box because she has stepped down from T-libs-api after the point that this feature got proposed for FCP.
:bell: This is now entering its final comment period, as per the review above. :bell:
@rbtcollins What other things might ERROR_CANT_RESOLVE_FILENAME
mean? If it might mean other kinds of things besides loops then I think Windows fails to reliably distinguish filesystem loops from other kinds of errors. Which might mean that portable Rust programs aren't allowed to expect the operating system to reliably distinguish loops from other problems :-/.
@rfcbot concern Windows ERROR_CANT_RESOLVE_FILENAME
, FilesystemLoop
If we can't resolve this question immediately, I will remove FilesystemLoop
from the to-be-stablised list and we will have to deal with that one later. I don't want to block all the rest.
(Maybe someone with some rfcbot permissions could note the concern. This MR shouldn't merge without this being resolved, if only by dropping the stabilisation of that one kind.)
Oh wait this isn't a stabilisation MR - it's just an issue, so we can just make the MR not include FilesystemLoop
. But it would be good to be clear whether we are doing that.
The win32 error ERROR_CANT_RESOLVE_FILENAME
currently maps only from the kernel error STATUS_REPARSE_POINT_NOT_RESOLVED
.
The kernel error has the English error message as "The symbolic link could not be resolved even though the initial file name is valid". I'm pretty sure this situation can only occur when a loop is hit. EDIT: Or a long sequence of links that's not a loop but its length is greater than whatever the maximum is.
However, the win32 error message is more generic "The name of the file cannot be resolved by the system" so I guess the argument can be made that it does allow for more errors to be mapped to it in the future, should the need arise. I'm not sure that that would ever happen but I'll admit it is at least a possibility.
some system calls on Linux also use ELOOP
to mean "ELOOP A loop exists in symbolic links encountered during resolution of the path argument, or O_NOFOLLOW was specified and the path argument names a symbolic link." so I think interpreting it as "symlink loop or similar symlink resolve error was encountered" might be an accurate description, although (bike-shedding!) I don't know if FilesystemLoop
is an accurate name then, and not something like SymlinkResolutionFailed
or such...
I think NotSameDevice
is a better name than CrossesDevices
. For me, it expresses its meaning more clearly. FWIW, the Linux man page also uses the word "same" to describe EXDEV
: oldpath and newpath are not on the same mounted filesystem
.
Also, I find NotSameDevice
to be stylistically more consistent with the other existing and proposed error identifiers.
Can FilesystemQuotaExceeded
be renamed QuotaExceeded
? AFAIK Linux has no other quota error and EDQUOT
is also used generically (not only for filesystem quotas). A few other uses are visible here: https://elixir.bootlin.com/linux/v4.5/ident/EDQUOT. The comment in errno.h
simply describes it as "Quota exceeded".
I think there's a good correspondence @ijackson - as @ChrisDenton says the current meaning of ERROR_CANT_RESOLVE_FILENAME maps well to the loop meaning of ELOOP, and as @zseri has said, ELOOP itself isn't returned solely when loops are detected. Add to that list mount(2) returning ELOOP for move operations where the target
is a child of the source
- something that has absolutely nothing to do with symlinks, and execve
returning ELOOP for exceeding recursion limits during recursive script execution (since Linux 3.8).
I suggest renaming it to LoopError
, but document that it means ELOOP on Linux and ERROR_CANT_RESOLVE_FILENAME
on Windows, and either describe what we know right now, or provide breadcrumbs for readers to catch up.
The final comment period, with a disposition to merge, as per the review above, is now complete.
As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.
This will be merged soon.
I might be a little too late, but I'm not sure combining ERROR_FILENAME_EXCED_RANGE
and ERROR_INVALID_NAME
into InvalidFilename
is a good idea. What was the reason behind this decision?
For example if one is writing an unarchiver to extract files, if a filename is too long one could simply truncate the filename and tell the user this. But if the filename is invalid/reserved (eg. COM) then the name would have to be completely different.
An app could guess which of the two it is, but it would be more straightforward if the two are distinguished from the start.
It is a bit late but technically not too late; the stabilization PR has yet to be merged.
For example if one is writing an unarchiver to extract files, if a filename is too long one could simply truncate the filename and tell the user this. But if the filename is invalid/reserved (eg. COM) then the name would have to be completely different.
I could definitely see the two cases being handled differently in some situations.
An app could guess which of the two it is, but it would be more straightforward if the two are distinguished from the start.
Note that the original error is recoverable using raw_os_error()
. Though that isn't particularly ergonomic and is of course platform specific.
I was speculating there might be an OS that returns "Invalid Filename" when the filename is too long. In which case, this decision will make sense. But otherwise, "Filename Too Long" is one of the few errors that seem to be consistent among platforms.
With current PR, Unix doesn't have InvalidFilename
mapped to anything else (at least from my quick skim through sys/unix/mod.rs
).
However, UNIX-like OS fails with varying error when invalid UTF-8 filename is used to open()
in C.
Mac APFS fails with EILSEQ
(Illegal byte sequence).
Linux EXT4 (formatted with strict case-insensitive option -O casefold -E encoding_flags=strict
, because otherwise any byte sequence is allowed) fails with EINVAL
(invalid argument).
In theory, I think these should be mapped to InvalidFilename
in the context of File::open()
, File::create()
, etc. But I understand that that would need major rewrite.
I don't know enough to say what would be the best way, but current use of InvalidFilename
doesn't feel optimal.
PS. I've never contributed to rust dev. so I'm not sure of the procedures if this should get reviewed before standardization
PS. I've never contributed to rust dev. so I'm not sure of the procedures if this should get reviewed before standardization
I'll make a note on the stabilization PR that there's still some concern around the current mappings. Maybe the libs-api team will want to look at this again.
Currently io::Error
stores OS errors internally as essentially OsError(i32)
so there's no context available when getting the io::ErrorKind
.
I think that a new error variant for the EMFILE
error code on Unix platforms could be useful for embarrassingly parallel applications. I would like to contribute a PR for such a new variant, would it be a welcome addition?
Is it blocked on t-libs-api? I feel like that’s an important feature since it’s really hard to avoid TOCTOU without it, e.g.
if let Err(err) = fs::remove_file(path) {
if matches!(err, io::ErrorKind::IsADirectory) { ... }
}
becomes something racy like
if !path.is_dir() {
fs::remove_file(path)?;
}
Maybe some less-controversial subset could be stabilized sooner? What’s blocking this issue from making progress?
one additional case that would be nice for this is "Too many open files" (EMFILE, 24 on linux)
i ran into this when writing async code, and there doesn't seem to be any way to test it besides checking the message.
@lolbinarycat On Unix-like you can do .raw_os_error()
to get an error code + nix::errno::Errno
for a portable list of errnos.
I would like to propose that we stabilize these. Based on the feedback above, these are very useful to have in stable, and there seem to be no real risks. Concretely, this would mean marking the following as stable:
* `io::ErrorKind::HostUnreachable` * `io::ErrorKind::NetworkUnreachable` * `io::ErrorKind::NetworkDown` * `io::ErrorKind::NotADirectory` * `io::ErrorKind::IsADirectory` * `io::ErrorKind::DirectoryNotEmpty` * `io::ErrorKind::ReadOnlyFilesystem` * `io::ErrorKind::FilesystemLoop` * `io::ErrorKind::StaleNetworkFileHandle` * `io::ErrorKind::StorageFull` * `io::ErrorKind::NotSeekable` * `io::ErrorKind::FilesystemQuotaExceeded` * `io::ErrorKind::FileTooLarge` * `io::ErrorKind::ResourceBusy` * `io::ErrorKind::ExecutableFileBusy` * `io::ErrorKind::Deadlock` * `io::ErrorKind::CrossesDevices` * `io::ErrorKind::TooManyLinks` * `io::ErrorKind::InvalidFilename` * `io::ErrorKind::ArgumentListTooLong`
The issues around
io::ErrorKind::Other
and the unexpected failures they caused have been resolved with the introduction ofio::ErrorKind::Uncategorized
. This also eliminates the potential breakage from exhaustive matching. As such, stabilization itself should not be a breaking change anymore.As to the concern raised by @m-ou-se, I have gone over all of these errors, and almost all of them seem to have compatible OS error conditions on both windows and any posix compliant system. As such, platform specific issues should only occur on unusual platforms outside of either the current tier 1 or tier 2 lists. I think we should be OK treating any cases where these errors are not generated currently as bugs. A further search of the current open issue list suggests no such issues are currently known.
There are two exceptions: First,
io::ErrorKind::FilesystemQuotaExceeded
andio::ErrorKind::FileTooLarge
do not seem to correspond to posix standardised errors. I think for these in particular it is acceptable if they fall back toio::ErrorKind::StorageFull
, which seems to be what the posix standard suggest would happen under those conditions. Second,io::ErrorKind::ExecutableFileBusy
andio::ErrorKind::Deadlock
seem to be best effort, and have limited system support, but this seems to me well enough documented.
Something I noticed that https://doc.rust-lang.org/stable/src/std/sys/pal/unix/mod.rs.html#261 uses HostUnreachable
as the output of kind()
however unless you're nightly you can't actually match to this since that variant is feature guarded. This is unfortunate since that's the kind that's returned by TcpStream when a host is offline. Thus when I'm trying to match this error type to attempt a retry, I can't unless I search for the code
instead, not as convenient. This list needs to make its way to stable ASAP if its already being used to report errors. Whats worse is that this was done 3 years ago and no one has since complained that they can't match this kind w/o nightly?!
17/21 of the added variants got stabilized: #128316
The remaining 4 got dedicated discussion issues with a recap of all the concerns voiced up to this point:
The previous stabilization PR (#106375) should probably be closed.
Bonus issue: #130193
Feature gate:
#![feature(io_error_more)]
This is a tracking issue for many
io::ErrorKind
variants being added in #79965.The additions were motivated by a perusal of Unix standards, but corresponding mappings for Windows were included in that PR.
Public API
std::io::ErrorKind::Foo
for manyFoo
. I will paste a list here when the MR is merged :-).Steps / History
Unresolved Questions
ErrorKind
variants ? It may not be necessary to resolve this wider question in order to stabilise this particular set ofErrorKind
s.