Closed nazar-pc closed 4 years ago
Thanks for the bug report! I was able to reproduce, identify the issue, and fix it.
async-fs
does not use async-io
. Instead, it uses the blocking
crate to simply offload blocking I/O operation onto a threadpool and hide that fact behind an async API. This is because operating systems have poor support for truly async file I/O. IOCP and AIO are kind of broken (they sometimes block), and io_uring is still very much new.
The understand the problem, note that blocking
reads the file on the threadpool in advance, as much as it can, storing that data into an internal buffer. That means if you do a read operation followed by a seek operation, the seek might be off because the blocking
crate decided to read more data from the fd than was actually consumed from the perspective of async_fs::File
. This is why async_fs::File
keeps track of how much data it has read, so if a read operation is followed by a write or a seek operation, it will first correct the actual file cursor.
The catch is in that not all files are seekable. Here, you're using pipes that are not seekable so whenever async_fs::File
wanted to "remember" the file cursor position before reading data from it, it would blow up with an error that says "Illegal seek". Then, that would case the thread to exit with that same error, thus dropping the async_fs::File
, which is why "broken pipe" ends up being reported.
The solution is to ignore seek errors and accept the fact that some files simply aren't seekable, so in that case we should not try doing anything to correct the file cursor: #5. I have published a new version so if you do cargo update
, everything should work fine.
Finally, one more thing. :) The async_io::Async
type actually does work with some kinds of Unix "files" (because everything is a file on Unix systems). It doesn't work with actual files on the filesystem, but it does work with pipes because pipes are fully supported by epoll. In fact, I tried replacing async_fs::File
in mediasoup with async_io::Async<StdFile>
and it worked fine! Maybe I should clarify that in the docs...
I have faced an issue when doing
async_fs::File::from(File::from_raw_fd(fd))
, wherefd
is a pipe.The issue is that in non-blocking mode I'm getting errors about broken pipe in child worker process that is trying to write to the pipe like
uv_write(): broken pipe
: https://users.rust-lang.org/t/piping-file-descriptors-beyond-std-io-out-err-to-child-process/48089There doesn't seem to be such issue when using blocking mode with child threads.
Is this related to
async-io
's note that is?:I've prepared reduced example that can be run with:
The issue happens when I'm doing reads on
async_fs::File
. Same reads with blocking code works fine, but requires a separate thread.