Tokio primitives not including in this port

[withoutboats]

This port is just supposed to be the core of tokio, to unblock people who want to build on top of async/await. Much of tokio is not included in this for that reason.

For example, tokio-threadpool is an executor implemented using a work-stealing threadpool. It's a valuable step up over the executors provided in the futures crate, but not an IO primitive, so not a part of this particular port's vision of itself.

Similar, tokio-codec defines a set of abstractions for writing network protocols; while these are very valuable they also aren't a part of the "just the async IO primitives" vision that this crate has.

However, other parts of tokio are more ambiguous:

1. tokio-signal provides signal abstractions on UNIX and Windows. It's not clear to me if this should be ported or not (its not included as part of tokio proper)
2. tokio-fs provides FS abstractions; however, it is not actually using nonblocking IO (there is no portable nonblocking filesystem IO), its using blocking IO on top of the executor, and tightly coupled to tokio-threadpool. Is it urgent to provide an FS abstraction as part of this, and how hard would it be to provide one that is decoupled from the executor implementation?
3. tokio-timer provides a timer implementation, not technically an IO primitive per se but still very important and relevant as a networking primitive. should we port that also?

[yoshuawuyts]

If I'm reading this correctly, it sounds like romio at this stage is more about unblocking people for their common cases rather than providing best-possible performance. I'm more or less interpreting that as: "Having APIs available is more important than how well they perform".

1. tokio-signal: I don't think this is too important. From experience, setting up signal handlers is usually something that's nice to have, but rarely crucial.
2. tokio-fs: I think this would actually be good to have. The fact that it's backed by sync system calls seems less important than getting people to use code that has the right shape. Backing it by (portable) async IO could then become a seamless upgrade later down the line, without requiring people to change function signatures again.
3. tokio-timer: similar to tokio-fs, this seems important to get the API right for. If the existing code is tricky to port, perhaps using a simpler implementation might make more sense (e.g. not using a hashed wheel timer, but something more straight forward).

[withoutboats]

The timer would not be hard to port, seems like a good idea. File system APIs OTOH are much harder because their implementation is currently coupled to tokio-threadpool.

[aturon]

@withoutboats I'm not deeply familiar with the API surface there, but (riffing on @yoshuawuyts's angle) it seems feasible to provide a more direct/naive implementation that doesn't require this coupling.

[withoutboats]

We got rid of the spawn API on the context object (now just a waker). I'm not sure how even a more naive implementation could do something like "spawn a blocking read as another task on the current executor" unless we brought the "default spawn API" stuff back either here or in futures. We could just implement them as blocking APIs that look nonblocking, but that seems nonideal.

I'll look more into the implementation in tokio and the APIs we do have in futures, maybe it can inform the executor APIs.

it would be a completely new implementation of async FS IO though, not a port. which is fine!

[aturon]

Ahha I see; I wasn't thinking about needing to have some canonical place to spawn. Of course one option is to just require an executor to be passed in.

FWIW, I feel like this points to an ecosystem-wide problem we should solve, in a way that's independent from any particular I/O stack. Rayon has a pretty nice story here. Probably worth opening an issue on WG-net.

I think it's fine for the fs APIs to not make the first cut, to be clear.

[withoutboats]

I'm going to port tokio-timer and not tokio-signal or tokio-fs. Later, I think trying to solve the filesystem situation that led to tokio-fs without tying the impl to a specific executor will have fruitful results for evolving on the executor API.

[kpp]

Similar, tokio-codec defines a set of abstractions for writing network protocols; while these are very valuable they also aren't a part of the "just the async IO primitives" vision that this crate has.

I would port tox-rs to your crate if you implemented a must-have feature codec. A lot of code depends on Encoder, Decoder and Framed, so if you have an idea of a workaround it would help a lot.

Also I noticed tokio is not usable on wasm target. We would love to implement a client in wasm but we need an async io for WebSocket. mio depends on epoll which is unavailable for wasm target, but it can be done by integrating web_sys::WebSocket and wasm_bindgen_futures (probably).

[Nemo157]

I think having the functionality of tokio-codec is definitely going to be important for trying out romio, but as @withoutboats mentions this isn’t specific to the low-level IO primitives. Since we have the futures-io@0.3 traits this could be implemented as something like futures-io-codec and work with any async io implementation. @kpp it might be worth bringing it up in an issue on the futures repo as to whether it’s something that should be prototyped there, or as a separate project.

[najamelan]

@kpp I recently implemented tokio AsyncRead/AsyncWrite for websockets in WASM. It's quite proof of concept at the moment, but the code base is small, and it can be improved (main issue, the unit tests fail in release mode and I haven't figured out why). For futures 0.1, the wasm-workgroup are implementing a nicer api to websockets in wasm.

@withoutboats would it make sense to implement an equivalent to tokio-codec for the futures::io::AsyncRead from 0.3. Mainly, I wonder is futures::io::AsyncRead the logical successor of tokio AsyncRead? Would that component shift from the tokio crate to the futures crate after async/await stabilizes?

[yoshuawuyts]

ould it make sense to implement an equivalent to tokio-codec for the futures::io::AsyncRead from 0.3.

I think definitely! I almost have something that can be released under the async WG; if not next week, then probably the week after. Fully with you that it's important to make converting between Async{Read,Write} and {Sink, Stream} easy.

[najamelan]

I almost have something that can be released under the async WG; if not next week, then probably the week after.

Awesome. tokio codec is the main reason I'm still using futures 0.1 in some places. Looking forward to migrating that out.

[najamelan]

btw, is your code in a public repository or is it to early?

[yoshuawuyts]

@najamelan it's private, but I can send you an invite on it to try out early if you like (: -- still working out some edges before a release, but having another set of eyes to look at it is always helpful!

[najamelan]

@yoshuawuyts I'd love to beta test. I have some codecs to write soon, and I'd rather target the future than the past ;) If you have specific review questions, it can help focus on the parts you're not happy with. Thanks for the invitation.

[kpp]

Awesome. tokio codec is the main reason I'm still using futures 0.1 in some places. Looking forward to migrating that out.

It is time to write futures-codec with a modern async/await design and with futures 0.3 support!

[cbs228]

@najamelan et. al., I have written a generic codec as part of my ardop_interface crate. It presently targets futures-preview 0.3.0-alpha.16. I'm not sure it's good enough to publish on its own, but you are welcome to use it if you need one. Like the old Tokio 0.1 codec, this one depends on bytes for its API. The examples include a newline framer and a server demo. The crate itself has more complicated framers internally.

[kpp]

@cbs228 there is https://crates.io/crates/futures_codec. Is it enough?

[cbs228]

@kpp: That crate escaped my notice when I began development. It very likely will suffice!

[yoshuawuyts]

@cbs228 alternatively it's also possible to use the codec pattern. This isn't always the right choice, especially when porting older code from e.g. tokio-codec, but might be a nice tool to use for newer projects.