smoltcp-rs / smoltcp

a smol tcp/ip stack
BSD Zero Clause License
3.74k stars 420 forks source link

[RFC] Splitting EthernetInterface into composable components #166

Open batonius opened 6 years ago

batonius commented 6 years ago

I've been thinking about #55 for a while and I've come up with a much broader plan which would require significant refactoring, so I'm posting my not-so-concrete-yet thoughts here to get some early feedback.

The basic idea is pretty obvious - to split up the EthernetInterface monstrosity into components representing layers in the classical network scheme, specifically the Network (aka IP) Layer, and the Link Layer, to which I would add the third, intermediate layer - the Routing Layer. The next logical step is to express these layers as traits and split the stack into independent composable modules, making each implementation of a layer generic over the specific implementation of the underlying layer, something like EthernetInterface can be parametrized with Device now. It also would be possible to create middleware layers which consume an instance of a layer and implement the same layer on top of it with additional features.

Now, one of the problems with smoltcp as a project IMO is the inherently conflicting requirements the users have - coming from Redox, I'm all for using std and alloc everywhere to get more advanced features like routing and multiple interfaces, something the embedded users understandably want to be able to avoid. With the proposed organization, smoltcp becomes not so much a fits-all network stack, but a set of components which could be easily put together to create a network stack with the desired set of features, with some components unapologetically relying on std and alloc, while the others being no_std. Since the concrete organization of the stack will be known at the compile time, all the calls to the intermediate layers will be inlined and no dynamic dispatch will be used implicitly.

The current stack would be equivalent to NL -> Default Gateway RL -> Ethernet LL -> Device, the simplest would be NL -> Loopback LL (since there's nothing preventing a type from implementing both NetworkLayer and RoutingLayer traits), and Redox would use something like NL -> IP Fragmentation RL -> Looback RL -> Table Routing RL-> Multiplexing LL, the latter using dynamic dispatch to LinkLayer to operate on a set of Ethernet LL, PPP LL, 802.11 LL, etc. Alternatively, the multiplexing could be fixed with the predefined set of Link Layers:

smoltcp

batonius commented 6 years ago

Now the layers themselves, the main difference between the LinkLayer and RoutingLayer traits is the IfaceId argument used in the former to identify the interface a packet has been received from or should be transmitted to. The Network Layer knows nothing about the interfaces, their IPs, neighbors, routes, etc. and uses the abstraction provided by a RoutingLayer, but receives a reference to IfaceCapabilites struct with each packet to see if checksum validation is necessary. Neighbour discovery, ARP tables, are implemented at the Link Layer since the specifics depend on the technology.

The specifics of the traits is the key point of the solution and I'm far from being happy with them, but right now I see them roughly as

enum RouteError<E> {
    CantRoute,
    Other(E),    
}

enum LinkError<E> {
    CantLink,
    Other(E),    
}

enum Action<'a> {
    None,
    Arp(ArpRepr),
    Icmpv4((Ipv4Repr, Icmpv4Repr<'a>)),
    Icmpv6((Ipv6Repr, Icmpv6Repr<'a>)),
    Raw((IpRepr, &'a [u8])),
    Udp((IpRepr, UdpRepr<'a>)),
    Tcp((IpRepr, TcpRepr<'a>))
}

trait RouteLayer {
    fn poll<'a, F, E>(&mut self, f: F) -> Result<usize, RouteError<E>>
        where F: Fn(&IfaceCaps, &'a IpPacket, &'a [u8]) -> Result<Action<'a>, E>;

    fn emit<F, R, E>(&mut self, &IpRepr, f: F) -> Result<R, RouteError<E>>
        where F: Fn(&IfaceCaps, &mut [u8]) -> Result<R, E>;
}

trait LinkLayer {
    fn poll<'a, F, E>(&mut self, f: F) -> Result<usize, LinkError<E>>
        where F: Fn(IfaceId, &IfaceCaps, &'a IpPacket, &'a [u8]) -> Result<Action<'a>, E>;

    fn emit<F, R, E>(&mut self, IfaceId, &IpRepr, f: F) -> Result<R, LinkError<E>>
        where F: Fn(&IfaceCaps, &mut [u8]) -> Result<R, E>;
}
whitequark commented 6 years ago

I really like your proposal! With one exception though:

with some components unapologetically relying on std and alloc, while the others being no_std.

I don't think any of smoltcp's components should ever rely on std. Indeed, what is there in std that we need that isn't in alloc? Hash tables? I'd like moving HashMap into alloc somehow far more than adding an std dependency.

I also have pretty strong doubts about any of smoltcp's componets having a hard dependency on alloc. The managed crate provides dispatch for Box, Vec and BTreeMap; surely any other collections can be added to it in similar ways.

fintelia commented 6 years ago

I also like that this proposal has the potential to enable smoltcp to be used for only part of a network stack. As best I can tell, this isn't really possible right now (though if it is, I'd be interested to hear!)

batonius commented 6 years ago

@whitequark It would be nice to be able to use std-depended crates for some fancy data structures, for example, something managed won't support. Then again, with the proposed scheme, there's nothing stopping Redox from implementing its own routing layer using whatever we want and then just putting it in between stock layers.

whitequark commented 6 years ago

As best I can tell, this isn't really possible right now (though if it is, I'd be interested to hear!)

You can use the TCP sockets with all underlying layers your own.

batonius commented 6 years ago

The first attempt: https://github.com/m-labs/smoltcp/compare/master...batonius:layers , I've decided against separate traits for routing and link layers. Currently, the trait looks like

pub struct IfaceDescr<'a> {
    pub dev_caps: DeviceCapabilities,
    pub addrs: ManagedSlice<'a, IpCidr>,
    pub id: IfaceId,
}

pub struct RoutingData {
    pub iface_id: IfaceId,
    pub dst_host: IpAddress,
}

pub trait Layer {
    fn poll<F, R>(&mut self, F) -> Result<R>
    where
        F: Fn(&IfaceDescr, IpPacket) -> R;

    fn emit<F, R>(&mut self, RoutingData, F) -> Result<R>
    where
        F: Fn(&IfaceDescr, &mut [u8]) -> Result<R>;
}

I've implemented PoC loopback and default route layers, next I'm gonna implement the network layer on top of the interface.

whitequark commented 6 years ago

Well this doesn't work at all, what with the many ManagedSlice::Owned instances in your PoC code.

whitequark commented 6 years ago

On top of that, I think you're doing too much at once. I'd like to see the split of EthernetInterface into multiple layers separately from adding routing; either of these is complex enough for many weeks of work, trying to tackle them together is absurd!

batonius commented 6 years ago

what with the many ManagedSlice::Owned instances in your PoC code.

Well, that's PoC after all, I'm going to rewrite the Loopback layer using RingBuffer once I'm happy with the trait interface.

I'd like to see the split of EthernetInterface into multiple layers separately from adding routing;

The routing layer is equivalent to the current routing scheme using the default gw, plus support for routing to multiple interfaces. The alternative would be to keep routing in the networking layer, which defeats the purpose of the proposal.

Another issue I want to bring up is the fact that it won't be possible to reuse ingress packets to construct responses anymore since the trait interface assumes the underlying layer is responsible for both ingress and egress packets, so it's not possible to borrow it mutably to send a packet while borrowing it to get one. Even if we split the interface the way we do with Device, it won't always be possible to implement, for example for a loopback layer based on RingBuffer. I'm going to keep a fixed local buffer where the relevant parts of ingress packets could be copied to to be used in responses, I don't think it would be that great of a problem considering it's only required for some ICMP packets.

whitequark commented 6 years ago

I'm not happy about this. You're making smoltcp perform significantly worse for the use case it is primarily intended for, which is small embedded devices. I don't think this is acceptable.

batonius commented 6 years ago

Then I'll try to rethink the design to keep it this way, maybe along the lines of the old Device trait, with handler types owning packet buffers and returning them in destructors. The counterexample of loopback based on RingBuffer would still be impossible tho.

whitequark commented 6 years ago

with handler types owning packet buffers and returning them in destructors

Hmm, are you sure? That seems like potentially a waste of time; the scheme with Drop was a major design smell and it caused no end of headache with lifetime issues.

Could it make sense to split ingress and egress on every layer?

batonius commented 6 years ago

I'm back, sorry for disappearing for so long.

The last couple of weeks I've been playing with the splitting approach to the issue, mostly trying to resolve the lifetime issues. The designs I have in mind all require HKT, or at least ATC to work - not being able to have an associated type generic over lifetimes really hurts expressiveness. Still tho, I've got an example working and I would like to know your opinion on the results so far: https://github.com/m-labs/smoltcp/compare/master...batonius:layers .

The main breakthrough was the realization that I don't have to use a single RingBuffer/Vec to implement a loopback interface - I can use separate input and output queues and 'sync' them regularly by swapping. This allowed me to implement the splitting API able to emit packets while polling them.

As previously, this is a PoC and not a PR-ready code. I think we should first decide on the layers API before moving on.

cc @dlrobertson I would like to know your thoughts too.