Open oberstet opened 8 years ago
oberstet
commented
8 years ago If WAMP is supposed to run over (regular) UDP, and we don't want to implement fragmentation/reassembly of WAMP messages on top of UDP (which I'd be sceptical doing), then we need to cope with
args/kwargs)oberstet
commented
8 years ago The GNRC does not seem to provide a streaming API for receiving UDP datagram payload. There needs to be buffer space capable of holding the whole payload.
This might touch on our discussion of a SAX like API for user/app payload, as in: if transport payload (UDP datagram) isn't received streaming, so why do SAX/streaming for exposing the args/kwargs anyway?
At least the CoAP example doesn't do it like this. It works like this in an endless loop:
conn_udp_recvfrom(&conn, (char *)_udp_buf, sizeof(_udp_buf), raddr, &raddr_len, &rport);
coap_parse(&pkt, _udp_buf, n)
coap_handle_req(&scratch_buf, &pkt, &rsppkt);
coap_build(_udp_buf, &rsplen, &rsppkt)
conn_udp_sendto(_udp_buf, rsplen, NULL, 0, raddr, raddr_len, AF_INET6, COAP_SERVER_PORT, rport);with memory:
static uint8_t _udp_buf[512]; /* udp read buffer (max udp payload size) */
uint8_t scratch_raw[1024]; /* microcoap scratch buffer */
coap_packet_t pkt;
coap_packet_t rsppkt;Now, I had a nice, longer discussion on IRC with the RIOT guys (will post below), but here: https://github.com/RIOT-OS/RIOT/issues/5371
oberstet
commented
8 years ago -riot-notifier/#riot-os- [RIOT] authmillenon opened pull request #5369: gnrc_ndp: don't let addresses timeout (2016.04-branch...gnrc_ndp/fix/gua-hack-bp) https://github.com/RIOT-OS/RIOT/pull/5369
<oberstet> Hi! I'm new to RIOT, and have a few questions / need some design advice. We want to expand WAMP (http://wamp-proto.org/, http://crossbar.io/) to smallish devices, in particular MMU-less MCUs, and in particular RIOT based ones. This effort: https://github.com/crossbario/autobahn-c (which is supposed to be a new member of the Autobahn family)
-riot-notifier/#riot-os- [RIOT] kYc0o pushed 2 new commits to master: https://github.com/RIOT-OS/RIOT/compare/01650a446008...25356dffc697
-riot-notifier/#riot-os- RIOT/master 39043b5 Hauke Petersen: cpu/samd21: fix ISR mapping for PB27
-riot-notifier/#riot-os- RIOT/master 25356df kYc0o: Merge pull request #5353 from haukepetersen/fix_samd21_pb27...
<OlegH> Hi oberstet and welcome to RIOT! :)
<OlegH> Looking at the links
<cgundogan> looks fine
<eintopf> I do understand autobahn only
<cgundogan> (:
<cgundogan> oberstet: what is the actual question?
<oberstet> eg, am I right, that the "new way" to use UDP on RIOT is via the Generic (GNRC) network stack, and NOT via the (presumably) Posix UDP adaption? That is, I should follow the https://github.com/RIOT-OS/RIOT/tree/master/examples/microcoap_server, and NOT https://github.com/RIOT-OS/RIOT/tree/master/examples/posix_sockets as a blueprint / role model?
<oberstet> ^
<oberstet> another one: looking at this line https://github.com/RIOT-OS/RIOT/blob/master/examples/microcoap_server/microcoap_conn.c#L21 .. to receive full, regular IPv6 UDP datagram (from some border router like thing), that would need to be 64k sized? = buffer fully preallocated? isn't that "big" / problematic?
<cgundogan> oberstet: this actually depends on your use case. If you want your app to be more or less portable across posix systems, theyn you would use posix sockets. But the posix layer adds unnecessary complexity if you just want to run your app in RIOT. Then I would recommend the GNRC way.
<oberstet> cgundogan: cool. we'll write that with main target RIOT, no need for Posix. For a full Posix sys, we have AutobahnC++ (Boost/ASIO based) anyway ..
<oberstet> thanks for that info already. answers one question.
<OlegH> oberstet, well, yes, if you want to receive a full UDP datagram (64kB) on a memory constrained device, you have a problem.
<OlegH> But I guess that was not the question. ;)
<oberstet> in a way, it is. eg we've been thinking about a streaming, SAX like, not-DOM like API for exposing WAMP to user app code. because of no-malloc and limited memory. but now I wonder: GNRC doesn't seem to stream payload in, but needs fully allocated datagram buffer?
<oberstet> neither the GNRC nor the Posix API expose a streaming API to UDP datagram payload, right?
<OlegH> Not that I know of. I guess none of us had thought much about streaming applications in an IoT context. :-/
<cgundogan> I am not quite sure what you mean by streaming?
<OlegH> In principle, you have malloc() (e.g. by using the tlsf package), but the problem of limited memory stays.
<oberstet> FWIW, a streaming API to UDP datagram payload would allow to register a user code callback that is called from the networking stack when new octets arrive for that UDP datagram being received. So there wouldn't be a preallocated, whole datagram buffer ..
<oberstet> eg the UBJSON thing in RIOT is streaming like ..
<oberstet> OlegH: we do want to avoid any malloc in AutobahnC
-riot-notifier/#riot-os- [RIOT] authmillenon opened pull request #5370: gnrc_ipv6: Revert #5179 (backport) (2016.04-branch...gnrc_ipv6/fix/revert5179-bp) https://github.com/RIOT-OS/RIOT/pull/5370
<OlegH> You could take a look a the GNRC UDP implementation. It's not big or complicated. I guess it can be extended by such a mechanism rather easily.
<OlegH> https://github.com/RIOT-OS/RIOT/blob/master/sys/net/gnrc/transport_layer/udp/gnrc_udp.c
<cgundogan> you can also use the gnrc API that is below the conn api to register to
<cgundogan> basically, what OlegH said
<OlegH> oberstet, and I would not object this. IMO malloc/free add a lot of unnecessary complexity on embedded systems.
<cgundogan> this way you would get the packet handed-over from the network stack. Only if you have multiple services requesting the same packet, then this packet will be copied, otherwise you just receive a pointer to incoming packets
<oberstet> looking at the gnrc_udp thing
<OlegH> FYI: inside GNRC we use a packet buffer that allows for a kind of dynamic memory allocation for headers and payload.
<cgundogan> but I still miss the connection to "streaming" (: Maybe I just think of something else when I hear "streaming"
<OlegH> cgundogan, stop thinking about https://de.wikipedia.org/wiki/Striemen-Schilfeule
<cgundogan> OlegH: stop reminding me then (;
<OlegH> btw do you know https://de.wikipedia.org/wiki/Schokoladen-Fruchtzwerg ?
<cgundogan> subfamily of fruitvampires?
<cgundogan> this world is a strange place
<oberstet> OlegH: where is 6LoWPAN reassembly done?
<oberstet> cgundogan: "streaming" as in process bytes on the fly, without preallocated buffers for the whole thing
<cgundogan> oberstet: probably somewhere in https://github.com/RIOT-OS/RIOT/blob/master/sys/net/gnrc/network_layer/sixlowpan/frag/gnrc_sixlowpan_frag.c
<cgundogan> or maybe https://github.com/RIOT-OS/RIOT/blob/master/sys/net/gnrc/network_layer/sixlowpan/gnrc_sixlowpan.c
<oberstet> is reassembly there only up to IPv6 MTU 1280, or right up to 64k full UDP datagram?
<eintopf> so far I know 1280, because IPv6 fragmentation isn't supported
<OlegH> Not 100% sure, but I think it reassmbles until IPv6 MTU (which is per default configured to 1280).
<eintopf> we (linux) has ipv6 fragmentation :-)
<eintopf> OlegH: 1280 is defined by IETF
<OlegH> eintopf, yes, but it's not a fixed value.
<eintopf> I think yes
<OlegH> it's just the just the minimum MTU for IPv6.
<oberstet> minimum transmission unit 1280 is in IPv6 RFCs
<eintopf> https://tools.ietf.org/html/rfc4944#section-4
<cgundogan> I think the 6low fragmentation related fields are not big enough to map to a full 64k packet
<oberstet> sorry for all that questions, but we need to decide about a couple of critical design questions with AutobahnC quite upfront ..
<OlegH> eintopf, yes, it says: "This is obviously far below the minimum IPv6 packet size
<OlegH> of 1280 octets"
<OlegH> *minimum*
<OlegH> oberstet, no need to apologize, sounds interesting
<eintopf> The first sentence is: The MTU size for IPv6 packets over IEEE 802.15.4 is 1280 octets.
<OlegH> which is not accurate if you look it up in 2460
<OlegH> https://tools.ietf.org/html/rfc2460#section-5
-riot-notifier/#riot-os- [RIOT] cgundogan closed pull request #5343: gnrc_sixlowpan_iphc.c: handle forwarded GNRC_NETTYPE_IPV6 packet (backport) (2016.04-branch...gnrc_6lo_nhc/fix/forwarding) https://github.com/RIOT-OS/RIOT/pull/5343
<oberstet> 1280 is _link_ layer minimum for IPv6, rather IP packet size. an IP packet can be larger, and then needs to be fragmented/reassembled
<eintopf> yes this is IPv6, I think it's invalid to send 6LoWPAN fragments which are reassembled above 1280 IPv6 packet
<cgundogan> eintopf: I think its not just invalid, its just impossible
<eintopf> for a special value I agree the dgram_size/dgram_offset will reach some maximum
<eintopf> but 1281 is possible :-)
<eintopf> but will maybe break some 6LoWPAN implementations
<OlegH> 2^11 is the maximum for 6LoWPAN fragmentatio
<oberstet> ok, I haven't read much into the 6LoWPAN RFCs .. but I was under the impression, that 6LoWPAN is exactly about providing the 1280 MTU required by IPv6 via reassmbly mechanisms working at the link layer
<OlegH> oberstet, that is correct
<cgundogan> OlegH: is right, its 2047 bytes
<oberstet> OlegH: ok, thanks for confirming
<OlegH> I don't know if authmillenon has thought about IPv6 fragmentation.
<eintopf> it's 2047 bytes, but I think if you send above 1280 bytes IPv6 packets then it's just invalid. If you reach that maximum, you need to start with IPv6 fragmentation
<oberstet> so reassembly would be at multiple layers: 802.15.4 -> 1280 MTU -> IP packet reassembly -> UDP datagram assembly
<OlegH> eintopf, do you have a source for this assumption?
<cgundogan> oberstet: another question, I read somewhere your license will be MIT. OlegH: as a license-guru, what again was the compatability with LGPL?
<OlegH> cgundogan, good
<oberstet> MIT is compatible with any GPL flavor
<eintopf> OlegH: I thought it is: The MTU size for IPv6 packets over IEEE 802.15.4 is 1280 octets.
<eintopf> and no other MTU should be set
<oberstet> which is the reason we moved from Apache 2.0 to MIT for AutobahnXXX
<OlegH> from the frequently used open source liceneses only Apache and Eclipse license are problematic with (L)GPL.
<cgundogan> okay
<OlegH> eintopf, not sure I read it this way, but okay, I see
<eintopf> "The MTU size for IPv6 packets over IEEE 802.15.4 is 1280 octets." means in other words, don't send IPv6 packet which has more than 1280 bytes (L3 header + payload)
<eintopf> OlegH: maybe we can ask this on 6lo
<eintopf> now I am also not sure
<eintopf> I mean you can send more, but then with IPv6 fragmentation
<oberstet> IP has a (fixed) maximum packet size of 64k. if the MTU is lower than that (most often), then the stack needs to do IP fragmentation/reassembly
<cgundogan> which is currently not supported in RIOT
<oberstet> huh. ok.
<OlegH> eintopf, reading https://tools.ietf.org/html/rfc4944#section-5.3 I think you're right
<oberstet> no IP fragmentation/reassembly: how can you then use UDP with datagram payloads larger than 1280?
<OlegH> It doesn't explicitely say that you SHOULD discard a bigger datagram though.
<OlegH> oberstet, on 6LoWPAN you currently cannot
<eintopf> we do that @linux implementation
<oberstet> whaat?
<cgundogan> oberstet: application-layer fragmentation? (;
<oberstet> weeell;)
<OlegH> Do you have a use case where you want to send that much data over 6LoWPAN?
<oberstet> alright. I guess then my questions rgd a streaming API to UDP payloads .. are moot for now;)
<eintopf> OlegH: the datagram_size: i.e., this field needs to encode a maximum length of 1280 octets -> says to me: filter everthing above that
<OlegH> eintopf, to me it says: this field must be big enough to encode this
<cgundogan> I don't see the connection to streaming apis and the packet payload size (:
<oberstet> thing is, we want to define a UDP based transport for WAMP, which does handle reordering/retransmission at the app layer, but NOT fragmentation/reassembly. that WAMP transport of course is then limited to 64k WAMP message payloads, but that is ok. but I don't want to get into fragmentation/reassembly ..
<oberstet> cgundogan: 1280 is small enough that a streaming API isn't that important. but 64k ..
<oberstet> but yes, it's not related in principle
<OlegH> oberstet, understandable
<eintopf> I transmitted one day 64k over 2.4 Ghz 802.15.4 6lowpan
<cgundogan> oberstet: is maybe event-based api a better name for streaming based? Where you get all udp packets chuck for chunk from an application view of point?
<eintopf> took 10 seconds
<eintopf> rtt
<oberstet> cgundogan: yeah, call it event based. its similar when compared to XML (shudder) APIs: SAX vs DOM
<OlegH> eintopf, that one _can_ do it, does not make it a use case ;)
<oberstet> eintopf: huh, ok. interesting.
<eintopf> OlegH: exactly
<oberstet> the UBJSON thing in RIOT _is_ a SAX like API for example ..
<eintopf> OlegH: the qdisc (tx queue strategy) will drop such things
<oberstet> aka event based, callback based, whatever you call it
<cgundogan> (:
<cgundogan> if you call it this way, then yes we have such an api (pointed out by OlegH earlier)
<oberstet> cgundogan: ok, but its deep down inside the GNRC stack, right? what OlegH point me to ..
<cgundogan> for me, streaming and networking is TCP (:
<oberstet> I call it a "streaming API to UDP datagram payloads" - but thats just words ..
<cgundogan> oberstet: its a core part of the GNRC network stack, but I wouldn't say its too deep to use it for an application's use
<OlegH> Well, point is: there's no IP fragmentation in GNRC (don't know about the other stacks in RIOT).
<cgundogan> a lot of people prefer it actually isntea dof using the conn api
<OlegH> I do, for example. ;)
<eintopf> OlegH: okay, we currently filter everthing which is above 1280 MTU. So it would be interested what's now what's right... but we still can change it if we figured out what's right.
<eintopf> Most people says maximum 4-5 fragments, nothing more for 802.15.4 6lowpan networks
<eintopf> in practical use-case
<OlegH> oberstet, I would agree with cgundogan. Depends on what exactly your needs are, but it might be doable to use netapi/netreg and register for particular UDP packets yourself.
<OlegH> eintopf, I think it is not a problem to drop bigger datagrams. But I don't see a reason to implement an explicit dropping in RIOT/GNRC.
<OlegH> eintopf, I like the famous slogan from Jon Postel here.
<oberstet> with no IP fragmentation/reassembly support, having a callback/event based API to the <1280 payload of UDP datagrams is practically not really interesting =(
<OlegH> oberstet, true
<OlegH> But I don't know how complicated IP fragmentation would actually be.
<cgundogan> well, you would save the buffer copying
<oberstet> so what is the maximum, usable UDP datagram payload as of today in RIOT? it'll be <1280, but what exactly?
<OlegH> oberstet, with GNRC and 6LoWPAN, yes.
<oberstet> given there is no IP frag/reas.
<eintopf> what I would advice the RIOT people, share the implementation between 6lowpan/ipv6 reassembly. This is what we also do in Linux, there exists a fragmentation API.
<OlegH> I don't know about emb6 and LWIP and for Ethernet it would be probably 1500.
<OlegH> oberstet, how about you open an issue on Github with a feature request for IP fragmentation?
<oberstet> sure, will do
<oberstet> ok, thanks all for all the infos and the nice conversation;) I'll be around more often here I guess upcoming ..
<OlegH> cool
<cgundogan> keep us up-to-date with your autobahn traffic - eh - progress (:
<oberstet> ok;)
<cgundogan> I used to use the autobahn framework for java
<cgundogan> a long time ago
<oberstet> cgundogan: AutobahnAndroid? that needs more love. however, jawampa is good .. netty based
<cgundogan> yeah, could be AutobahnAndroid
-riot-notifier/#riot-os- [RIOT] haukepetersen pushed 3 new commits to master: https://github.com/RIOT-OS/RIOT/compare/25356dffc697...4ace70199e54
-riot-notifier/#riot-os- RIOT/master 9789eb2 Martine Lenders: Revert "doc: mandatory netif snip for gnrc_ipv6 receive()"...
-riot-notifier/#riot-os- RIOT/master d64b922 Martine Lenders: Revert "gnrc ipv6: replace check by assert"...
-riot-notifier/#riot-os- RIOT/master 4ace701 Hauke Petersen: Merge pull request #5326 from authmillenon/gnrc_ipv6/fix/revert5179...
<oberstet> FWIW, I expect AutobahnC to be "the last" Autobahn I write;) this is exciting, because it is supposed to allow us to take WAMP where no Linux/TCP can go .. smallish devices.
<oberstet> AutobahnC++ can go very small, but not MMU-less, non-TCP thingsHere are the most important takeaways from above discussion (as relevant to AutobahnC):
ericchapman
commented
8 years ago @oberstet Allot of information here. I read the entire conversation :). My take (please correct anything I misunderstood)
oberstet
commented
8 years ago So are we limited to 1280 or will we get 64K? Wasn't clear to me.
Not sure yet either. See my latest comments here: https://github.com/RIOT-OS/RIOT/issues/5371
oberstet
commented
8 years ago From my current understanding, it might very well be the case that on 802.15.4 networks, the practically usable maximum UDP datagram size is something like 500-1000 octets (regardless if there is support for IP frag/reas or not).
And if that would be true, than this effects our discussion with streaming API for WAMP args/kwargs .. because then we could have a DOM like thing - which is much easier/nicer for users. And also easier to implement probably.
ericchapman
commented
8 years ago @oberstet Got ya. Actually, in thinking about it, maybe that is the way to go, just limit the payload size in the library as a design decision, then everything can flow from that. That would simplify allot, and we can always use "progressive calls" to get larger payloads across the wire. That basically is fragmentation.
ericchapman
commented
8 years ago @oberstet Just saw your other message. We are on the same page :)
oberstet
commented
8 years ago @ericchapman same page .. cool;) Glad ... because, last time, we concluded we closed discussion of some design aspects, but above is reopening one - but there is no point in "closing eyes", better now another iteration than being sad later on;)
oberstet
commented
8 years ago Reordering/Retransmission should be possible but we will have to have a few buffers lying around to store the packets
That would be one option (coping with it above UDP, but below WAMP).
However, the other option would be: cope with it at the WAMP layer. This is what CoAP does. Eg with a WAMP RPC, there should be a timeout for the call not being answered inside the WAMP session state machine anyway. And when the timeout fires, the app gets an WAMP call error (not from the WAMP peer - the callee - but from inside the session state machine). And the app has to handle that (as it wishes) anyway: err, or do the call again. Or the state machine could automatically reissue the call.
Doing it at the WAMP protocol / session state machine level will require a few new WAMP message types though (which isn't a big issue, as we can do that "upwards" compatible), because: we don't have WAMP ACK-like message for each interactions.
Eg. when a CALL_RESULT is delivered to a peer by a router, that CALL_RESULT is assumed to arrive for sure (as up to now, WAMP has these assumption about a transport: https://github.com/wamp-proto/wamp-proto/blob/master/rfc/text/basic/bp_transports.md#transports). If we cope with unreliability at the WAMP level, then we'd need a CALL_RESULT_ACK message. When the router doesn't receive that in time, it'll retransmit the CALL_RESULT.
Now, ordering. This is what the WAMP spec guarantees at the app level: https://github.com/wamp-proto/wamp-proto/blob/master/rfc/text/basic/bp_ordering_guarantees.md
We could "relax" that. As in, if we have a (negotiated) maximum WAMP message size, this is already a limitation. And if we'd just say, alright, if you use this WAMP transport, ordering isn't as strict as with other WAMP transports, for many IoT apps, that could be practically still fine.
I haven't thought this through to the end, but these are my current thoughts.
This is all interconnected. I hope this isn't too overwhelming;)
I agree with GNRC and not worrying about posix. In too small of an environment to care about posix
Yep. Agreed.
I have done fragmentation/reassembly. You really need malloc to pull that off so we shouldn't touch it
Also agreed. Even thinking about doing full frag/reas. without malloc makes my brain hurt;)
oberstet
commented
8 years ago One more note on this: having a WAMP transport that comes with some limitations rgd WAMP - there is MQTT and there is MQTT-SN. The former requires TCP and doesn't run on RIOT (or other MCU like devices). The latter does indeed, but comes with limitations vs MQTT(-full).
ericchapman
commented
8 years ago This is all interconnected. I hope this isn't too overwhelming;)
Naaaa :).
rgd CALL_RESULT_ACK, lets think about this a day because this also implies we would need "SUBSCRIBED_ACK", "UNSUBSCRIBED_ACK", etc. Trying to think where to draw the line on "ACKing your ACKs".
This is normally why I take care of this in the transport layer because that is message agnostic. It just knows "I have packet 12 and 14, I am missing packet 13".
oberstet
commented
8 years ago Hehe;) For the record: I had zero doubts it would overwhelm you intellectually (who am I?) - just was afraid you'd see this "little" project spiraling in complexity, consuming significant time on your side. But hey, I did "warn" it would be non-trivial before;)
Yep, agreed. Lets think about the options a little. As with the API questions (user/OS facing), this has deep ramifications.
ericchapman
commented
8 years ago @oberstet Good news is once we get to where it is a coding exercise, it will be straight forward. This part is the hard part.
I've read more about RIOT, in particular networking.
This from the release announcement of RIOT 2015/09:
See: http://zolertia.io/blog/riot-5th-official-release-features-re-mote#sthash.FBlksRzR.dpuf
The GNRC network stack is described here.
There is an example (CoAP) that is using GNRC here.
The example "uses the GNRC network stack through RIOT's conn socket API.", in particular the GNRC UDP API, eg:
conn_udp_createconn_udp_recvfromconn_udp_sendtoThis example is interesting, as it could serve as a blueprint for what we could do with WAMP.
For example,
conn_udp_recvfromseems to require abuffer that will be filled with the complete UDP datagram payload, and app layer (in this case CoAP, but in ours WAMP), then parses this raw payload (into a CoAP request) and constructs a response in that very same buffer to send via
conn_udp_sendto.The example is also interesting as it uses RIOT's "threads" and message queues to wire up the different layers.
Now, as far as I get that, 6LoWPAN is an IPv6 (only) adaption layer that should allow IPv6 UDP with full, maximum payload (64k).