Can the server be notified on successful delivery of a notification?

[koalatux]

I would like an application server using aiocoap to be notified about the delivery of notifications (which are CON) of an observed resource to a client. Especially I would like to receive events for successful delivery.

With aiocoap 0.3 (currently in Debian stable) I was able to create a child class from ObservationExchangeMonitor and override the methods timeout, rst, cancelled and response and then create a custom method copied from ServerObservation.trigger(), which passes my custom ObservationExchange to sobs.original_protocol.send_message(). This worked fine for me.

With the latest version I am more out of luck, there is no more ObservationExchangeMonitor. I've seen there is is a messageerror_monitor in messagemanager.py, but it is only called in error cases but not the successful case.

Can you please give me any hints how I can achieve this? For a simplification, I need only one observation per resource to be handled. In the case it needs changes in aiocoap, I am also happy to provide patches if you point me in the right direction.

[chrysn]

Before we look into how it's doable stably in aiocoap, let's check first if this is something the protocol supports.

Notifications in CoAP are generally eventually-consistent. aiocoap may or may not do this right now (off my head I think it doesn't), but even if a notification were CON, the library could stop retransmitting a single notification once a later notification is available. Even if the library did not do that, any proxy inbetween would be allowed by the semantics of the Observe option to do the same.

What is the action you'd like to take on successful delivery? Is, given the above, observation and tracking what has been delivered an application design that can work conceptually?

[koalatux]

I might first explain my goals: What I intend to do might not be fully compatible with a RESTful approach. I would like to stream some data from the server to the client. The data I'd like to stream is not fully available in advance, also the total size of the data is not known in advance. Because of this I ruled out blockwise transfers. My client can only use UDP and TCP is not an option, CoAP is already used on the client for other things. The client is directly connected to the server without other network hops or proxies in between.

Upon successful delivery, the server application triggers a notification with the next chunk in the stream, this also solves the problem of notifications being replaced by new ones.

To make it conceptually work, we would have to extend CoAP, like in this proposed extension. But I was hoping to keep the protocol more simple.

[chrysn]

I don't think that you'll get the "keep it simple" this way, because you'll encounter components that rely on CoAP's REST properties all over the way. (Here it's aiocoap that's making things difficult, next it might be a different library, or all of a sudden you might find that you need a load balancing proxy but proxies would stop this).

But maybe we can find a way to make it work here in a way that's CoAP/REST compatible.

How do you expect your streaming protocol to react to networks that are slower than the stream? Should it jump at one point, or will the server buffer indefinitely? How do you intend to find out whether you're overloading the network, are you only sending CONs? How large are the pieces of data as they become available? How real-time does everything need to be?

(My gut feeling is that it should well be possible to do this with block-wise transfer, but which measures need to be taken to make it work might depend on some of the answers to the above).

[koalatux]

Yes, you're right, aiocoap might not be the only library involved in this or it might be replaced later.

The client is running on a device running Zephyr OS and the server is running on an an IoT-Gateway, which is an embedded Linux device. The client is connected to the server through a slow Sub-GHz link. The server is intended to forward data it receives from a TCP (or optionally even TLS) socket from the Internet to the client. The server can build up some back pressure by not reading from the TCP socket, the underlying OS will then handle flow control. Implementing flow control in the other direction is more difficult, but because the Internet connection is much faster than the Sub-GHz link, I'm less concerned about this.

We can assume, the server has lots of buffer space. Furthermore most limiting is the Sub-GHz link, we should avoid re-transmissions there whenever possible, thus we should immediately ACK packets received by the Sub-GHz link.

I intend to only send CONs and always wait to send the next chunk of data, when the previous one was ACKed. The initial goal is to handle chunks of data of maximum 256 Bytes, but that might change and generally it would be nice to be able to handle any kind of data stream. There are no real-time requirements, delays from some hundreds of milliseconds should be acceptable, more important is to detect a connection loss and inform the client application about it, so it can handle it accordingly.

(I think I'll look into the details of block-wise transfers again.)

[chrysn]

So just to check my understanding, layout is:

Zephyr device                         Linux device
CoAP client  --(constrained radio)--  CoAP server --(some TCP)-- something out of scope

You mention "the other direction" -- does this run bidirectionally? The constrained radio is probably 6LoWPAN? What is limiting the data transfer -- is it the constrained radio, or the processing in the Zephyr device?

All in all, yes, I think this would best be served by the CoAP client sending requests for more data. If you can tolerate chunking up to the block size (i.e. you don't need to get the next 3 bytes when they are there, because the next 300 could take some time), this can use blockwise transfer -- otherwise you'd need to go with something like STP. There is nothing in the spec that says that the next block needs to be available right away, so if the buffer ever does happen to underrun, things will just delay naturally. aiocoap will automatically send an ACK (preventing retransmissions, placing the initiative with the server again) when there is a delay of >0.1s.

How do you start things up? Thinking of cache validity and a second client here: A different device would start at block 0 again (unless it knows from somewhere else where to index into the stream), is that desirable?

(Generally, it might help if you told a bit more of the kind of stream you transmit, that'd answer some more questions more quickly and maybe precisely).

[koalatux]

A more complete layout is even more complicated, it looks like this:

  another MCU                          Zephyr device             custom device           Linux device
some application --(UART or CAN bus)--  CoAP client  --(radio)-- radio module  --(PPP)-- CoAP server --(some TCP + optionally TLS)-- server out of scope

Data which the Zephyr device has to forward comes from another MCU, which is connected via a CAN bus or UART to the Zephyr device. The sub-GHz link is not 6LoWPAN, it is something proprietary, but it is very similar, it also does IPv6 header compression (this protocol is the reason, I can't use TCP here :-/ ). On the Linux device, there is a custom radio module, it is connected via UART to the Linux SoC and forwards IP packets using PPP to the Linux system. The Linux device is connected via Wi-Fi and/or Ethernet to the Internet. The radio is mostly constrained due to regulatories, we are using the 868 MHz ISM band, and there the duty cycle is limited to a few percents. The Zephyr device is not that constrained, it has 1 MiB of Flash and 256 KiB of RAM.

Yes, I want to transfer data bidirectionally. My idea is the client sends data to the server using POST messages, which are defined not to be idempotent and should be handled as I intended with any CoAP implementation. My first idea was to also create a server on the Zephyr device for the other direction, but it seems also a bit odd to have two servers.

Actually I want the other MCU to be able to open a TCP connection to an Internet server without having to handle the complexity of TCP itself (and in the case of TLS, having the certificate validation done on the Linux device). Also retransmissions can be handled in a better way, when the Linux device acts as some kind of a proxy, because then the retransmissions will be done separately on each link. Because CoAP is already used on the Zephyr device and on the Linux device, my idea was to re-use it for the retransmission capabilities of CoAP instead of re-implementing everything from scratch using plain UDP.

I looked into block-wise transfers again, they would have been nice but the limitation of having to fill the block-size is kind of a bummer, because I can't always wait for a block to fill up, because sometimes there are no data transferred for some time.

There will be only one client per resource, another client would create a new resource, so there won't be any problems with caches.

The protocol within the stream is a custom request and response protocol with additional events one can subscribe to, but it is not RESTful, so I can't just map it to CoAP. And even if I could, I'd like to keep it abstracted, so the Zephyr device does not have to know about the inner workings of the protocol and only forwards data streams of any kind.

[chrysn]

My first idea was to also create a server on the Zephyr device for the other direction, but it seems also a bit odd to have two servers.

There's nothing odd about having both devices taking both roles in CoAP -- it is designed for that. Many CoAP applications switch role mid-protocol if it suits the needs (eg. Resource Directory in Simple Registration, or CoAP-EAP), and symmetric streams in two directions definitely do look like it's simple that way.

having to fill the block-size is kind of a bummer

I'd have suggested the STP link above, but given we're in a symmetrical situation and having a server on the Zephyr side is just for non-oddity, it seems to me like POSTing in the stream directions (as you envisioned originally) would be the best way to go.

---

when the Linux device acts as some kind of a proxy

Well actually (but here we might be swaying off what is practical right now), it'd be fully within what CoAP is designed to do to have the MCU already express its stream in CoAP (maybe it can be mapped, and now that mapping would be local to sensor MCU), which can be sent over UARTs when using slipmux, and both the Zephyr device and the Linux device could act as CoAP proxies, each doing a bit of spooling, and each taking the best possible care of retransmissions (the Linux-Server link could even be over CoAP-over-TCP/TLS). That'd keep application specifics off both the Zephyr and the Linux device, but I'm aware that there are a few gaps in this story that make it not immediately practical yet.

[chrysn]

I think this is resolved with the latest comments; please reopen if questions remain.

[koalatux]

Hi, yes, it works fine for now. Thanks a lot for your help.

The symmetrical design with POSTs in both directions works much better than my original design with the OBSERVE. I didn't consider back then that ACKs don't always have to be piggy-backed. With this new design acknowledging the transmission with the ACKs and signalling the MCU is ready to process new packets by responding to the POST gives us a simple flow-control without unnecessary retransmissions.