Client's Observe Notification not Synchronized

[aleparmi]

Hello everyone,

Is it correct to assume that an observation value will not be updated until the client triggers a registration update?

I am currently using Wakaama client where the battery value takes a random number every now and then. When the Leshan server triggers the observation it instantly synchronizes the device's battery value. Nevertheless, after a new battery value is available, the client sends a NON (non confirmable) but the server does not update the value field. Also Leshan 1.x seems to have same behavior.

Is it an issue on front-end or does the server refuse the observe notification on purpose? Is there any branch available where the client's notification (NON message) is let through?

Thank you in advance.

[sbernard31]

did your device IP address:port change between 2 communication (e.g. you device is behind a NAT) ?

If this is the case you should configure your device to work in queue mode and so make an update before to send your notification (because you address probably changed).

See :

• https://github.com/eclipse/leshan/wiki/LWM2M-Devices-with-Dynamic-IP
• https://github.com/eclipse/leshan/wiki/LWM2M-Observe

HTH

[aleparmi]

This is correct, my device is behind NAT.

As far as I can understand, there is no way to allow a notification through other than using an update request before (queue mode) or after it. Is this correct? And is the same true for UDP connections where we can just select the relaxed mode (https://github.com/eclipse/leshan/wiki/LWM2M-Observe#for-udp-without-security) or the token expedient allows to avoid the update?

[sbernard31]

As far as I can understand, there is no way to allow a notification through other than using an update request before (queue mode) or after it. Is this correct?

The update should be sent just before the notification when using queue mode. (If you have dynamic IP you MUST using queue mode) This is my understanding of the specification. This is not really a technical constraint. This means this is technically possible to not send the update request but you could have interoperability issue with other server than Leshan.

And is the same true for UDP connections where we can just select the relaxed mode (https://github.com/eclipse/leshan/wiki/LWM2M-Observe#for-udp-without-security) or the token expedient allows to avoid the update?

This is the same for UDP, the specification says the update is mandatory.

So either you send the Update request and you respect the specification. OR you use LeshanServerBuilder.setUpdateRegistrationOnNotification(true) + relaxed mode for UDP and you don't need the Update request but you can face interoperability issue.

Please let me know which part of the documentation is not clear in case I need to adapt it.

[aleparmi]

If I am not wrong the code for this part https://github.com/eclipse/leshan/wiki/LWM2M-Observe#for-udp-without-security changed meanwhile. Btw are Leshan and Wakaama clients supporting queue mode?

Thank you for the detailed explanation, it really helped me. I am currently measuring the power consumption of my device to analyze LwM2M's performances in comparison with other IoT protocols. Knowing this aspect will help me motivating better my research!

[sbernard31]

If I am not wrong the code for this part https://github.com/eclipse/leshan/wiki/LWM2M-Observe#for-udp-without-security changed meanwhile

I'm not sure I get you. :confused:

Btw are Leshan and Wakaama clients supporting queue mode?

At client side, queue mode is mainly stopping to communicate then when you want to communicate again :

• be sure you have a DTLS connection. (Depending of your network configuration and dtls feature available)
• do an update before to communicate.
• (If you need support of "Notification Storing When Disabled or Offline" this is a bit more complicated)

So as both Wakaama and Leshan are library which help to implement your own client, you should be able to do that, but not as simple as just set a QueueMode parameter to true... :grin:

For Wakaama, I'm not 100% sure. So better is to ask the question on the dedicated repository. (You can refer to this issue if you want) For Leshan, Some pieces are already in place but maybe there is missing one especially sending update before notification or "Notification Storing When Disabled or Offline" stuff. Maybe this is something I could try to add :thinking:, but let me know first exactly what you want to test.

Thank you for the detailed explanation, it really helped me. I am currently measuring the power consumption of my device to analyze LwM2M's performances in comparison with other IoT protocols. Knowing this aspect will help me motivating better my research!

Note that this will depends a lot of the network/dtls layer :

• Dynamic IP means more Update requests
• DTLS handshake consumption depends on used cipher.
• With DTLS, Dynamic IP means more handshake (except if you are using connection Id)
• DTLS handshake consumption depends if you do full or abbreviated handshake.
• I guess this can also depends if you are using "Notification Storing When Disabled or Offline".

So very hard to make a simple comparison.

By the way, I ask myself if device behind a NAT without DTLS (I mean using UDP only) is really a production use-case. (hard to me to see a case where this could be used safely ...)

Please do not hesitate to share your result. :pray:

[sbernard31]

I read again myself and I'm maybe not so clear.

There is 2 different points.

1. about dynamic IPs

When your IP addr:port changed the spec says you must send an update. (generally with dynamic IP you are using queue mode and so you must send an update on wake-up)

2. about CoAP observation with dynamic IPs

For UDP, there is no way to respect the CoAP RFC but californium offer relaxed mode (out of the spec) For DTLS, same DTLS connection must be used so only connection id allow to respect the CoAP RFC but Leshan by default allow same identity instead of same connection.

---

Again I think all of this is better explain in :

• https://github.com/eclipse/leshan/wiki/LWM2M-Devices-with-Dynamic-IP
• https://github.com/eclipse/leshan/wiki/LWM2M-Observe

If you give me exact use cases, for each of them I can provide you :

• expected behavior which strictly follow all specifications.
• the optimized way with Leshan but probably out of specification.

[aleparmi]

Hello Simon,

I'm not sure I get you. 😕

Is it possible that the following part of the doc is not anymore in line with the actual code, or am I wrong? (https://github.com/eclipse/leshan/wiki/LWM2M-Observe#for-udp-without-security)

coapConfig = LeshanServerBuilder.createDefaultNetworkConfig();
coapConfig.set(NetworkConfig.Keys.RESPONSE_MATCHING, MatcherMode.RELAXED);
builder.setCoapConfig(coapConfig);

For Wakaama, I'm not 100% sure. So better is to ask the question on the dedicated repository. (You can refer to this issue if you want) For Leshan, Some pieces are already in place but maybe there is missing one especially sending update before notification or "Notification Storing When Disabled or Offline" stuff. Maybe this is something I could try to add 🤔, but let me know first exactly what you want to test.

One of my goal at the moment is to compare the MQTT publish with the LwM2M notification (done via a CoAP NON Post operation). Therefore, if I could notice it correctly, the device posts the message and it actually gets to destination. The fact that the server does not recognize it is out of my scope at the moment. Also sending an update before the notification would somehow influence badly the power performance compared to a single MQTT publish. So I think I will utilize the clients as they are for now. Thank you anyways for your offer 😉

Note that this will depends a lot of the network/dtls layer :

• Dynamic IP means more Update requests
• DTLS handshake consumption depends on used cipher.
• With DTLS, Dynamic IP means more handshake (except if you are using connection Id)
• DTLS handshake consumption depends if you do full or abbreviated handshake.
• I guess this can also depends if you are using "Notification Storing When Disabled or Offline".

So very hard to make a simple comparison.

By the way, I ask myself if device behind a NAT without DTLS (I mean using UDP only) is really a production use-case. (hard to me to see a case where this could be used safely ...)

You made fair points here, I guess a perfect comparison will never be possible (even more when analyzing MQTT and LwM2M which are very different concepts). I think that NAT without DTLS could hardly be a production use-case too.

If you give me exact use cases, for each of them I can provide you :

• expected behavior which strictly follow all specifications.
• the optimized way with Leshan but probably out of specification.

At the moment I am measuring in NB-IoT and LTE-M:

• LwM2M with NoSec
• LwM2M with DTLS-PSK
• LwM2M with DTLS-Certificates

This scenario in different phases:

• connection
• periodical notifications
• steady-state with periodical updates

Thank you again for your great help!

[sbernard31]

Is it possible that the following part of the doc is not anymore in line with the actual code, or am I wrong?

AFIAK this part of the documentation is up to date. Do you have something specific in mind?

One of my goal at the moment is to compare the MQTT publish with the LwM2M notification.

It's pretty hard to compare MQTT to LWM2M as there are not really at same level. See https://github.com/eclipse/leshan/wiki/F.A.Q.#what-is-the-difference-between-mqtt-and-lwm2m-

(done via a CoAP NON Post operation).

I'm not sure to get you. Notifications are not POST. Notifications are responses to a previous GET request with observe option.

Maybe you should have a look to LWM2M v1.1 Send Operation. This is maybe more comparable to a MQTT publish ? :thinking:

At the moment I am measuring in NB-IoT and LTE-M:

LwM2M with NoSec LwM2M with DTLS-PSK LwM2M with DTLS-Certificates This scenario in different phases: connection periodical notifications steady-state with periodical updates

And device have dynamic IP ? No constraints about feature you can use at DTLS level (cipher, abbreviated handshake, connection id) ?

[sbernard31]

Also Note that an MQTT publish (even with a QoS=0) is not so comparable to a NON CoAP message over UDP(and so DTLS) as with UDP message can be lost.

[boaks]

One of my goal at the moment is to compare the MQTT publish with the LwM2M notification

Just go for coap CON-POST requests using DTLS CID. Nothing compares with that efficiency. In both, bandwidth and energy consumption. Especially for mobile devices, which are moving around and may move through regions with bad connectivity.

[sbernard31]

Just go for coap CON-POST requests using DTLS CID.

So equivalent in LWM2M will be Send Operation using DTLS CID. (As said before, for dynamic IP if you want to strictly respect the LWM2M specification you also need to send an Update request)

[sbernard31]

Here a discussion which could interest you : https://github.com/OpenMobileAlliance/OMA_LwM2M_for_Developers/issues/293

[aleparmi]

AFIAK this part of the documentation is up to date. Do you have something specific in mind?

I am probably wrong, could you show me where to insert this code's snippet?

It's pretty hard to compare MQTT to LWM2M as there are not really at same level. See https://github.com/eclipse/leshan/wiki/F.A.Q.#what-is-the-difference-between-mqtt-and-lwm2m-

Yes, the differences are enormous, but the work includes an overall evaluation and one part consists in a high-level power consumption analysis to determine (and hopefully confirm) the benefits of using LwM2M in constrained devices in LPWAN.

I'm not sure to get you. Notifications are not POST. Notifications are responses to a previous GET request with observe option.

Maybe you should have a look to LWM2M v1.1 Send Operation. This is maybe more comparable to a MQTT publish ? 🤔

You are right it is not a POST. But I see very little difference in the packet itself compared to a GET Response. In the end what it counts is the effective and somehow typical usage of a LwM2M device. I see a SEND as a particular case. Correct me if I am wrong.

And device have dynamic IP ? No constraints about feature you can use at DTLS level (cipher, abbreviated handshake, connection id) ?

Not dynamic IP but port is for sure changing. I guess the DTLS capabilities are dependent to the client I am using: in my case Wakaama or Leshan test clients (DTLS 1.2 ?) and I'm not completely sure if they support CID 🤨

Also Note that an MQTT publish (even with a QoS=0) is not so comparable to a NON CoAP message over UDP(and so DTLS) as with UDP message can be lost.

Absolutely, another point of my work is to evaluate a TCP based IoT protocol vs UDP. Therefore, pro and cons are also relevant. Trade-off between packet losses and low energy consumption is one topic of discussion within my analysis. On one side I am also happy that the two are not exactly the same.

[aleparmi]

Just go for coap CON-POST requests using DTLS CID. Nothing compares with that efficiency. In both, bandwidth and energy consumption. Especially for mobile devices, which are moving around and may move through regions with bad connectivity.

True, this especially important in NB-IoT where the network is very sensitive to devices' mobility. In LTE-M a NON-POST may still be acceptable but I have not tried it yet...

[boaks]

Trade-off between packet losses and low energy consumption is one topic

Therefore CoAP offers CON. Just chose, what you really want, and not the wrong and later blame that.

The "reliability" of TCP is based on IP-retransmission, done by the TCP layer. For CoAP, just use CON messages, that will retransmit the IP-messages as well. You may also use CON for notifies, if your application really requires that. The NON notifies are designed by the intention to lose them sporadically (tradeoff for frequently changing states). In my experiments the loss is about 2-4%. Only, if some components get overloaded, then that may get larger ... but then you may be even more surprised by the outcome of using TCP :-).

In LTE-M a NON-POST may still be acceptable but I have not tried it yet...

When a NON leaves the "mobile network layer" and is transmitted through a IP network, then it may get lost as every UDP message.

[sbernard31]

I am probably wrong, could you show me where to insert this code's snippet?

Just where you create your LeshanServer (where you handle your LeshanServerBuilder) If you are using leshan-server-demo, no need to modify the code. Just edit the Californium.properties file by setting RESPONSE_MATCHING to RELAXED.

I see a SEND as a particular case. Correct me if I am wrong.

I'm not sure to get what you mean :thinking: but yeah Observe is probably more used than Send because Send is newer but in the other hand Send was created because of issues with Observe.

I'm not completely sure if they support CID

Leshan demos can be easily adapted to support CID, as this is already implemented in Californium/Scandium. I should add a command line option for this and even maybe activated by default :thinking:

I know some people succeed to use wakaama with CID but AFAIK this is not available out of the box with wakaama demos.

[sbernard31]

If I understand you correctly you are measuring power consumption. Protocol could impact it but also implementation itself.

Note that Leshan Client Library wasn't really design with limiting battery consumption in mind and leshan-client-demo even more. I guess currently the client library is mainly used to write tests or simulator and leshan-client-demo as toy for testing interoperability.

[sbernard31]

I created a branch cid with support of CID for demos : #999 (Not so much tested, I will finalize it when I'm back on Monday)

[aleparmi]

In my experiments the loss is about 2-4%. Only, if some components get overloaded, then that may get larger ... but then you may be even more surprised by the outcome of using TCP :-).

Cool findings, I also want to analyze the packet loss. For TCP I'd expect an high packet loss rate in mobility use-cases especially in NB-IoT. Let's see

[aleparmi]

Note that Leshan Client Library wasn't really design with limiting battery consumption in mind and leshan-client-demo even more. I guess currently the client library is mainly used to write tests or simulator and leshan-client-demo as toy for testing interoperability.

I see, but do you think it could impact that bad on the consumptions? In the end my comparison will take in account Paho for MQTT. Do you think that they put more effort into power consumptions aspects?

I created a branch cid with support of CID for demos : #999 (Not so much tested, I will finalize it when I'm back on Monday)

Really cool, thank you :)

[aleparmi]

During my first tests I came across an interesting behavior in the dtls handshake. Looking at the Wireshark captures the Leshan Demo Client (after the Server Hello) sends in one single packet key exchange, cipher spec and encrypted handshake message. This solution seems to cause a bit of overhead on my raspberry. Looking a the Wakaama demo client using equal cipher, authentication method and dtls version, the same content is transferred in three different packets. This seems to cause less overhead. In this case though the client registration takes much longer (see application data).

Does the Leshan Demo Client offer the possibility to split key exchange, cipher spec and encrypted handshake in three different packets?

I'm currently using LTE-M. Are these topics related to these links?

• https://github.com/eclipse/leshan/wiki/Using-CoAP-Block-Wise-to-transfer-large-file-%3F
• https://github.com/eclipse/leshan/wiki/Request-Timeout

Leshan Wakaama

[boaks]

In DTLS 1.2 you have three layers:

• UDP-Layer
• Record-Layer, a couple of records may be sent together in one UDP package (that's Californiums default). But you may send them also in single UDP packages.
• Handshake-Message-Layer, a handshake message may be split into several fragments fit each into one record, or place even multiple handshake-messages into one record.

What is most efficient depends more on the assumed PMTU, the idea is to use the DTLS fragmentation and retransmission (handshake only) in order to overcome the UDP specific transport.

This solution seems to cause a bit of overhead on my raspberry.

I'm wondering, which overhead? Maybe that's is a mislead assumption focused on the application code. If you consider the UDP/IP stack execution code as well, then this may change. At least in my experience focusing on the send UDP package is the only thing, which pays-off. Per default Californium assembles therefore a couple of dtls-records into one udp-package. That depends on the configured MTU (default is the MTU of the network interface, so the local MTU is assumed to be a good guess for the path MTU).

[boaks]

See DtlsConnectorConfig.setEnableMultiRecordMessages and DtlsConnectorConfig.setEnableMultiHandshakeMessageRecords.

I don't think, changing the default will have positive effects. See getters for default descriptions.

[boaks]

In this case though the client registration takes much longer (see application data).

It's unclear, in which case the registration takes much longer. Or even what you mean by registration in the domain of dtls and lwm2m.

According the wireshark log, Californium takes 0.7s the send the first application record. Wakaama 1.5s. In both cases, the server answer is similar fast, about 1-2ms.

So, what do you consider as "registration"? The dtls handshake? Then Californium's approach seems to be faster. The lwm2m registration? I would guess, the difference will turn into "nothing", if you repeat the test a couple of 100s times. Even if not, then you buy 1ms lwm2m-registration with 800ms dtls handshake. Not a that good deal.

[aleparmi]

Notice that the captures were recorder on server-side

I'm wondering, which overhead?

From those captures I notice that there is a bigger time discrepancy (400 ms) than in Wakaama (150 ms) between the Server Hello and the combined dtls handshake (Leshan packet number 5). I was connecting this delay to a possible Raspberry's ram overhead in handling a bigger packet at once.

So, what do you consider as "registration"?

LwM2M registration. Doesn't it correspond the exchanged application data after the handshake? In Wakaama the registration starts after 1.2 s following the dtls handshake. In Leshan just after 200 ms (see above).

[aleparmi]

See DtlsConnectorConfig.setEnableMultiRecordMessages and DtlsConnectorConfig.setEnableMultiHandshakeMessageRecords.

I don't think, changing the default will have positive effects. See getters for default descriptions.

After inserting these lines in LeshanClientDemo.java

dtlsConfig.setEnableMultiHandshakeMessageRecords(false);
dtlsConfig.setEnableMultiRecordMessages(false);

I got 6/10 tests showing an equal result to the one above, 2/10 better performances (~100 ms) and 2/10 worse (~100 ms). I guess, nothing really meaningful

[boaks]

Is the wakaama client also running on a raspberry pi?

Then you may require more then just 10x times oversampling in order to mitigate the runtime varieties. And sure, these are different implementations with different domains. Californium is mainly implemented as server side. Therefore it utilizes machines with multiple CPU cores. It's able to handle many request simultaneously. C-libraries are usually intended to run a single client, and with NSTART-1, requests are not processed simultaneously. That has also slightly influence on the times, but in the end, the differences are not that big.

I got 6/10 tests showing an equal result to the one above, 2/10 better performances (~100 ms) and 2/10 worse (~100 ms). I guess, nothing really meaningful

Just to mention: the modem and CAT-M1 (or CAT-NB) also shows some timing varieties. In my experiments with CAT-NB from "deep sleep", it usually takes 4s for one request/response (using DTLS CID). But in same cases it also takes up to more than 60s to even get a air-slot.

[boaks]

One general thing to mention:

There may be a difference between efficiency and total power consumption. The one considers, how much power is required, when data is sent, while the other also consider, what is required, if no data is transferred. The difference is somehow the ratio between "preparing" and "sending application data". If "application data" is sent frequently, the "preparation" may be only once and so "negligible". If it is intended to sent "application data" "occasionally", then a 2k TLS handshake may be significant for 1k application data.

[sbernard31]

I see, but do you think it could impact that bad on the consumptions? In the end my comparison will take in account Paho for MQTT. Do you think that they put more effort into power consumptions aspects?

No idea. Just prefer to warm you that AFAIK Leshan client is mainly used as a simulator (but maybe it has good performance anyway, I didn't get so many feedback about it)

I'm currently using LTE-M. Are these topics related to these links?

• https://github.com/eclipse/leshan/wiki/Using-CoAP-Block-Wise-to-transfer-large-file-%3F
• https://github.com/eclipse/leshan/wiki/Request-Timeout

This page was writting with "classic" UDP in mind, I didn't know so much LTE-M... so I'm not sure if this is directly applicable. But reading the https://github.com/eclipse/leshan/wiki/Using-CoAP-Block-Wise-to-transfer-large-file-%3F I think the documentation is not up to date. (I need to modify it)

[aleparmi]

Is the wakaama client also running on a raspberry pi?

Correct

Just to mention: the modem and CAT-M1 (or CAT-NB) also shows some timing varieties. In my experiments with CAT-NB from "deep sleep", it usually takes 4s for one request/response (using DTLS CID). But in same cases it also takes up to more than 60s to even get a air-slot.

Yes, PSM may completely change the results, specially in NB-IoT. Im my thesis I will probably not take it into account since I already have a lot of different scenarios to analyze.

There may be a difference between efficiency and total power consumption. The one considers, how much power is required, when data is sent, while the other also consider, what is required, if no data is transferred. The difference is somehow the ratio between "preparing" and "sending application data". If "application data" is sent frequently, the "preparation" may be only once and so "negligible". If it is intended to sent "application data" "occasionally", then a 2k TLS handshake may be significant for 1k application data.

If you do not mind, I will take this wise consideration into my work. I guess the CID plays a big role in keeping the preparation part as tiny as possible.

I created a branch cid with support of CID for demos

I have played with it around and it seems to work. My only perplexity is understanding the usage:

• What would be the benefit of having a big CID size?
• Does using the 0 value mean that the server will generate the CID for you?

[boaks]

If you do not mind, I will take this wise consideration into my work. I guess the CID plays a big role in keeping the preparation part as tiny as possible.

Sure, and yes, the CID minimizes the "preparation" by obsoleting many dtls handshakes.

hat would be the benefit of having a big CID size?

Depends, on which data you will put into it. Some may use that for low-level message routing (e.g. dtls-cid baes load balancer, see Built-in Support for DTLS Connection ID Cluster using basic UDP-Load-Balancers). But usually something as 8 or 10 bytes fits well.

Does using the 0 value mean that the server will generate the CID for you?

The server always generates the CID, which the client is intended to use. With 0 the client indicates, that the server is not intended to send dtls-records with CID, but may assign a CID to the client in order to enable the client to send dtls-records with a CID.

[sbernard31]

I have played with it around and it seems to work. My only perplexity is understanding the usage:

@aleparmi, Could we discuss about it at #999 ? @boaks you're welcomed too.

[aleparmi]

I have another question regarding my research. This time is about DTLS handshake dealing with x.509 certificate mode. There is a double Server Hello Packet with an interval of almost exactly 1 s between them. It is a constant behavior (not only in cIoT) that I would not know how to explain. Is it a retransmission because the client is not able to handle the request in a give time?

1   0.000000000 Client Server DTLSv1.2  156 Client Hello
2   0.001591307 Server Client DTLSv1.2  102 Hello Verify Request
3   0.158051217 Client Server DTLSv1.2  188 Client Hello
4   0.163377760 Server Client DTLSv1.2  1023    Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
5   1.163765102 Server Client DTLSv1.2  1023    Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
6   1.484575702 Client Server DTLSv1.2  677 Certificate
7   1.511484386 Client Server DTLSv1.2  133 Client Key Exchange
8   1.511585011 Client Server DTLSv1.2  141 Certificate Verify
9   1.518347905 Client Server DTLSv1.2  60  Change Cipher Spec
10  1.528000471 Client Server DTLSv1.2  110 Connection ID
11  1.530784217 Server Client DTLSv1.2  117 Change Cipher Spec, Encrypted Handshake Message
12  1.687437229 Client Server DTLSv1.2  231 Connection ID
13  1.689498387 Server Client DTLSv1.2  105 Application Data

Capture from Server-Side

[boaks]

Yes, looks like.

With 2.6.2, you may use DtlsConnectorConfig.setRetransmissionTimeout(int timeout)

With 3.0, you may use DtlsConnectorConfig.setAdditionalTimeoutForEcc(int timeout) as alternative.

[boaks]

@sbernard31

Is the dtls retransmission timeout of 1s (according RFC6347) intended? For "embedded cloudy" scenarios, I relax that to 2s. The recommendation of RFC7925

An initial timer value of 9 seconds with exponential back off up to no less then 60 seconds is therefore RECOMMENDED.

seems for me to be too relaxed.

[sbernard31]

Is the dtls retransmission timeout of 1s (according RFC6347) intended?

We are using the default value and I never think about changing this until now.

Thinking about it, I don't know if we should change this default value in Leshan (library and/or demos). Increasing the timeout value could make the handshake longer and so keep the device longer awake. I'm not sure there is a "right" value, I feel this depends too much of use cases. So I prefer to use a value defined in one of the RFCs. So 1 seconds or 9 seconds. As I also feel that 9 seconds sounds a too relaxed value, I guess staying with default Californium/RFC6347 value is not so bad.

Is it a retransmission because the client is not able to handle the request in a give time?

I guess yes. In my experience, this can happen with too costly ciphersuite. Which one are you using ? Do you try with TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 ?

[boaks]

Increasing the timeout value could make the handshake longer and so keep the device longer awake. I'm not sure there is a "right" value, I feel this depends too much of use cases.

Sure, and in an "embedded iot cloud" case, I would go for 2s :-). First, because only 2% of the messages seems to get dropped under "normal conditions", so the negative effect is overall not that large. Second, with that, and the experience report here above, using 2s will have advantages (with the 3.0, that larger value could be limit to ECC).

So I prefer to use a value defined in one of the RFCs. So 1 seconds or 9 seconds.

Just to mention, the default in Californium for the CoAP ACK timeout is 2s (at least since 2017).

I guess yes. In my experience, this can happen with too costly ciphersuite.

This time is about DTLS handshake dealing with x.509 certificate mode.

I would guess, it's about x509 (or RPK), so the "TLS_ECDHE_ECDSAWITH???". But the other details should not be that additional time consuming (e.g. AES_128_CCM_8 or AES_128_GCM).

@aleparmi

If you want to optimize it, you may try out a larger dtls retransmission timeout.

[sbernard31]

Just to mention, the default in Californium for the CoAP ACK timeout is 2s (at least since 2017).

This is default value defined in RFC 7252. I don't get your point here :thinking: Or maybe you mean that a consistent config should have same value for DTLS Retransmission and CoAP ACK_TIMEOUT ? In this case this is also true for Californium default value ?

[boaks]

This is default value defined in RFC 7252. I don't get your point here

Uups, you're right. My fault.

[sbernard31]

But the other details should not be that additional time consuming (e.g. AES_128_CCM_8 or AES_128_GCM).

I didn't test it so it just something I read (I can remember where) but the idea was that CCM could be faster on constraint devices. (but maybe this is insignificant comparing to ECDHE_ECDSA cost)

Uups, you're right. My fault.

Anyway do you think those parameters should be set with same value ? maybe it makes sense ?

[boaks]

I didn't test it so it just something I read (I can remember where) but the idea was that CCM could be faster on constraint devices. (but maybe this is insignificant comparing to ECDHE_ECDSA cost)

My consideration was the the time for the server flight including the certificates. Just for that flight, I don't consider that there will be a difference in AES_128_CCM_8 or AES_128_GCM.

Anyway do you think those parameters should be set with same value ? maybe it makes sense ?

That's at least my impression from the experience collected over the last years. May be a good "point in time" to do that with the next 3.0.0-M.

[aleparmi]

I guess yes. In my experience, this can happen with too costly ciphersuite. Which one are you using ?

Until now: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)

Do you try with TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 ?

After a few tests I was not able to find big differences in latency. The client still responds between 1 and 2 seconds, causing the server retransmission.

On server side I extended the timeout to 2 seconds, just to spare bytes over the air and power consumption on client side.

Having also analyzed how MQTT works with TLS, I am starting to notice how the raspberry pi handles better OpenSSL than Californium. With PSK the difference was not that big but certificates really challenge the device: the response takes much longer with DTLS than with TLS.

Californium is mainly implemented as server side. Therefore it utilizes machines with multiple CPU cores. It's able to handle many request simultaneously

Is it for this reason?

[boaks]

(D)TLS and certificates may surprise. Please provide your wireshark logs :-).

[boaks]

Therefore it utilizes machines with multiple CPU cores. It's able to handle many request simultaneously

Is it for this reason?

I guess, not.

May be the handshake chosen by MQTT isn't the same, maybe openssl is somehow faster than java (JCE). For MQTT, what do you use on client and server side?

If you use only CoAP (instead of LwM2m), you may use libcoap with different DTLS bindings, e.g. openssl as well.

[aleparmi]

(D)TLS and certificates may surprise. Please provide your wireshark logs :-).

LwM2M DLTS Cert with TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (DTLS Handshake + LwM2M Registration)

1   0.000000000 Client Server DTLSv1.2  156 Client Hello
2   0.000691275 Server Client DTLSv1.2  102 Hello Verify Request
3   0.158375445 Client Server DTLSv1.2  188 Client Hello
4   0.163616829 Server Client DTLSv1.2  1025    Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
5   1.164013773 Server Client DTLSv1.2  1025    Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
6   1.484569704 Client Server DTLSv1.2  677 Certificate
7   1.511500076 Client Server DTLSv1.2  133 Client Key Exchange
8   1.519142269 Client Server DTLSv1.2  142 Certificate Verify
9   1.528726965 Client Server DTLSv1.2  60  Change Cipher Spec
10  1.538148418 Client Server DTLSv1.2  110 Connection ID
11  1.539499308 Server Client DTLSv1.2  117 Change Cipher Spec, Encrypted Handshake Message
12  1.678636358 Server Server DTLSv1.2  231 Connection ID
13  1.682296030 Server Server DTLSv1.2  105 Application Data

MQTT TLS Cert with TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (TLS Handshake + MQTT Connect)

1   0.000000000 Client Server TCP   74  51295 → 8884 [SYN] Seq=0 Win=64240 Len=0 MSS=1360 SACK_PERM=1 TSval=2902666670 TSecr=0 WS=128
2   0.000041345 Server Client TCP   74  8884 → 51295 [SYN, ACK] Seq=0 Ack=1 Win=62643 Len=0 MSS=8961 SACK_PERM=1 TSval=446559665 TSecr=2902666670 WS=128
3   0.142227704 Client Server TCP   66  51295 → 8884 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=2902666916 TSecr=446559665
4   0.180494019 Client Server TLSv1.2   317 Client Hello
5   0.180539608 Server Client TCP   66  8884 → 51295 [ACK] Seq=1 Ack=252 Win=62464 Len=0 TSval=446559846 TSecr=2902666937
6   0.180908767 Server Client TLSv1.2   1414    Server Hello, Certificate, Server Key Exchange
7   0.180911579 Server Client TLSv1.2   122 Certificate Request, Server Hello Done
8   0.315071842 Client Server TCP   78  [TCP Dup ACK 3#1] 51295 → 8884 [ACK] Seq=252 Ack=1 Win=64256 Len=0 TSval=2902667119 TSecr=446559846 SLE=1349 SRE=1405
9   0.361722519 Server Client TCP   1414    [TCP Retransmission] 8884 → 51295 [ACK] Seq=1 Ack=252 Win=62464 Len=1348 TSval=446560027 TSecr=2902667119
10  0.547792151 Client  Server TCP  66  51295 → 8884 [ACK] Seq=252 Ack=1405 Win=64128 Len=0 TSval=2902667348 TSecr=446559846
11  0.850783089 Client Server TLSv1.2   1389    Certificate, Client Key Exchange, Certificate Verify, Change Cipher Spec, Encrypted Handshake Message
12  0.850832175 Server Client TCP   66  8884 → 51295 [ACK] Seq=1405 Ack=1575 Win=61184 Len=0 TSval=446560516 TSecr=2902667366
13  0.851502876 Server Client TLSv1.2   836 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message
14  0.861866347 Client Server TCP   78  [TCP Dup ACK 10#1] 51295 → 8884 [ACK] Seq=1575 Ack=1405 Win=64128 Len=0 TSval=2902667668 TSecr=446559846 SLE=1 SRE=1349
15  1.107590259 Client  Server TCP  66  51295 → 8884 [ACK] Seq=1575 Ack=2175 Win=64128 Len=0 TSval=2902667904 TSecr=446560517
16  1.126201024 Client Server TLSv1.2   132 Application Data
17  1.126227045 Server Client TCP   66  8884 → 51295 [ACK] Seq=2175 Ack=1641 Win=61184 Len=0 TSval=446560791 TSecr=2902667904
18  1.126342042 Server Client TLSv1.2   99  Application Data
19  1.262155018 Client Server TCP   66  51295 → 8884 [ACK] Seq=1641 Ack=2208 Win=64128 Len=0 TSval=2902668056 TSecr=446560792

To me it suprises the time difference between the Server Hello and Client certificate (packet 6 for LwM2M and 11 for MQTT) Reponse -> much lower in TLS. I am using the same Raspberry for both cases.

For MQTT I am using mosquitto for both client and broker

[boaks]

OK, same cipher suite, and same authentication. I guess also same or similar certificates.

Your much is about 400ms.

(And still with a retransmission caused by the 1s timeout.)

I considered the difference to be larger. A really larger difference may be caused e.g. by different authentication modes (e.g. anonymous client).

But 400ms is something not too bad :-). Please consider, that later you need also to exchange application data. And with CID, this 400ms extra will fast and easily pay off, when TLS requires a new handshake, but DTLS CID not.

[aleparmi]

Yes also similar certificates, x.509. Yes absolutely, CID is already showing positive effects in the steady-state comparison (periodical connection update without data exchange).

Would a library like mbedTLS perform better on client-side?

[sbernard31]

I'm not sure if this could be significant but often performance tests for java used warm-up run before measurement. If you do several handshakes, do you have same result for 1st handshake and next ones ? (you can use -f -r option to force a full handshake on update and -cp to choose time between 2 updates)

[sbernard31]

Would a library like mbedTLS perform better on client-side?

I guess, it should.

[boaks]

A raspberry PI is not really a "constrained" device. So, usually, if a embedded mcu is used, ecc ends up at a couple of seconds (5-10s). Only, if the mcu support ECC (or RSA) with hardware, or extra ECCc-hw is used, then the times will get fast again. So your results are not that characteristic.

On the cloud-server side usually also much faster CPUs are used as on a raspberry. In the future I hope, that I can support also some HSM, but more for keeping the private key really private, than for speed.

Some java deployment use openssl with a wrapper instead of the JCE. I'm not sure, if that pays off. At least for the current usage numbers, it seems to fit for me. Once, if at all, that technique gets much more used, then I think using Ed25519/X25519 with a fast GPU support will beat a lot of other stuff in the field :-).