Implement URC Handling. (IDFGH-8812)

[diplfranzhoepfinger]

also discussed in:

156

168

179

[diplfranzhoepfinger]

@david-cermak

[franz-ms-muc]

also in the Modem Console URC might be helpful: see here:

issuing a extra cmd AT does not really look smart.

[david-cermak]

This could be implemented the same way as https://github.com/espressif/esp-protocols/pull/156, i.e. installing a permanent callback which would check for a global set of URC first and only after that for replies to requests. So we could still create a DCE class that's command-able (=could add commands) and at the same time handles the URCs.

I'll prepare a simple demo.

[franz-ms-muc]

Thanks so much.

Meisterschulen am Ostbahnhof Mühldorfstr. 6 81671 München

Tel.: 089 416002 - 0 Fax: 089 416002 - 111

Internet: www.meisterschulen-münchen.dehttp://www.meisterschulen-mchn.de

[franz-ms-muc]

I'll prepare a simple demo.

any Progress @david-cermak ?

[david-cermak]

Added as https://github.com/espressif/esp-protocols/pull/156/commits/c61fe1f5f915b2d39dda81ea8ce701660991194c

it's very simple and WIP. It only adds a user callback that could handle all received data. (needs to be removed before switching modes and applied again after)

The other option is to reuse CMUX mode for that, we already have two virtual terminals, so we could make another one for handling URC. Unfortunately the DTE/CMUX classes are not fully designed to support variable number of virtual terminals, but that would be an interesting addition.

[diplfranzhoepfinger]

i will test it. have a Good Test-case here:

a SIMCOM A7672E-FASE

we can send here a AT+CGNSSPWR=1 command, and it will answer with a URC of +CGNSSPWR: READY!

will test now...

[diplfranzhoepfinger]

your URC Handler works:

[franz-ms-muc]

Hi,

Question: (from Vasil Nikolov)

I understand the URC implementation following way:

1. You can switch on/off the reaction on URC a. This is nice
2. ESP Modem receives all of the URC a. For me it is important to understand how modem implementation differentiates between URCs and a normal response to AT commands i. There are commands which require longer time until a response is received. b. Can we receive URCs while we are waiting for a response to a certain AT Command? i. If yes then how can we be sure that responses to the AT Commands are forwarded to the URC handler instead to the command handler
3. In our user code we shall define one URC handler with a switch case based on the data and length of received URC a. This is a nice approach, we have full control over all URCs we are interested in

[VasilNikolovRilabs]

@david-cermak could you please provide some feedback about the last questions from Franz? We would like to really assure that the URC implementation wont impact the normal AT command request/response handling

Thank you!

[david-cermak]

Hi Franz and Vasil,

sorry for not responding earlier. The example of injecting a URC callback doesn't really change anything in the esp_modem layers, it just overrides the default command implementation using these two methods:

1) Traditional method for sending commands makes the class command-able, so it could be employed by the command library to run all/defined AT commands:

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L51

2) The data callback that processes the replies

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L70

This way, we're able to

• first call the user urc-callback (and after that)
• continue with processing the replies by the command library

---

a. For me it is important to understand how modem implementation differentiates between URCs and a normal response to AT commands

we process the user callback first

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L73-L75

...and then let the command library to parse the same data.

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L76-L87

i. There are commands which require longer time until a response is received.

You need to be aware that the urc handler would still received all the replies, including the ones that belong to unrelated commands issued by the command library (by user)

b. Can we receive URCs while we are waiting for a response to a certain AT Command?

Yes, the urc-handler gets always called (if enabled). The other part (lib-command processing) is called only if an AT command is in progress.

i. If yes then how can we be sure that responses to the AT Commands are forwarded to the URC handler instead to the command handler

This works like a fork (tee command in bash :-), we pass the same data to two handlers

The only trouble might be the time, physically spent on processing the urc responses, since it actually delays processing the simultaneously active AT command. This wouldn't delay data reception, but might influence timeouts of the defined commands. (for example, if we're sending AT+CGC\r with timeout of 500ms and spend 600ms on urc processing, then it would fail, but should still work if we spend like 450ms on urc-processing, as the actual reply would simply queue so the data handler should make it in time)

[VasilNikolovRilabs]

Hello David,

thank you for the detailed explanation, what you really describe is a very good approach for the URC and it really makes sure that the command handling will not be broken. As last takeout from your notes – we need to make sure that the URC handler is a very short and thus we will not impact the timeout handling of the AT commands.

Thank you again for the support!

Best Regards, Vasil Nikolov CEO and CO-Founder

Email: @.**@.> Phone: +359 888 200510

Banat 1 1407 Sofia Bulgaria

@.***

www.rilabs.iohttp://www.rilabs.io/

From: david-cermak @.> Sent: Thursday, March 2, 2023 6:14 PM To: espressif/esp-protocols @.> Cc: Vasil Nikolov @.>; Comment @.> Subject: Re: [espressif/esp-protocols] Implement URC Handling. (IDFGH-8812) (Issue #180)

Hi Franz and Vasil,

sorry for not responding earlier. The example of injecting a URC callback doesn't really change anything in the esp_modem layers, it just overrides the default command implementation using these two methods:

1. Traditional method for sending commands makes the class command-able, so it could be employed by the command library to run all/defined AT commands:

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L51

1. The data callback that processes the replies

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L70

This way, we're able to

• first call the user urc-callback (and after that)
• continue with processing the replies by the command library

---

a. For me it is important to understand how modem implementation differentiates between URCs and a normal response to AT commands

we process the user callback first

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L73-L75

...and then let the command library to parse the same data.

https://github.com/espressif/esp-protocols/blob/c61fe1f5f915b2d39dda81ea8ce701660991194c/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L76-L87

i. There are commands which require longer time until a response is received.

You need to be aware that the urc handler would still received all the replies, including the ones that belong to unrelated commands issued by the command library (by user)

b. Can we receive URCs while we are waiting for a response to a certain AT Command?

Yes, the urc-handler gets always called (if enabled). The other part (lib-command processing) is called only if an AT command is in progress.

i. If yes then how can we be sure that responses to the AT Commands are forwarded to the URC handler instead to the command handler

This works like a fork (tee command in bash :-), we pass the same data to two handlers

The only trouble might be the time, physically spent on processing the urc responses, since it actually delays processing the simultaneously active AT command. This wouldn't delay data reception, but might influence timeouts of the defined commands. (for example, if we're sending AT+CGC\r with timeout of 500ms and spend 600ms on urc processing, then it would fail, but should still work if we spend like 450ms on urc-processing, as the actual reply would simply queue so the data handler should make it in time)

— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-protocols/issues/180#issuecomment-1452137962, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AISW2OROTO7EKPSAGUUAFNTW2DBNXANCNFSM6AAAAAASKKZCCY. You are receiving this because you commented.Message ID: @.**@.>>

[franz-ms-muc]

https://gist.github.com/franz-ms-muc/b28a70d42a86bb2ed292e638a11fa152

i test the URC HAndler now in the CMUX Sample.
it is installed in this Line: https://gist.github.com/franz-ms-muc/b28a70d42a86bb2ed292e638a11fa152#file-gistfile1-txt-L257

but then never called. However, as i have extended Logging active, i see the URCs are coming in.

however when i was testing it with the Console Sample it was working.

think there is something to fix still.

[franz-ms-muc]

another Log with Net: https://gist.github.com/franz-ms-muc/454ecbf8fbbfabfaeb27f68b54ea98b3

[franz-ms-muc]

@david-cermak did you look into this ?

[david-cermak]

@franz-ms-muc Could you please also share the code where you install the handler? Note that the change has already been merged to master (and release as modem-v1.0.0), so you should be able to play with URC directly in the console example. Does it work for you in the default example?

[franz-ms-muc]

Ah, thanks, i will look into the new master. Cool.

Von: david-cermak @.> Gesendet: Dienstag, 21. März 2023 17:57 An: espressif/esp-protocols @.> Cc: Franz Höpfinger @.>; Mention @.> Betreff: Re: [espressif/esp-protocols] Implement URC Handling. (IDFGH-8812) (Issue #180)

@franz-ms-muchttps://github.com/franz-ms-muc Could you please also share the code where you install the handler? Note that the change has already been merged to master (and release as modem-v1.0.0https://github.com/espressif/esp-protocols/releases/tag/modem-v1.0.0), so you should be able to play with URC directly in the console example. Does it work for you in the default examplehttps://github.com/espressif/esp-protocols/tree/master/components/esp_modem/examples/modem_console?

— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-protocols/issues/180#issuecomment-1478253793, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQSZUH2UNRFN3WNLQAD346DW5HMUHANCNFSM6AAAAAASKKZCCY. You are receiving this because you were mentioned.Message ID: @.**@.>>

[https://www.meisterschulen-mchn.de/images/meisterschulen/allgemein/4-logos/logofc.jpg]

Meisterschulen am Ostbahnhof Mühldorfstr. 6 81671 München

Tel.: 089 416002 - 0 Fax: 089 416002 - 111

Internet: www.meisterschulen-münchen.dehttp://www.meisterschulen-mchn.de

[david-cermak]

Could you please close this issue if it work for you? Or comment and share the details if URC handling doesn't work as expected.

[diplfranzhoepfinger]

hi,

i tested like this:

use this Commit: https://github.com/diplfranzhoepfinger/esp-protocols/commit/0a33ce531c5543391e32cf2296423709ae660b32

and the Outcome is this Log: https://gist.github.com/diplfranzhoepfinger/73da3e69cbee3edbaaa61def2f50eba1

+CGNSSPWR: READY!

is a URC. it is not handeled by the URC Handler.

[diplfranzhoepfinger]

making another Try:

https://gist.github.com/diplfranzhoepfinger/6a3062d8861535f2b3927cdc6e5d299f

there the URC Handler works.

it seems the URC HAndler only works on "cmd xx" and not on "build in" commands.

pls look into this. Thanks ! Franz

[diplfranzhoepfinger]

@david-cermak can you have a look into this ?

Thanks!

[diplfranzhoepfinger]

so, Test also URC Handling with latest Sample:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client

[diplfranzhoepfinger]

at least seems working:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/URC_Handling

[diplfranzhoepfinger]

https://gist.github.com/diplfranzhoepfinger/359e821d89635118333169d75226fe55

[diplfranzhoepfinger]

with the A7672 it does not work:

see https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/A7672_gnss

[diplfranzhoepfinger]

Try to switch to "Shiny" again:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/shiny_again

[diplfranzhoepfinger]

so we have:

URC Handler working:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/shiny_again

URC Handler not working:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/A7672_gnss

[diplfranzhoepfinger]

@david-cermak can you have a Look into this:

maybe the Inheritance is core of this Problem. 

esp_modem::GenericModule --> MyShinyModem

versus

esp_modem::GenericModule --> esp_modem::A7600 --> A7672_gnss

is the 3-stage Design avoid the URC Handler to work ?

[diplfranzhoepfinger]

diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client@b018980

[diplfranzhoepfinger]

Hi, 

https://gist.github.com/diplfranzhoepfinger/780f68da1da1e73565d3a63bccdf9836

removing the URC Handler before switching modes did help to get it running also in CMUX Mode. 

see Commit: diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client@b018980

[diplfranzhoepfinger]

Now test it for the A7672_gnss

for "shiny" it works. 

switch to A7672_gnss diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client@39b3129

Outcome: https://gist.github.com/diplfranzhoepfinger/6cea8065dcead903a0a55b63f31f058c

NOT Working at all. 

so URC Handling works for "Shiny" but not for "A7672_gnss"

@david-cermak Please help. 

THANKS !

[david-cermak]

Okay, so the main issue why it doesn't work in your implementation lies in here:

template <typename ...Agrs> esp_modem::return_type name(Agrs&&... args)...

Perfect forwarding is a great and elegant feature to pass various arguments, but here we just pass it symmetrically, so in the end the command executed from DCE context was symmetrically passed to the device and it's DTE, skipping the URC handler (which belongs to DCE!)

The reason for DCE inheriting also from CommandableIf is that we could have the same command executed directly by DCE (and use the URC handler, which belongs to the DCE), and this is also why we pass this (this DCE) to the implementation of those commands:

https://github.com/espressif/esp-protocols/blob/68392f0ba9d4fd9c9c7fa1f7d54d0b3de53a1990/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L46

To fix this, you just replace the perfect forwarding by a macro magic:

@@ -62,21 +80,20 @@ public:
     }

 #define ESP_MODEM_DECLARE_DCE_COMMAND(name, return_type, num, ...) \
-    template <typename ...Agrs> \
-    esp_modem::return_type name(Agrs&&... args)   \
-    {   \
-        return device->name(std::forward<Agrs>(args)...); \
+    esp_modem::return_type name(__VA_ARGS__) \
+    {                                               \
+        return esp_modem::dce_commands::name(this ARGS(num)); \
     }

I'd suggest adding the implementation to cpp module, as you'd have to use the utility expansion macros

https://github.com/espressif/esp-protocols/blob/68392f0ba9d4fd9c9c7fa1f7d54d0b3de53a1990/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L27-L37

as well as namespacing and includes that you might not like to have in hpp module

https://github.com/espressif/esp-protocols/blob/68392f0ba9d4fd9c9c7fa1f7d54d0b3de53a1990/components/esp_modem/examples/modem_console/main/my_module_dce.cpp#L21

[timbo100]

Hi I've been following this and related issue because I think they relate to a capability I need. But since details are incomplete I'm having difficulty.
I'm working with a SIM7600 modem and need to be notified of latent AT command responses or unanticipated events such as SMS text message arrival. Maybe I'm off base here but it sounds like URC is what I need. Can you help understand:

• where is the concept as discussed herein for URC clearly defined?
• if it is a concept/feature that I'm looking for and tests of it's implementation are successful when might I be able to use in the esp-modem component?
• can I get use of it sooner?
• otherwise, if you know of how I can implement a method for notification in the interim I would appreciate tc@nuspatial.com (timbo100) Thx

[david-cermak]

@timbo100 You can experiment with the URC directly in the console example. It's not something super smart, it just installs a callback so you receive everything that's passed to the modem layers as well.

if you type in the

$>urc

you'd also get the response printed out to the console, sine the command installs this buffer-hexdump callback:

https://github.com/espressif/esp-protocols/blob/68392f0ba9d4fd9c9c7fa1f7d54d0b3de53a1990/components/esp_modem/examples/modem_console/main/modem_console_main.cpp#L93-L97

[timbo100]

@david-cermak Thanks for your response. I saw that in the console example, however it appears to only be implemented if the modem type is "SHINY" and despite my exploring I couldn't figure out how that differs from the implementation of SIM7600.

For future reference can you comment on what SHINY refers to other than a favorite label that one of the contributors uses when creating test?

Can you comment when this feature will be available in the esp_modem component for use?

THANKs to you and the team of contributors.

[franz-ms-muc]

Hi, 

i will test to implement it now for SIMCOM A7672

[diplfranzhoepfinger]

incorporating your Changes:

i get:

https://gist.github.com/diplfranzhoepfinger/0e2f5e3ff5e273d9ba4d955ac4a1f5ea

which is simple to fix by:

diff --git a/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp b/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
index 9c24631..cc0e0ae 100644
--- a/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
+++ b/simple_cmux_client/components/simple_cmux_client/simple_cmux_client_main.cpp
@@ -267,8 +267,9 @@ extern "C" void simple_cmux_client_main(void)

     /* Read some data from the modem */
     std::string str;
+    int a;
 #ifndef CONFIG_EXAMPLE_MODEM_DEVICE_SHINY
-    while (dce->get_operator_name(str) != esp_modem::command_result::OK) {
+    while (dce->get_operator_name(str, a) != esp_modem::command_result::OK) {
         // Getting operator name could fail... retry after 500 ms
         vTaskDelay(pdMS_TO_TICKS(500));
     }

[diplfranzhoepfinger]

see changes there:

https://github.com/diplfranzhoepfinger/esp-protocols-tree-master-components-esp_modem-examples-simple_cmux_client/tree/feature/URC-Handler

[diplfranzhoepfinger]

and YEAH !!!! the URC Handler runs on the A7672

log here:

https://gist.github.com/diplfranzhoepfinger/2d663b68e47f5cbf7a11a980548166ee

i think it is a cool Feature ! 

now must implement the 2nd Layer above the URC Handler, a "URC Handler Parser" 

[diplfranzhoepfinger]

@timbo100 you can see my Cmux - Example, there i got it working.

[timbo100]

@diplfranzhoepfinger and team, thanks.

while connected to a SIM7600 modem, I changed modem type to SHINY and it worked... I got connected, IP address, and tested the comm with mqtt broker.

I really want to test sending an SMS Text message, does anyone know how to use the console to do this since that command is rather complicated???

It's unclear to me, other than new functionality added to custom modem, what the difference is between a SIM7600 vs SHINY. It would seem then that the Generic Modem is very close to SIM7600(?).

Very much appreciated.

[pavellevitsky]

Switch modem from SIM76000 to SHINY - how? Using menuconfig (Development Framework Configuration) --> Connect configuration ? There is no SHINY in the list...

[david-cermak]

While this is supported in the example by creating a custom module, it's not very user friendly. I've added a new milestone and will support URC in a better way, directly in the library.

[diplfranzhoepfinger]

Hi @david-cermak

Added as c61fe1f

it's very simple and WIP. It only adds a user callback that could handle all received data. (needs to be removed before switching modes and applied again after)

this was working well with the CMUX Example. but now we switched to the Dual Channel USB Example. how to add the same URC Handler in this Example ?

https://github.com/espressif/esp-protocols/tree/master/components/esp_modem/examples/pppos_client

I mean one Problem is it is C, so the Syntax will be different. But we would more like to stay in C.

Thanks, Franz

[david-cermak]

Hi Franz,

If you want to use the idea in the modem example, you'd have to stay with C++. You can use Dual channel mode in C++, just need to create the DTE and DCE separately (same as in other C++ examples). You can check the USB example and the C-API wrappers.

(as mentioned above, I'd like to add URC handler support to the library itself, so those overrides and customized factories won't be needed, planned for modem-v1.2 )

[diplfranzhoepfinger]

does this mean: if we want to stay with C, it will be possible with 1.2 if we swith the pppos_client Sample to C++ we can do it like now.

[david-cermak]

does this mean: if we want to stay with C, it will be possible with 1.2 if we swith the pppos_client Sample to C++ we can do it like now.

Yes, it means that the URC will be an official API. The solution I offered before was just a workaround, and since the C++ implementation is very modular, we were able to simply inject a handler without changing the library code (100% user space)

[VasilNikolovRilabs]

Hello,

Could you please check which URC solution was proposed by ESP?

Best Regards, Vasil Nikolov CEO and CO-Founder

Email: @.**@.> Phone: +359 888 200510

Banat 1 1407 Sofia Bulgaria

@.***

www.rilabs.iohttp://www.rilabs.io/

From: david-cermak @.> Sent: Tuesday, September 17, 2024 10:55 AM To: espressif/esp-protocols @.> Cc: Vasil Nikolov @.>; Comment @.> Subject: Re: [espressif/esp-protocols] Implement URC Handling. (IDFGH-8812) (Issue #180)

Closed #180https://github.com/espressif/esp-protocols/issues/180 as completed via #620https://github.com/espressif/esp-protocols/pull/620.

— Reply to this email directly, view it on GitHubhttps://github.com/espressif/esp-protocols/issues/180#event-14286272072, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AISW2ORFBS7OKYY5IXEUQSTZW7N35AVCNFSM6AAAAAASKKZCC2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGI4DMMRXGIYDOMQ. You are receiving this because you commented.Message ID: @.**@.>>

[diplfranzhoepfinger]

Hello @david-cermak i do not find the C-API URC Handler for ESP-Modem in Version 1.2.0 did you forget to add it ?

Thanks, Franz

[diplfranzhoepfinger]

@david-cermak

also next, we need to implement a URC Parser. so that we can react on the received URCs.

Thanks, Franz