esp_modem_dte dual-port mode not working with all modems (IEC-58)

[DaStoned]

Answers checklist.

• [X] I have read the component documentation ESP-IDF Components and the issue is not addressed there.
• [X] I am using target and esp-idf version as defined in component's idf_component.yml
• [X] I have searched the issue tracker for a similar issue and not found any related issue.

Which component are you using? If you choose Other, provide details in More Information.

Other

ESP-IDF version.

v5.1.1

Development Kit.

ESP32S3-DevKitC

Used Component version.

esp_modem_usb_dte 1.1.0~1

More Information.

Quoting public documentation for esp_modem_usb_dte 1.1.0:

Dual port modems

Some modems provide two equivalent AT ports. One of the ports can be used for AT commands, while the other one can be used for network data. This way, you don't have to switch between command and data modes of one terminal.

To use this feature, specify interface number of the second port in esp_modem_usb_term_config.

This feature does not seem to be implemented. Sure, I can set the secondary inteface in esp_modem_usb_term_config and create_usb_dte() will happily give me an USB DTE. The PPP dialup connection comes up, but when I try to issue any AT commands they time out. I can see from debug logs of usb_terminal that it tries to inject the AT command directly into the PPP data stream which doesn't work for obvious reasons.

    struct esp_modem_usb_term_config usb_config =
        ESP_MODEM_DEFAULT_USB_CONFIG_DUAL(GSM_USB_VID, GSM_USB_PID, GSM_USB_IDX_CMD, GSM_USB_IDX_DATA);
    esp_modem_dte_config_t dte_usb_config =
        ESP_MODEM_DTE_DEFAULT_USB_CONFIG(usb_config);

    _uart_dte = create_usb_dte(&dte_usb_config);
    _modem = create_SIM7070_dce(&_modem_config, _uart_dte, _esp_netif);
...

    if(!_modem->set_mode(modem_mode::DATA_MODE)) {
       tag_error("Fail activate DATA_MODE");
       return false;
    }

I cannot issue commands to the USB-connected modem (SIM7070) when it's in data mode. When I go and look at relevant code in the component "esp_modem", there is absolutely nothing that actually implements a dual mode feature.

Clearly DTE::set_mode(modem_mode m) does absolutely nothing when told to activate DUAL_MODE: https://github.com/espressif/esp-protocols/blob/ea14e15a6f113b1335f2a2511119c63b897f354d/components/esp_modem/src/esp_modem_dte.cpp#L199

And DCE_Mode::set_unsafe() explicitly ignores DUAL_MODE: https://github.com/espressif/esp-protocols/blob/ea14e15a6f113b1335f2a2511119c63b897f354d/components/esp_modem/src/esp_modem_dce.cpp#L95

Am I missing something obvious or are you advertising a feature that's not implemented?

[tore-espressif]

@DaStoned Thanks for the issue. I had it working on my setup, but I must admit that I tested only with 2 modems.

First of all, you must make sure that your modem supports USB dual-mode. The easiest way to do so, is by plugging the modem to any PC and check whether it creates two AT serial ports. Not all modems support this.

There is a regression in esp_modem 1.0.2 that breaks this function. Could you please try with esp_modem 1.0.1? In your main idf_component.yml only define esp_modem: "1.0.1"

Here is a link to the 1.0.1 source code for your convenience https://github.com/espressif/esp-protocols/tree/modem-v1.0.1/components/esp_modem

---

More info:

Clearly DTE::set_mode(modem_mode m) does absolutely nothing when told to activate DUAL_MODE: And DCE_Mode::set_unsafe() explicitly ignores DUAL_MODE:

That is correct. These functions manage transitions of the DTE from 'data' to 'command' mode. Since your DTE is in DUAL_MODE there is no need to change anything. The DTE DUAL_MODE is set in the constructor: https://github.com/espressif/esp-protocols/blob/modem-v1.0.1/components/esp_modem/src/esp_modem_dte.cpp#L30

[DaStoned]

@tore-espressif - thank you for the quick response. Dual mode is tested and supposed to work - that's good to hear.

My modem is a SIMCOM SIM7070G which creates 6 virtual serial ports when plugged into a Linux PC. Their documentation (_SIM7070_SIM7080_SIM7090 Series_Linux_Application NoteV1.02.pdf) says:

Interface number | Endpoint Type | Function
0                | USB serial    | Diagnostic Interface
1                | USB serial    | GPS NMEA Interface
2                | USB serial    | AT port Interface
3                | USB serial    | QFLOG Interface
4                | USB serial    | DAM Interface
5                | USB serial    | Modem port Interface

I plugged in indexes 2 and 5 into the macro ESP_MODEM_DEFAULT_USB_CONFIG_DUAL() (as the code example above shows, GSM_USB_IDX_CMD being set to 2 and GSM_USB_IDX_DATA 5), started a PPP dialup with _modem->set_mode(modem_mode::DATA_MODE) and expected to be able to issue simple AT commands (e.g. AT+CSQ or AT+CIMI) during an ongoing PPP session. This does not work, as I see that the AT commands are spit right into the raw PPP stream and time out. So my conclusion is that dual port mode is not active.

Perhaps I'm using some part of the API wrong in dual port mode. E.g. how am I supposed to start a PPP session in dual port?

And I'm using esp_modem version 1.0.3, but I tried with 1.0.1 - no change. I'm calling get_signal_quality() and get_imsi() from the C++ API during an active PPP session and they both time out:

I (165470) gsm_modem: Testing GSM dual mode...
D (166470) usb_terminal: 0x3fcc2a4c   41 54 2b 43 53 51 0d                              |AT+CSQ.|
E (166970) gsm_modem: Fail read signal quality: TIMEOUT
D (166970) usb_terminal: 0x3fcc2a4c   41 54 2b 43 49 4d 49 0d                           |AT+CIMI.|
I (168960) ocpp_protocol: Sending Heartbeat.req
D (168960) usb_terminal: 0x3c270784   7e 21 45 00 00 4b 00 63  00 00 ff 06 2c ff 0a e9  |~!E..K.c....,...|
D (168960) usb_terminal: 0x3c270794   ae 39 ac 42 28 e6 f6 4f  01 bb c2 47 18 42 16 5f  |.9.B(..O...G.B._|
D (168970) usb_terminal: 0x3c2707a4   9e 17 50 18 13 d0 76 e6  00 00 17 03 03 00 1e 00  |..P...v.........|
D (168980) usb_terminal: 0x3c2707b4   00 00 00 00 00 00 22 0b  06 ca a3 0c e7 37 01 df  |......"......7..|
D (168990) usb_terminal: 0x3c2707c4   97 6b cf 21 b9 7d 5e 11  cf 67 a8 6b 1d 1e 8d 27  |.k.!.}^..g.k...'|
D (169000) usb_terminal: 0x3c2707d4   7e                                                |~|
I (169010) ws_client: WS TX 57 B: [2,"56c52991-fff8-4ecd-fb1a-8bdafd244afb","Heartbeat",{}]
D (169990) usb_terminal: 0x3fcb482c   7e 21 45 00 00 28 ae 75  40 00 33 06 0b 10 ac 42  |~!E..(.u@.3....B|
D (169990) usb_terminal: 0x3fcb483c   28 e6 0a e9 ae 39 01 bb  f6 4f 16 5f 9e 17 c2 47  |(....9...O._...G|
D (170000) usb_terminal: 0x3fcb484c   18 65 50 10 f9 28 a1 32  00 00 07 1b 7e           |.eP..(.2....~|
D (170010) usb_terminal: 0x3c2713a4   7e 21 45 00 00 7d 5e 00  64 00 00 ff 06 2c cb 0a  |~!E..}^.d....,..|
D (170020) usb_terminal: 0x3c2713b4   e9 ae 39 ac 42 28 e6 f6  4f 01 bb c2 47 18 65 16  |..9.B(..O...G.e.|
D (170030) usb_terminal: 0x3c2713c4   5f 9e 17 50 18 13 d0 35  4e 00 00 17 03 03 00 51  |_..P...5N......Q|
D (170040) usb_terminal: 0x3c2713d4   00 00 00 00 00 00 00 23  25 37 75 6e 0f d0 c4 bf  |.......#%7un....|
D (170050) usb_terminal: 0x3c2713e4   57 00 4e b5 ad 9f 09 9f  a3 87 d9 d4 97 98 9c 0a  |W.N.............|
D (170050) usb_terminal: 0x3c2713f4   63 f5 18 b2 a8 26 6c 06  41 78 46 1a d7 00 a1 ae  |c....&l.AxF.....|
D (170060) usb_terminal: 0x3c271404   7b 92 5b d8 82 6d ce 7b  79 ec fe 71 52 8a a8 a9  |{.[..m.{y..qR...|
D (170070) usb_terminal: 0x3c271414   22 bb bb 28 f8 3d 5c a6  34 68 f9 bf 58 b0 06 fb  |"..(.=\.4h..X...|
D (170080) usb_terminal: 0x3c271424   71 7f 33 7e                                       |q.3~|
D (170380) usb_terminal: 0x3fcb482c   7e 21 45 00 00 28 ae 76  40 00 33 06 0b 0f ac 42  |~!E..(.v@.3....B|
D (170390) usb_terminal: 0x3fcb483c   28 e6 0a e9 ae 39 01 bb  f6 4f 16 5f 9e 17 c2 47  |(....9...O._...G|
D (170390) usb_terminal: 0x3fcb484c   18 bb 50 10 f9 28 a0 dc  00 00 1d ea 7e           |..P..(......~|
D (170410) usb_terminal: 0x3fcb482c   7e 21 45 00 00 98 ae 77  40 00 33 06 0a 9e ac 42  |~!E....w@.3....B|
D (170410) usb_terminal: 0x3fcb483c   28 e6 0a e9 ae 39 01 bb  f6 4f 16 5f 9e 17 c2 47  |(....9...O._...G|
D (170420) usb_terminal: 0x3fcb484c   18 bb 50 18 f9 28 51 94  00 00 17 03 03 00 6b 00  |..P..(Q.......k.|
D (170430) usb_terminal: 0x3fcb485c   00 00 00 00 00 00 12 b8  4e 54 2a 96 15 d7 07 0f  |........NT*.....|
D (170440) usb_terminal: 0x3fcb486c   ac 18 4e 34 f2 fb 67 3e  d8 6f ad 52 d2 f9 dd 0c  |..N4..g>.o.R....|
D (170450) usb_terminal: 0x3fcb487c   bc 2f a9 68 2c a6 e8 84  fc a4 f4 bf 82 10 ae 03  |./.h,...........|
D (170460) usb_terminal: 0x3fcb488c   5b 6f 3a 74 dc dc 70 01  51 e1 4b 32 ec c2 75 c5  |[o:t..p.Q.K2..u.|
D (170470) usb_terminal: 0x3fcb489c   2c 84 72 46 50 2c 6e 56  02 35 fa eb f8 1a 45 eb  |,.rFP,nV.5....E.|
D (170480) usb_terminal: 0x3fcb482c   f5 b1 6a 8e 4d eb 05 24  0d a7 9f a6 f8 af 71 60  |..j.M..$......q`|
D (170490) usb_terminal: 0x3fcb483c   3d a7 dd 3d 74 d9 6b 61  ba 16 d9 f0 7e           |=..=t.ka....~|
I (170510) ws_client: WS RX 81 B: [3,"56c52991-fff8-4ecd-fb1a-8bdafd244afb",{"currentTime":"2023-10-12T11:46:09Z"}]
I (170510) ocpp_protocol: Ingress Heartbeat.conf
I (170520) ocpp_protocol:  └─> currentTime: 2023-10-12T11:46:09Z
D (170590) usb_terminal: 0x3c26ec44   7e 21 45 00 00 28 00 65  00 00 ff 06 2d 20 0a e9  |~!E..(.e....- ..|
D (170590) usb_terminal: 0x3c26ec54   ae 39 ac 42 28 e6 f6 4f  01 bb c2 47 18 bb 16 5f  |.9.B(..O...G..._|
D (170590) usb_terminal: 0x3c26ec64   9e 87 50 10 13 60 86 35  00 00 70 c5 7e           |..P..`.5..p.~|
E (171990) gsm_modem: Fail read IMSI: TIMEOUT

[DaStoned]

I'll add the modem & PPP bringup code for reference:

    auto ipCb = [](void *arg, esp_event_base_t evBase, int32_t evId, void *evData) {
        assert(arg);
        static_cast<NetworkInterfaceGSM*>(arg)->onIpEvent(evBase, evId, evData);
    };
    if (ESP_OK != esp_event_handler_register(IP_EVENT, ESP_EVENT_ANY_ID, ipCb, this)) {
        tag_error("Fail register IP event handler");
        return false;
    }

    auto pppCb = [](void *arg, esp_event_base_t evBase, int32_t evId, void *evData) {
        assert(arg);
        static_cast<NetworkInterfaceGSM*>(arg)->onPppEvent(evBase, evId, evData);
    };
    if(ESP_OK != esp_event_handler_register(NETIF_PPP_STATUS, ESP_EVENT_ANY_ID, pppCb, this)) {
        tag_error("Fail register PPP event handler");
        return false;
    }

    /* Configure the PPP netif */
    _modem_config = ESP_MODEM_DCE_DEFAULT_CONFIG(CONFIG_GSM_PPP_APN);
    _netif_ppp_config = ESP_NETIF_DEFAULT_PPP();
    _esp_netif = esp_netif_new(&_netif_ppp_config);
    if(!_esp_netif) {
        tag_error("Fail create PPP netif");
        return false;
    }
    hwBringUp();

    struct esp_modem_usb_term_config usb_config =
        ESP_MODEM_DEFAULT_USB_CONFIG_DUAL(GSM_USB_VID, GSM_USB_PID, GSM_USB_IDX_CMD, GSM_USB_IDX_DATA);
    usb_config.timeout_ms = GSM_USB_APPEAR_TIMEOUT_MS;
    esp_modem_dte_config_t dte_usb_config =
        ESP_MODEM_DTE_DEFAULT_USB_CONFIG(usb_config);
    dte_usb_config.dte_buffer_size = 2048; // Must fit PPPoS MTU (1500 B?)

    tag_info("Waiting for USB device connection...");

    _uart_dte = create_usb_dte(&dte_usb_config);
    if (!_uart_dte) {
        tag_error("Fail create USB DTE");
        return false;
    }

    auto onUsbTerminalErr = [this](esp_modem::terminal_error err) {
        switch(err) {
            case esp_modem::terminal_error::BUFFER_OVERFLOW:
                tag_warn("USB event: BUFFER_OVERFLOW");
                break;
            case esp_modem::terminal_error::CHECKSUM_ERROR:
                tag_warn("USB event: CHECKSUM_ERROR");
                break;
            case esp_modem::terminal_error::UNEXPECTED_CONTROL_FLOW:
                tag_warn("USB event: UNEXPECTED_CONTROL_FLOW");
                break;
            case esp_modem::terminal_error::DEVICE_GONE:
                tag_warn("USB event: DEVICE_GONE");
                m_networkManager->reportConnectionFailure();
                break;
        }
    };

    _uart_dte->set_error_cb(onUsbTerminalErr);
    _modem = create_SIM7070_dce(&_modem_config, _uart_dte, _esp_netif);
    if (!_modem) {
        tag_error("Fail create SIM7070 modem");
        return false;
    }

    command_result cmd_err;
    std::string str_out;
    // Module may take some time to start from cold boot. Wait until responsive.
    uint32_t timeout = bootSecs() + GSM_STARTUP_TIMEOUT_S;
    do {
        cmd_err = _modem->get_module_name(str_out);
        if (command_result::OK != cmd_err) {
            tag_error("Fail get module name: %s (%u)", modem_err_to_name(cmd_err), static_cast<int>(cmd_err));
            vTaskDelay(pdMS_TO_TICKS(1000));
        }
    } while (command_result::OK != cmd_err && bootSecs() < timeout);
    if (command_result::OK != cmd_err) {
        tag_error("Fail module wakeup in %lu s", GSM_STARTUP_TIMEOUT_S);
        return false;
    }
    tag_info("Module name: %s", str_out.c_str());

    cmd_err = _modem->get_imei(str_out);
    if (command_result::OK == cmd_err) {
        tag_info("IMEI: %s", str_out.c_str());
    } else {
        // Odd, but we'll proceed.
        tag_error("Fail read IMEI: %s", modem_err_to_name(cmd_err));
    }

    cmd_err = _modem->get_imsi(str_out);
    if (command_result::OK == cmd_err) {
        tag_info("IMSI: %s", str_out.c_str());
    } else {
        // No SIM card. We'll proceed, though success is not likely.
        tag_error("Fail read IMSI: %s", modem_err_to_name(cmd_err));
    }

    int attached = 0;
    timeout = bootSecs() + GSM_ATTACH_TIMEOUT_S;
    do {
        vTaskDelay(pdMS_TO_TICKS(1000));
        cmd_err = _modem->get_network_attachment_state(attached);
        if (command_result::OK == cmd_err) {
            if(attached == 1){
                tag_info("Network attached.");
                break;
            }else{
                tag_info("Waiting for network...");
                _modem->set_network_attachment_state(true);
            }
        }else{
            tag_error("Fail get attachment state: %s", modem_err_to_name(cmd_err));
        }
    } while (bootSecs() < timeout);
    if (!attached) {
        tag_error("Fail attach to network in %lu s", GSM_ATTACH_TIMEOUT_S);
        return false;
    }

    cmd_err = _modem->get_operator_name(str_out);
    if (command_result::OK == cmd_err) {
        tag_info("Operator: %s", str_out.c_str());
    } else {
        tag_error("Fail read operator: %s", modem_err_to_name(cmd_err));
    }

    cmd_err = _modem->at("AT+CPSI?", str_out, 3000);
    if (command_result::OK == cmd_err) {
        tag_info("UE info: %s", str_out.c_str());
    } else {
        tag_error("Fail read UE info: %s", modem_err_to_name(cmd_err));
    }

    int rssi, ber;
    vTaskDelay(pdMS_TO_TICKS(1000)); // Signal measurements take time
    cmd_err = _modem->get_signal_quality(rssi, ber);
    if (command_result::OK == cmd_err) {
        tag_info("Signal quality: RSSI=%d BER=%d", rssi, ber);
    } else {
        tag_error("Fail read signal quality: %s", modem_err_to_name(cmd_err));
    }

    if(!_modem->set_mode(modem_mode::DATA_MODE)) {
       tag_error("Fail activate DATA_MODE");
       return false;
    }

[tore-espressif]

Interface number | Endpoint Type | Function 0 | USB serial | Diagnostic Interface 1 | USB serial | GPS NMEA Interface 2 | USB serial | AT port Interface 3 | USB serial | QFLOG Interface 4 | USB serial | DAM Interface 5 | USB serial | Modem port Interface

The dual port mode assumes that the two ports are interchangeable. It doesn't seem to be the case in your modem.

You can easily test it out. In your application, open the USB modem with single interface 2 and single interface 5. They should both work the same.

Alternatively, you can connect your modem to a PC and send AT commands to interface 5.

In case interface 5 can't accept AT commands, your modem does not support dual mode.

[DaStoned]

That's strange. When I conduct an experiment with a SIM7070G devkit connected to my Linux PC, those two ports are interchangeable. I can open either port 2 or 5 with minicom and issue commands AT+CGDCONT=1,"IPV4V6","terminal.apn" and ATD*99#. This successfully brings up a dial-up connection on either port.

Port /dev/ttyUSB4, 13:05:52

Press CTRL-A Z for help on special keys

at
OK
AT+CGDCONT=1,"IPV4V6","terminal.apn"
OK
ATD*99#
CONNECT 150000000

OK
ath
OK

But when I try the experiment that you suggested with my board where a module of the same model is connected to ESP32S3 via USB, then port 5 doesn't receive any reply to any commands issued through it.

[DaStoned]

I talked to the SimCom FAE and they confirmed - SIM7070G is a dual-port modem:

You should use interface 2 for AT cmd and interface 5 for modem PPP protocol, they can work in parallel

My experiments on Linux confirm this. So I suspect there's something going on in the Espressif stack which doesn't allow me to use port 5.

[tore-espressif]

Thanks for all the information, I'll try to get the modem and test for regressions

[DaStoned]

I'm not very far from you (Estonia), so I can ship my evaluation board to you if there's nothing easily available.

[diplfranzhoepfinger]

i did test the https://github.com/espressif/esp-protocols/tree/modem-v1.0.1/components/esp_modem/examples/pppos_client and it worked for a A7672E-FASE

i opened another issue https://github.com/espressif/idf-extra-components/issues/284, how we could use more than only Dual Port, but also the Diag Port and the NMEA Port.

[tore-espressif]

@diplfranzhoepfinger Thanks for the confirmation.

It must something specific to SIM7070G then

[diplfranzhoepfinger]

@tore-espressif you sit in Brno ? 

@david-cermak has a A7672E-FASE from me. so you could ask him for a Demo.

[tore-espressif]

you sit in Brno ? @david-cermak has a A7672E-FASE from me. so you could ask him for a Demo.

Yep, will do

[diplfranzhoepfinger]

@diplfranzhoepfinger Thanks for the confirmation.

It must something specific to SIM7070G then

SIM7070G has at least a Problem:

the GPS and the Network cannot work simultaneous. 

this Reason we went from SIM7070G to A7672E-FASE

Regards, 

Franz

[DaStoned]

@diplfranzhoepfinger Thanks for the confirmation. It must something specific to SIM7070G then

SIM7070G has at least a Problem:

the GPS and the Network cannot work simultaneous.

Just to clarify, we're not using GPS in our project, just network connection.

[tore-espressif]

@DaStoned I got SimTech SIM7070G and could do more testing.

This is devices that appear on windows 10:

Interface 2 corresponds to AT port, which is correct and works fine. Interface 5 creates a "modem" device. We don't have support for this class in esp_modem.

For now, you must use single interface 2 and switch between CMD and DATA modes. At the moment, we don't plan to implement a driver for USB interfaces such as the one in SIM7070G

[tore-espressif]

I'll rename this issue so we can collect similar request here and possibly reconsider this feature in the future.

[DaStoned]

OK, a negative answer is still much better than no answer :) Thank you. May I suggest a list of not supported modems in the README?

[tore-espressif]

May I suggest a list of not supported modems in the README?

With the significant differences among modems with similar names (eg. SIM7070 and SIM7670) I'd prefer a list of supported devices. We will add it in next release