SIM7670G MQTT(S) WILL not working

[cho0p]

Hello,

I have a Lilygo SIM7670G with the ESP32-S3. I use the TinyGSM Fork. The device connects to the internet without any problems. It also connects to my MQTT server via SSL which is great but I need to use the WILL functionality.

I have modified the TinyGsmMqttA76xx.h file and added the WILL AT commands according to the SIM767XX Series_AT_Command_Manual_V1.01.pdf:

public:
bool mqttWillTopic(uint8_t clientIndex, const char* topic) {
    if (clientIndex > muxCount) {
        Serial.println("Error: Client index out of bounds");
        return false;
    }

    // Set the Will topic
    // +CMQTTWILLTOPIC: <client_index>,<req_length>
    thisModem().sendAT("+CMQTTWILLTOPIC=", clientIndex, ',', strlen(topic));

    int response = thisModem().waitResponse(10000UL, ">");
    if (response != 1) {
        Serial.print("Error: Did not receive expected '>' prompt, response: ");
        Serial.println(response);
        return false;
    }

    // Send the actual topic
    thisModem().stream.write(topic);
    thisModem().stream.println();

    response = thisModem().waitResponse();
    if (response != 1) {
        Serial.print("Error: Did not receive 'OK' after sending the Will topic, response: ");
        Serial.println(response);
        return false;
    }

    return true;
}

public:
bool mqttWillMessage(uint8_t clientIndex, const char* message, uint8_t qos) {
    if (clientIndex > muxCount) {
        Serial.println("Error: Client index out of bounds");
        return false;
    }

    // Set the Will message
    // +CMQTTWILLMSG: <client_index>,<req_length>,<qos>
    thisModem().sendAT("+CMQTTWILLMSG=", clientIndex, ',', strlen(message), ',', qos);

    int response = thisModem().waitResponse(10000UL, ">");
    if (response != 1) {
        Serial.print("Error: Did not receive expected '>' prompt, response: ");
        Serial.println(response);
        return false;
    }

    // Send the actual message
    thisModem().stream.write(message);
    thisModem().stream.println();

    response = thisModem().waitResponse();
    if (response != 1) {
        Serial.print("Error: Did not receive 'OK' after sending the Will message, response: ");
        Serial.println(response);
        return false;
    }

    return true;
}

My code looks like this:

void mqtt_callback(const char *topic, const uint8_t *payload, uint32_t len)
{
    Serial.println();
    Serial.println("======mqtt_callback======");
    Serial.print("Topic:"); Serial.println(topic);
    Serial.println("Payload:");
    for (int i = 0; i < len; ++i) {
        Serial.print(payload[i], HEX); Serial.print(",");
    }
    Serial.println();
    Serial.println("=========================");
}

bool mqtt_connect()
{
    Serial.print("Connecting to ");
    Serial.print(broker);

    bool ret = sim_modem.mqtt_connect(mqtt_client_id, broker, broker_port, clien_id, mqtt_username, mqtt_password);
    if (!ret) {
        Serial.println("Failed!"); return false;
    }
    Serial.println("successed.");

    if (sim_modem.mqtt_connected()) {
        Serial.println("MQTT has connected!");
    } else {
        return false;
    }
    // Set MQTT processing callback
    sim_modem.mqtt_set_callback(mqtt_callback);
    // Subscribe to topic
    sim_modem.mqtt_subscribe(mqtt_client_id, subscribe_topic);

    // Publish the birth message
    const char* birth_topic = "esp32-00000-04/status";
    const char* birth_message = "online";
    sim_modem.mqtt_publish(0, birth_topic, birth_message);

    return true;
}

void handle_mqtt() {
    // MQTT handling
    sim_modem.mqtt_handle();
    delay(5);
}

Within void setup():

Serial.println("Starting Modem MQTT SSL Setup");

Serial.println("Setting will message in 5 seconds...");
delay(5000);

// Enable verbose error reporting
sim_modem.sendAT("+CMEE=2");
delay(1000);

sim_modem.sendAT("+CMQTTSTART");
if (sim_modem.waitResponse(30000UL, "+CMQTTSTART: 0") != 1) {
    Serial.println("Failed to start MQTT");
    return;
}

// Acquire MQTT client
// sim_modem.sendAT("+CMQTTACCQ=0,\"client_id\"");
sim_modem.sendAT("+CMQTTACCQ=0,\"client_id\",0");
delay(1000);

// Configure MQTT context
Serial.println("Configuring MQTT context...");
sim_modem.sendAT("+CMQTTCFG=\"version\",0,4");  // Example: Set version to MQTT 3.1.1
delay(1000);

const char* will_topic = "esp32-00000-04/status"; 

Serial.println("Setting will topic...");
if (!sim_modem.mqttWillTopic(0, will_topic)) {
    Serial.println("Failed to set Will topic");
    return;
}
delay(5000);

Serial.println("Setting will message...");
if (!sim_modem.mqttWillMessage(0, "offline", 2)) {
    Serial.println("Failed to set Will message");
    return;
}
delay(5000);

// Initialize MQTT with SSL
sim_modem.mqtt_begin(true);
if (!mqtt_connect()) {
    Serial.println("Failed to connect to MQTT broker");
    return;
}

Within void loop():

handle_mqtt();

As said the code works and it connects. I cannot see any error message in the Serial with AT debug being enabled. In fact the modem replies with "OK" to the WILL AT commands:

09:59:58:877 -> AT+IPR=115200
09:59:58:879 -> OK
09:59:58:879 -> Modem baud rate set to 115200
09:59:58:879 -> Starting Modem MQTT SSL Setup
09:59:58:879 -> Setting will message in 5 seconds...
10:00:03:889 -> AT+CMEE=2
10:00:04:880 -> AT+CMQTTSTART
10:00:04:882 -> AT+CMEE=2
10:00:04:882 -> OK
10:00:04:885 -> AT+CMQTTSTART
10:00:04:886 -> OK
10:00:04:889 -> 
10:00:04:889 -> +CMQTTSTART: 0AT+CMQTTACCQ=0,"client_id",0
10:00:05:892 -> Configuring MQTT context...
10:00:05:892 -> AT+CMQTTCFG="version",0,4
10:00:06:894 -> Setting will topic...
10:00:06:895 -> AT+CMQTTWILLTOPIC=0,21
10:00:06:896 -> 
10:00:06:896 -> AT+CMQTTACCQ=0,"client_id",0
10:00:06:901 -> OK
10:00:06:901 -> AT+CMQTTCFG="version",0,4
10:00:06:901 -> OK
10:00:06:901 -> AT+CMQTTWILLTOPIC=0,21
10:00:06:903 -> >esp32-00000-04/status
10:00:06:909 -> esp32-00000-04/status
10:00:06:911 -> 
10:00:06:911 -> OK
10:00:11:912 -> Setting will message...
10:00:11:912 -> AT+CMQTTWILLMSG=0,7,2
10:00:11:918 -> AT+CMQTTWILLMSG=0,7,2
10:00:11:920 -> >offline
10:00:11:924 -> offline
10:00:11:927 -> 
10:00:11:927 -> OK
10:00:16:928 -> AT+CMQTTSTART
10:00:16:933 -> AT+CMQTTSTART
10:00:16:934 -> ERROR
10:00:16:934 -> Connecting to xxx.xxx.xxAT+CSSLCFG="authmode",0,0
10:00:16:942 -> AT+CSSLCFG="authmode",0,0
10:00:16:944 -> OK
10:00:16:945 -> AT+CMQTTREL=0
10:00:16:950 -> AT+CMQTTREL=0
10:00:16:951 -> OK
10:00:16:951 -> AT+CMQTTACCQ=0,"esp32-00000-04",1
10:00:16:960 -> AT+CMQTTACCQ=0,"esp32-00000-04",1
10:00:16:962 -> OK
10:00:16:962 -> AT+CMQTTCFG="version",0,4
10:00:16:969 -> AT+CMQTTCFG="version",0,4
10:00:16:971 -> OK
10:00:16:971 -> AT+CMQTTCONNECT=0,"tcp://xxx.xxx.xx:9883",60,1,"xxx","xxx"
10:00:16:996 -> AT+CMQTTCONNECT=0,"tcp://xxx.xxx.xx:9883",60,1,"xxx","xxx"
10:00:19:222 -> OK
10:00:19:224 -> 
10:00:19:224 -> +CMQTTCONNECT: 0,0successed.
10:00:19:225 -> AT+CMQTTDISC?
10:00:19:226 -> 
10:00:19:229 -> AT+CMQTTDISC?
10:00:19:235 -> +CMQTTDISC: 0,0
10:00:19:236 -> +CMQTTDISC: 1,1
10:00:19:237 -> 
10:00:19:237 -> OK
10:00:19:237 -> MQTT has connected!

The problem is that my Mosquitto broker doesn't receive any WILL message when the client connects. MQTT server logs:

2024-10-20 10:00:19: New client connected from xxx:10001 as esp32-00000-04 (p2, c1, k60, u'xxx').
2024-10-20 10:00:19: No will message specified.
2024-10-20 10:00:19: Sending CONNACK to esp32-00000-04 (0, 0)

The WILL message works fine with Pubsubclient on other ESP32 devices.

What could be wrong? @lewisxhe Do you have a suggestion? I couldn't find any further information in the SIM767XX AT commands documentation. Yesterday I flashed the modem to the latest firmware 2374B03SIM767XM5A_M_SIM7670G-MNGV_V202240702 but it didn't fix the issue either.

Thanks, cho0p

[lewisxhe]

Please update to the latest firmware, then use the log capture tool to capture all interaction logs and send them to us.

[cho0p]

Thanks for your swift reply @lewisxhe . The modem was already running on the latest firmware B03. I have created the modem log file. The safe functionality gave me an error message - I was only able to safe it as *.csv. Hope this works as well. I have sent you the file via email to lewishe@***.com.

[lewisxhe]

I checked my email and didn't receive the log you sent. Please upload it directly to this issue.

[cho0p]

SIM7670G_WILL_Log_20241021_105737.csv

Here is the modem log file.

[lewisxhe]

The exported log should be a zip archive, not a csv. Have you checked my usage video?

[cho0p]

Yes, but as mentioned in the last message I get an error message when I click on the safe button in the logger software. I was only able to safe the log as csv.

[lewisxhe]

This log is invalid. Simcom can only parse the correct log file. At which step did you encounter the error?

[cho0p]

The error appeared after clicking on the safe button in the top menu of the tool. I will try to create another log in a few hours and share either the log or the error message when trying to safe it.

[lewisxhe]

OK

[cho0p]

SIM7670G_Log_20241022_102950.zip

Tried it with another PC and now it worked.

[lewisxhe]

Ok, I'll update here if I get a reply.

[lewisxhe]

SIMCOM replied that they used the same version of the modem for testing and MQTTWILLTOPIC could be used normally. They analyzed the logs you provided and found that the modem sent the CMQTTWILLTOPIC command normally. The problem may be on your server.

[cho0p]

I don't think it's related to the MQTT server as it successfully manages will messages from other clients. I have several LilyGo ESP32 with Simcom modems connected to the server. They all use the Pubsubclient library and the will works fine. It's the first time that I test the SIM7670G with the build-in MQTT capabilities and it doesn't work. This is how the server log looks like:

ESP32 with SIM7000G using Pubsubclient:

2024-10-24 11:00:55: New client connected from x.x.x.x:xxxxx as esp32-00000-02 (p2, c1, k15, u'00000-02-zentec').
2024-10-24 11:00:55: Will message specified (7 bytes) (r1, q0).
2024-10-24 11:00:55  esp32-00000-02/status
2024-10-24 11:00:55: Sending CONNACK to esp32-00000-02 (0, 0)

ESP32 with SIM7670G using build-in modem MQTT capability:

2024-10-24 11:02:00: New client connected from x.x.x.x:xxxxx as esp32-00000-04 (p2, c1, k60, u'00000-04-zentec').
2024-10-24 11:02:00: No will message specified.
2024-10-24 11:02:00: Sending CONNACK to esp32-00000-04 (0, 0)

Do you have any other suggestions for debugging this?

[lewisxhe]

After my own testing, the will message is normal and can be used. I will add examples later

[lewisxhe]

I added an example and the test is ok https://github.com/Xinyuan-LilyGO/LilyGO-T-A76XX/tree/main/examples/MqttsBuiltlnWill

[cho0p]

Thank you. The code works. Only one little remark regarding the example code in line 223:

    // Initialize MQTT, use SSL, skip authentication server
    modem.mqtt_begin(false);

I think it should be (if SSL should be used)

    // Initialize MQTT, use SSL, skip authentication server
    modem.mqtt_begin(true);

[lewisxhe]

Oh! Whether this uses SSL depends on the server you are connecting to. I used a server without SSL when testing, so I forgot to change it.

[cho0p]

The WILL message itself works now but I noticed that it doesn't have a retain flag. Unfortunately I couldn't find any retain option in the AT documentation. It's only mentioned for the AT+CMQTTPUB command. Do you know if it's somehow possible to set the retain flag for the WILL message as well when the client connects to the broker?

[lewisxhe]

SIMCOM replied that there is no method to set this parameter.