ATI response cannot be parsed as String, but can as Bytes

[ijager]

I noticed the following when trying out atat. When defining ATI response using heapless::Bytes it parses correctly.

#[derive(Debug, Clone, AtatResp)]
pub struct ATIResponse
{
    pub info: Bytes<64>,
}

The response:

ATIResponse { info: b'Quectel\r\nEC25\r\nRevision: EC25EFAR06A09M4G' }

But when using heapless::String it results in Error::Parse.

#[derive(Debug, Clone, AtatResp)]
pub struct ATIResponse
{
    pub info: String<64>,
}

Perhaps still the same issue as in #86.

[MathiasKoch]

Hmm. Without any more context, it's a bit hard to tell why, but the main difference between bytes & string, is that a string reply must be quoted explicitly (So you cannot just parse a bytes response as a string response currently)

[ijager]

Ah so I guess this is intended behaviour then. So only when the modem returns explicit "...." including the quotes we can use String

[MathiasKoch]

Ah so I guess this is intended behaviour then. So only when the modem returns explicit "...." including the quotes we can use String

Currently, this is the behavior, yes.

I would have no objections against adding some sort of derive attribute option to the string implementation, that allows skipping the quotes in either end? I currently have no use cases for this myself, so I won't be adding it, but you are more than welcome to open a PR with it? :smiley:

[jamwaffles]

I ran into this problem at work and came up with the following solution. I'm not particularly proud of it but it's an ok stopgap until something nicer comes along :)

/// A helper to deserialize unquoted strings in AT responses.
struct UnquotedStringVisitor<const N: usize>;

impl<'de, const N: usize> Visitor<'de> for UnquotedStringVisitor<N> {
    type Value = heapless::String<N>;

    fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
        formatter.write_str("an unquoted string")
    }

    fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
    where
        E: de::Error,
    {
        let s = std::str::from_utf8(v).map_err(serde::de::Error::custom)?;

        if s.len() > N {
            Err(serde::de::Error::custom("Source string too long"))
        } else {
            Ok(s.into())
        }
    }
}

#[derive(Debug, Clone, PartialEq)]
pub struct UnquotedHeaplessString<const N: usize>(pub heapless::String<N>);

impl<'de, const N: usize> Deserialize<'de> for UnquotedHeaplessString<N> {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: serde::Deserializer<'de>,
    {
        let out = deserializer.deserialize_bytes(UnquotedStringVisitor)?;

        Ok(Self(out))
    }
}

impl<const N: usize> From<&str> for UnquotedHeaplessString<N> {
    fn from(s: &str) -> Self {
        Self(s.into())
    }
}

impl<const N: usize> fmt::Display for UnquotedHeaplessString<N> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str(&self.0)
    }
}

[mschnell1]

Ah so I guess this is intended behaviour then. So only when the modem returns explicit "...." including the quotes we can use String

For me seemingly not.

Using MQTT on a Quectel, I also see that I can't get the result of the messages, which feature two texts included in double quotes, when defining the elements in the OnReceive struct as heapless strings.

This seems even more weird than the first post, as using double quotes for strings seems rather common.

I in fact in that case do not get a Parse error, but no reaction by ATAT at all (other than log). I did doublecheck that the modem response fits the Urc enum arm definition. (I do get appropriate ATAT reactions with other arms of Urc)

[mschnell1]

@ijager BTW.: In fact I did get such a parser error, when Trying ATI but instead of trying out "Bytes" in a response struct, I avoided using the parser by not deriving the "send" - Response struct from AtatResp but implementing my own parser. IMHO this in fact makes sense for not "true" URC (but directly on send() ) responses from the modem, as here you easily can handle the response in any way you like.

@MathiasKoch Is it intended to use the standard (URC) parser with not really URC (i.e. direct) responses from the modem, or is the intended way to implement a dedicated parser of each of those (that is what I did in my project).

-Michael

[mschnell1]

@MathiasKoch

I would have no objections against adding some sort of derive attribute option to the string implementation, that allows skipping the quotes in either end? I currently have no use cases for this myself, so I won't be adding it, but you are more than welcome to open a PR with it? smiley

Are there already any derive attributes that define the behavior of the parser differently from what the main derive macro extracts from the type of the struct elements ?

In the other issue thread I found #[at_arg(position = 0)]. Now trying to find any documentation on that....

[MathiasKoch]

I think you are mixing request/response and URCs here.

For the request response it is all handled by AtatCmd and AtatResponse that can both be derived, URC parser has nothing to do with these at all

[mschnell1]

I'm not in the office right now. I'll do more testing ASAP. (Hoping to be able to avoid the implementation of my own parser for the response of communication as simple as ATI with non quoted strings <by means of #[at_arg(position = 0)] or similar>. And of course to get a notification in the user code from an urc with quoted strings such as +QMTRECV. ) As discussed in the other thread, the urc-Parser - supposedly if the parser associated to the current send() is not able to handle the result message - needs to see that message, too. Obviously correct, but slightly confusing for "newcomers".

[ijager]

@mschnell1 I am also implementing Quectel MQTT commands for the EC25. Just managed to connect to a broker. Next step I think is also those URC's and some state machine to progress through the necessary commands.

I did notice that other commands such as ManufacturerId (AT+CGMI) do get parsed fine into a heapless::String even though those responses are not quoted as far as I know (at least not from my Quectel EC25). So far the only difference I can see is that the ATI response is multi line instead of a single line ending in \r\n. Which I guess makes sense that it would trip up the default String parsing, because I think it is usually only reading until the first \r\n.

@MathiasKoch I am not particularly interested in the ATI response, so for me it is enough to know that Bytes + manual parsing is the way to go for those cases. If you want you can close this issue.

[mschnell1]

I am also implementing Quectel MQTT commands

I managed to get a connection, subscribe to a topic, send a message to the topic and (via the log) see the correct topic update message from the server. Have you already been able to decently be notified by ATAT about a received topic massage from the broker ?

as far as I know

You can use the log feature of ATAT to see the complete dialog with the modem.

So far the only difference I can see

Up till now I don't get the QMTT response even though it is quoted. The difference to ATI is also that it is not related to a sent(), but strictly an URC, the modem sends unexpectedly. That should force the response to (potentially) be parsed by different parser configurations, as different Receive structs can bne associated by the Send struct vs the URC enum.

-Michael

[ijager]

Yes when I turn on logging for atat it shows quotes, but when I sniff the communication with a serial - usb converter there are no quotes. So maybe the quotes are added somewhere in atat?

I will try to implement subscribing today.

[mschnell1]

Using MQTT on a Quectel, I also see that I can't get the result of the messages, which feature two texts included in double quotes, when defining the elements in the OnReceive struct as heapless strings. -Michael

[mschnell1]

Edit:

by some magic I now get a correct +QMTRECV notification by ATAT

This happend after I added a #[at_urc("+QMTPUB")] arm to the Urc enum that now is able to decently handle the URC that comes before the +QMTRECV Ist that normal ?

[MathiasKoch]

Well, if you are receiving URC's that are not defined in the Urc enum, it will kinda end up as undefined behaviour, as it will become part of the next attempt to parse a known URC or response, with varying success.

[ijager]

@mschnell1 I got it working now, the QMTRECV parses okay. But indeed you have to define and decorate the URC enum and define the struct properly so that it matches your settings. E.g. whether it returns the length as well.

This is my Urc enum so far:

#[derive(Clone, AtatUrc, Debug)]
pub enum Urc {
    #[at_urc("+QMTOPEN")]
    MqttOpen(commands::mqtt::urc::MqttOpen),
    #[at_urc("+QMTCONN")]
    MqttConnect(commands::mqtt::urc::MqttConnect),
    #[at_urc("+QMTSUB")]
    MqttSubscribe(commands::mqtt::urc::MQTTSubscribe),
    #[at_urc("+QMTRECV")]
    MQTTReceive(commands::mqtt::urc::MQTTReceive)
}

// and the urc structs

#[derive(Clone, Debug, AtatResp)]
pub struct MqttOpen {
    pub client_idx: u8,
    pub result: u8,
}

#[derive(Clone, Debug, AtatResp)]
pub struct MqttConnect {
    pub client_idx: u8,
    pub result: u8,
    pub ret_code: u8,
}

#[derive(Clone, Debug, AtatResp)]
pub struct MQTTSubscribe {
    pub client_idx: u8,
    pub msg_id: u8,
    pub result: u8,
    pub value: u8,
}

#[derive(Clone, Debug, AtatResp)]
pub struct MQTTReceive {
    pub client_idx: u8,
    pub msg_id: u8,
    pub topic: String<32>,
    pub len: u8,
    pub payload: String<32>
}

Here is my full log output in case it helps:

14:54:35.576 DEBUG - [EC25] state: Init
14:54:35.680 DEBUG - send next cmd EchoOn(EchoOn)
14:54:35.680 DEBUG - Sending command: ""ATE1\r\n""
14:54:35.783 DEBUG - Received OK
14:54:35.864 DEBUG - send next cmd GetIdentification(IdentificationInformation)
14:54:35.864 DEBUG - Sending command: ""ATI\r\n""
14:54:35.969 DEBUG - Received response: ""Quectel\r\nEC25\r\nRevision: EC25EFAR06A09M4G""
14:54:36.071 DEBUG - send next cmd GetManufacturerId(GetManufacturerId)
14:54:36.071 DEBUG - Sending command: ""AT+CGMI\r\n""
14:54:36.172 DEBUG - Received response: ""Quectel""
14:54:36.172 DEBUG - Manufacturer: ManufacturerId { id: "Quectel" }
14:54:36.278 DEBUG - send next cmd GetModelId(GetModelId)
14:54:36.278 DEBUG - Sending command: ""AT+CGMM\r\n""
14:54:36.384 DEBUG - Received response: ""EC25""
14:54:36.384 DEBUG - Model: ModelId { id: "EC25" }
14:54:36.460 DEBUG - send next cmd GetSoftwareVersion(GetSoftwareVersion)
14:54:36.460 DEBUG - Sending command: ""AT+CGMR\r\n""
14:54:36.565 DEBUG - Received response: ""EC25EFAR06A09M4G""
14:54:36.565 DEBUG - sw version: SoftwareVersion { id: "EC25EFAR06A09M4G" }
14:54:36.668 DEBUG - send next cmd SetPDPContextDefinition(SetPDPContextDefinition { cid: ContextId(1), pdp_type: "IP", apn: "portalmmm.nl" })
14:54:36.668 DEBUG - Sending command: ""AT+CGDCONT=1,\"IP\",\"portalmmm.nl\"\r\n""
14:54:36.773 DEBUG - Received OK
14:54:36.877 DEBUG - send next cmd ActivateContext(ActivateContext { cid: ContextId(1) })
14:54:36.877 DEBUG - Sending command: ""AT+QIACT=1\r\n""
14:54:36.980 DEBUG - Received OK
14:54:37.061 DEBUG - send next cmd MQTTConfigRecvMode(ConfigRecvMode { param: "recv/mode", client_idx: ContextId(1), msg_recv_mode: 0, msg_len_enable: Some(1) })
14:54:37.061 DEBUG - Sending command: ""AT+QMTCFG=\"recv/mode\",1,0,1\r\n""
14:54:37.162 DEBUG - Received OK
14:54:37.264 DEBUG - send next cmd MQTTOpenContext(Open { client_idx: ContextId(1), host_name: "mybroker.xyz", port: 1883 })
14:54:37.264 DEBUG - Sending command: ""AT+QMTOPEN=1,\"mybroker.xyz\",1883\r\n""
14:54:37.370 DEBUG - Received OK
14:54:37.473 DEBUG - send next cmd AT(AT)
14:54:37.473 DEBUG - Sending command: ""AT\r\n""
14:54:37.579 DEBUG - Received OK
14:54:37.659 DEBUG - urc: MqttOpen(MqttOpen { client_idx: 1, result: 0 })
14:54:37.659 DEBUG - [EC25] state: Connect
14:54:37.764 DEBUG - send next cmd MQTTConnect(Connect { client_idx: ContextId(1), mqtt_client_id: "Ingmar" })
14:54:37.764 DEBUG - Sending command: ""AT+QMTCONN=1,\"Ingmar\"\r\n""
14:54:37.870 DEBUG - Received OK
14:54:37.870 DEBUG - MQTT CONNECT: NoResponse
14:54:37.976 DEBUG - urc: MqttConnect(MqttConnect { client_idx: 1, result: 0, ret_code: 0 })
14:54:37.976 DEBUG - [EC25] state: Subscribe
14:54:38.077 DEBUG - send next cmd MQTTSubscribe(Subscribe { client_idx: ContextId(1), msg_id: 1, topic: "blink", qos: 0 })
14:54:38.077 DEBUG - Sending command: ""AT+QMTSUB=1,1,\"blink\",0\r\n""
14:54:38.155 DEBUG - Received OK
14:54:38.155 DEBUG - MQTT SUB: NoResponse
14:54:38.261 DEBUG - urc: MqttSubscribe(MQTTSubscribe { client_idx: 1, msg_id: 1, result: 0, value: 0 })
14:54:38.261 DEBUG - [EC25] state: Ready
14:54:42.555 DEBUG - urc: MQTTReceive(MQTTReceive { client_idx: 1, msg_id: 0, topic: "blink", len: 5, payload: "start" })
14:54:42.555 INFO - Received "start" on topic "blink"
14:55:03.898 DEBUG - urc: MQTTReceive(MQTTReceive { client_idx: 1, msg_id: 0, topic: "blink", len: 5, payload: "hello" })
14:55:03.898 INFO - Received "hello" on topic "blink"

And this is what I see on the uart lines:

RDY
AT
OK
ATE1
OK
ATI
Quectel
EC25
Revision: EC25EFAR06A09M4G
OK
AT+CGMI
Quectel
OK
AT+CGMM
EC25
OK
AT+CGMR
EC25EFAR06A09M4G
OK
AT+CGDCONT=1,"IP","portalmmm.nl"
OK
AT+QIACT=1
OK
AT+QMTCFG="recv/mode",1,0,1
OK
AT+QMTOPEN=1,"mybroker.xyz",1883
OK
AT
OK
+QMTOPEN: 1,0
AT+QMTCONN=1,"Ingmar"
OK
+QMTCONN: 1,0,0
AT+QMTSUB=1,1,"blink",0
OK
+QMTSUB: 1,1,0,0
+QMTRECV: 1,0,"blink",5,"start"
+QMTRECV: 1,0,"blink",5,"hello"

[mschnell1]

Well, if you are receiving URC's that are not defined in the Urc enum, it will kinda end up as undefined behaviour, as it will become part of the next attempt to parse a known URC or response, with varying success.

Nasty, but supposedly unavoidable.

This seems to mean that any possible response by the modem needs to be covered in the Urc enum....

[MathiasKoch]

So far i haven't found a better approach, but to be fair most URC's are default disabled on most modems, and needs to be explicitly enabled. At least on the modems i have used?

[mschnell1]

@ijager Seemingly you use a different logger. in my Debug output I don't see the double double quotes. It looks like this: DEBUG atat::ingress > Received response (51/51): "Quectel\r\nEC25\r\nRevision: EC25EFAR06A14M4G" I Can't find the type heapless::Bytes you use (the ATAT create does not republish same, while it dose provide atat::heapless::String ) So I tried to do

#[derive(Clone, Debug, AtatResp)]
pub struct QueryModemType2Response {
    #[at_arg(position = 0)]
    pub received_modem_type: atat::heapless::Vec<u8, 300>, //String<0x100>,
}

but I still get the send() result: Err(Parse) (with and without the #[at_arg(position = 0)] line)

Any further idea ?

Btw.: is it appropriate to do such "silly user" discussions in an "issue" thread ?

-Michael

[mschnell1]

Sorry for continuing being a PITA..... 1) I don't find out anything about the #[at_arg(position = 0)] stuff. ATAT seems somehow to forward this to some serdes functionality. Any starters on how to find any documentation on that ?

2) Right now, I use AT+QMTPUB to publish QMT topic content. That does work and easily is compatible with ATAT. But in fact that command is not even documented with the Quectel modem I use. The documentation states AT+QMTPUBEX instead. I in fact would like to use that, anyway, as it can send binary data instead of text only. With AT+QMTPUBEX , the modem expects a message length instead of the message as the 6th parameter. it then sends a > and expects as many bytes as given as message length. After that it sends OK . Is such a procedure implemented or anyhow supported in ATAT ?

3) when transferring binary data with QMTT, will ATAT be able to receive such +QMTRECV urc messages in the normal way ?

Thanks for listening ! -Michael

[MathiasKoch]

at_arg is documented as part of the relevant derive macro in atat_derive; https://github.com/BlackbirdHQ/atat/blob/master/atat_derive/src/lib.rs#L141-L165

[MathiasKoch]

As for two this is supported and i use it multiple places, but for now it requires you to split it into two commands as

https://github.com/BlackbirdHQ/ublox-cellular-rs/blob/master/ublox-cellular/src/command/device_data_security/mod.rs#L79-L121

And then they can be used as:

https://github.com/BlackbirdHQ/ublox-cellular-rs/blob/master/ublox-cellular/src/services/data/ssl.rs#L55-L70

[MathiasKoch]

As for question 3, i think that requires more knowledge about the modem than i have at present, and sounds less like an atat question?

If not i would need more info about what you are asking?

[ijager]

@ijager Seemingly you use a different logger.

I am using rtt-target in combination with log.

I Can't find the type heapless::Bytes you use

It is actually a different crate: heapless_bytes::Bytes.

Btw.: is it appropriate to do such "silly user" discussions in an "issue" thread ?

There is also the matrix chat

[mschnell1]

@MathiasKoch

As for question 3, i think that requires more knowledge about the modem than i have at present, and sounds less like an atat question?

As far as I understand, the Modem can handle binary payload in such way:

• configure the Modem to enable the optional length parameter in the +QMTRCVUrc
• use AT+QMTPUBEXinstead of the (not even documented but working) AT+QMTPUBcommand to send the binary payload.
• When receiving, the +QMTRCV Urc contains the topic in quotes, the length as digits, and then the payload in quotes

Now the payload can contain any characters including quotes, comma, etc. Only by relying on the length parameter, the receiver can know the end of the payload. The "URC parser" needs to support this in some way. -Michael

[MathiasKoch]

Ahh, yeah that is a very common way of doing it in AT commands, but it is not currently supported in ATAT :/

Contributions are welcomed.

[mschnell1]

@ijager

It is actually a different crate: heapless_bytes::Bytes.

Yep that works. Seemingly the Derive Macro recognizes Bytes<64> but not Vec<u8, 300> . Which is weird, as in heapless_bytes I see

pub struct Bytes<const N: usize> {
    bytes: Vec<u8, N>,
}

[mschnell1]

Ahh, yeah that is a very common way of doing it in AT commands, but it is not currently supported in ATAT :/

Contributions are welcomed.

Yep ! Once I find out how to do this and get to know more about the current options in ATAT, which I of course would not want to get in conflict with ....

[mschnell1]

Sorry for continuing being a PITA. Trying to implement AT+QMTPUBEX , I see that ATAT successfully detects a "prompt" (>): In the log I see: DEBUG atat::ingress > Received prompt (4/4) Seemingly this does release the client.send().await ; with an OK result. But how to see that this in fact is a prompt and not an "OK" from the modem ?

And after this I need to send a certain count of bytes as the raw payload.

How to do this ? I get close to that by

#[derive(Clone, Debug, AtatCmd)]
#[at_cmd(
    "",
    NoResponseMqttSetup,
    cmd_prefix = "",
    termination = "",
    value_sep = false,
    timeout_ms = 3000
)]
pub struct MqttSend2_1 {
    pub msg: Bytes<64>,
}

This does seem to work. Is that the recommended way to send raw data ?

-Michael

[mschnell1]

@ijager : How do you progress with MQTT ? -Michael

[ijager]

@ijager : How do you progress with MQTT ?

I just implemented +QMTPUBEX by splitting it up in two parts. Seems to work as expected.

[mschnell1]

Yep. That does work fine for me, as well. I just did not find out how to be sure that the callback I get between the parts in fact is a "Prompt" (which ATAT seems to be detecting) or something else (such as an OK or Error ), which would indicate a problem. Receiving ASCII (and supposedly UTF-2) data by URC +MQTRECV (if wanted with msg_lengthenabled) also works perfectly.

But In fact I would like to transfer binary data. The modem does handle this decently, but ATAT can't cope with a double quote or any bytes > 127 in the payload.

Thanks, -Michael