Cell Boradcast Test Command

[CoderThomasB]

I'm interested in trying to get cell broadcasts working in modem manager but there isn't any way to simulate a cell broadcast currently. It would be useful if there was an AT command or SMS to +223344556677 that would simulate a cell broadcast both using the cell broadcasts to SMS feature and the dedicated cell boradcast API.

There also seems to be no documentation on the format of cell broadcasts which would be really helpful when implementing them into modem manager.

[Biktorgj]

Been there, had that very same issue :) In some versions I had a command that would replay a cell Broadcast that was previously retrieved. I could add a test command that would send it on demand. Most of the CB transaction is the same as a normal SMS but the actual message PDU was different. Let me check if I still have the logs, and if I have them I will get you one or two test commands that will send a broadcast message to ModemManager

[airtower-luna]

Germany is going to do a national emergency alert test on December 8th, which is supposed to include cell broadcast (source in German). I'd be happy to try and capture that, if it'd be helpful and I can get some hint on how to do it.

[CoderThomasB]

That may be helpful but we need a way to save and relay cell broadcasts which doesn't currently exist in the modem firmware. I might have a go at trying to implement that code this week, if I manage to get something I'll share it on this thread.

[Biktorgj]

Hi @CoderThomasB, @airtower-luna

Cell broadcast relay is supported in the firmware. You can enable it from the chat with enable cell broadcast and disable cell broadcast If you want to make extra sure that is turned on, you can also use its specific AT command: AT+CSCB=0,"0-6000","0-15"

If enabled, when you receive a cell broadcast message from your network, the modem should send you an SMS with whatever the original message had. It should also split it in pieces, but it seems some offset is not correctly calculated and it might eat some letters. If CB is enabled and you receive a message, OpenQTI will enable debug mode for a second before parsing the message, dump the raw contents of the message to the log, and then go back to default logging mode. In its log, you should see intercept_cb_message: CB MESSAGE DUMP, then a ton of hex data, followed by intercept_cb_message: CB MESSAGE DUMP END.

That hex dump should be enough to recreate the message you got, and if it's multipart, might help me debug those eaten letters too :)

[airtower-luna]

So having CB enabled in the firmware and retrieving openqti.log after receiving the message should be enough? That's easy enough. :slightly_smiling_face:

[airtower-luna]

I just got a couple surprise CBs that say they're a regional test.

Modem log: cb_openqti.log

I first got two batches of modem generated SMS: the first one with the German version, the second one with the English version. As you can see in the screenshots below there are some letters (or half words) missing in both languages, and the German additionally has messed up umlauts (replaced with ~ or {). The first issue is definitely the more important one, though, what the umlauts should be is pretty obvious. :wink:

Then there were two more batches, repeating the same messages in opposite order, with the same issues, so I didn't bother with more screenshots. Only the timestamps are different.

Note that the SMS timestamps said 2022-01-02 for all of the messages.

Please let me know if I can provide any other data that'd help!

[airtower-luna]

Got five more this morning, after updating to 0.7.1: cb2_openqti.log

These were all separate messages, not split, but are still missing a few letters at the end. The last word in the messages in the Screenshot should clearly be "Gefahr".

[Biktorgj]

At least we know the headers are always correctly parsed, from the first log:

 ID: 0x11
  LEN: 0x0060
  - Serial 64
 - Message ID 7185
 - Encoding: 0x00
 - Page 16

But, Interesting that every message from the second log shows they are independent messages and it still eats some letters (this is the first time I get a cell broadcast dump of a single page message I think):

 ID: 0x11
  LEN: 0x005a
  - Serial 12352
 - Message ID 47891
 - Encoding: 0x0f
 - Page 11

For context: Cell broadcast messages are handled independently instead of waiting for all of them to arrive on purpose. Normally, a phone would wait for all the pages in a message to arrive before showing up on the UI, but if a message gets lost, or communication to the tower is abruptly cut, then the message would be lost. That page 11 means actually page 1/1, in the multipart message, you see 16 to 66, or 1/6 to 6/6. I still relay those message as soon as they come, since this is used as an emergency system, I rather let you know "A tornado is heading your [CUT]" than wait for the entire message to tell you "A tornado is heading your way, look for shelter".

I think all the logs I got so far were multipart messages, so I never know what the end of the message, always ending in something something 0x00 0x00 was for (if it was some Qualcomm thing -some padding for the QMI message- or part of the SMS PDU that might be described in some obscure 3GPP document I haven't found or overlooked). Removing the last 4 bytes from the first message in your first log, reveals BBK erprobt das neue Handy-Warn-Syst, which would perfectly align with the second message in that multipart; and if I remove the last 4 bytes in the single message, I get Dies ist eine Test Warnnachricht. Bitte ignorieren Sie diese. Es besteht keine Gefahr. 4 bytes are 32 bits, and GSM is 7Bit/ character, so 32 bits would be 4 characters and 4 leftover bits, or 5 characters -3 bits... and we're missing 4-6 characters (infor[mation],f[irmwar]e in your first log) ... So maybe the trick is to remove that garbage from the end but keep the length field we get and the missing letters might magically appear?

Will surely do some tests and report back :)

[Biktorgj]

Small question, are those ~ in the middle of the phrase supposed to be there? With the fix in place it looks like this:

EDIT: I changed it a bit more, so the "Incoming Cell broadcast message" text only appears once per multipart message, so it's a bit less annoying :)

Now, for the test commands that @CoderThomasB needs. Maybe I could add support to retrieve some sort of hexdump from a file in the modem and process it as if it was generated from the network? I don't know if we can fake the entire transaction, as I only have the actual messages and not the notification->ack->receive->ack->delete->ack sequence, but as long as we have one I think it could be done...

[airtower-luna]

Small question, are those ~ in the middle of the phrase supposed to be there?

From context those should clearly be umlauts ("verf~gt" -> "verfügt", "~ber" -> "über"). Similar for the { in "Endger{t", that should be "Endgerät".

I looked up the specification for the German CB implementation (DE-Alert). Here's version 1.1, and here the overview page. Unfortunately there doesn't seem to be an English version of the specification.

I assume these are the critical parts (my translation, you can find the original on p.26 of the PDF):

8.28 A CBC [Cell Broadcast Center] must encode the content of a message (Message Content or CB data) of a CB message either using the GSM 7-Bit Default Alphabet according to 3GPP TS 23.038 or using the Universal Coded Character Set 2 (UCS-2) according to ISO/IEC 10646.

8.29 If the language listed in the IE [information element] info.language is supported by the GSM 7-Bit Default Alphabet, the message content (Message Content or CB data) of the CB message must be encoded using the GSM 7-Bit Default Alphabet. Otherwise the message content (Message Content or CB data) of the CB message must be encoded using the UCS-2 character set.

8.30 The IE Data Coding Scheme (DCS) of the generated CB message must contain a binary value 0000 0000 to 0000 1110 (MSB ... LSB) in case of encoding with the GSM 7-Bit Default Alphabet, and the binary value 0001 0001 (MSB ... LSB) in case of encoding with UCS-2 character set with "language indication". The transmitted language must be encoded according to 3GPP TS 23.038 "CBS Data Coding Scheme".

I'm not exactly familiar with GSM standards so I can't say if it lines up with the international standards, but it seems to imply there are variations of the 7 bit encoding for different languages. The DE-Alert specification currently requires support for German and English but states others may be added in the future.

Thank you very much for digging into this! :smile_cat:

[CoderThomasB]

I don't know if we can fake the entire transaction

All I need is something that looks close enough to a CB so that I can have a go at trying to implement it in Modem Manager.

The modem is currently able to send SMSs over +223344556677. So maybe we could reuse some of that code but change it to send what look like Cell Broadcasts?

Another option is to make Modem Manager able to take dumps from a file or something and then feed that directly into the code.

[airtower-luna]

Here's the log from today's test alert: de-alert-test.log

Should be English first, then German. The latter has another umlaut issue in the output (with 0.7.1), with | where ö should be.

[snm247]

Logs from today's test alert with test version 0.7.2:

PinePhone:

pp_openqti.log

PinePhone Pro:

ppp_openqti.log

Edit: Maybe the following screenshots are helpful for comparison: one shows the CB messages on the PinePhone and the other one the German CB message on an Android phone (no missing characters):

Fun fact: the English CB message on the Android phone was incorrect (with some German words mixed in) and had missing characters, too ;) - I will not post it here to avoid confusion.

[Biktorgj]

So, a bit of background just in case: CB messages can be multipart, and they're similar to SMS messages, though they aren't stored in memory. They also can come in GSM-7 or UCS-2 encodings. Here's an example packet:

QMUX header
0x01 QMUX CTL
0x73 0x00  PKT size
0x80 control
0x05 service (WDS)
0x01 instance

QMI Header
0x04 control ID
0x03 0x00  transaction ID
0x01 0x00 message ID
0x67 0x00 size

0x11 <-- Transfer-route MT message
0x60 0x00 <-- The size of the message
0x00 0xff <-- Service category (unset)
0xff 0xff <-- Language (unset)
0xff <-- Is category selected? Nope (0xff)

0x07 <-- Type 0x07: Message starts here
0x58 0x00  <-- Size of the entire PDU
Message PDU starts here:
0x40 0xd0 Serial number (u16)
0x11 0x12 Message ID (u16)
0x00 Encoding (u8)
0x24 Page (2/4)
Actual Message:
0x63 0x34 0x3b 0xec 
0x26 0x83 0x5a 0xa0 0xe2 0x1c 0x24 0x2e 0xcf 0xe9 
0x65 0x34 0x1d 0xb4 0x2e 0xa7 0xdd 0x65 0xd0 0xb1 
0x6c 0x0e 0xa3 0xe5 0x2e 0x50 0x0b 0x74 0x2d 0xa7 
0xe9 0x65 0x79 0x19 0x94 0x74 0x9b 0xdf 0x73 0x50 
0xb8 0x6e 0x06 0xa1 0xe9 0x74 0xf8 0x5c 0xf7 0x7a 
0xdd 0xc3 0x72 0x77 0xdd 0x7d 0x76 0x89 0xeb 0x6e 
0xb2 0x8b 0x5c 0x7e 0xb5 0xcb 0x6c 0x72 0xdd 0x7d 
0x2e 0xbb 0x41 0x2d 0x10 0xb2 0x2c 0x07 

0x16 <-- Type 0x16: Are we using IMS to send this message?
0x01 0x00  <-- Size (1 byte)
0x00 <-- No we're not using it || 0x01 -> Yes this message has been sent through IMS

You might also want to check https://www.etsi.org/deliver/etsi_ts/123000_123099/123038/10.00.00_60/ts_123038v100000p.pdf (Page 12 explains the coding schemes)

I'm adding three new AT commands:

• AT+SEND_SAMPLE_CB
• AT+SEND_RANDOM_CB
• AT+SEND_MULTIPART_CB

AT+SEND_SAMPLE_CB -> Sends a single CB message via QMI to the port with Page indication 1/1 AT+SEND_RANDOM_CB -> Sends one of a multipart message via QMI, with a page indication x (so you can test how things behave when you miss one of the packets) AT+SEND_MULTIPART_CB -> Sends a stream of messages (1/5 to 5/5) from a previous capture, in order

Maybe it's a good idea to have an example stream with the message parts out of order? I don't know what's modemmanager supposed to do, as these don't seem to stay in memory in the modem and ModemManager has to keep them all if it wants to merge them as one...

I have a set of UCS-2 encoded Cell Broadcast message too, but it comes from Ukraine, so they're a bit sensitive in nature. I can try to redact them, though that will involve changing sizes and stuff and want to avoid the possibility of a mistake from me making you waste time. When you get some piece of it working we can give it a shot

[CoderThomasB]

https://www.theguardian.com/society/2023/mar/19/uk-launches-emergency-phone-alerts-public-warning-system

There is going to be a test of the emergency alert system in the UK, which is probably going thorough cell broadcast. If anyone lives there and can record the modem logs, that would be great!

[Biktorgj]

Oh damn me I forgot to post here, my bad, sorry.

The commands I posted above are working in latest firmware versions! The only problem left to solve is that we don't really know how we should answer to the modem, though it doesn't seem to care if you reply to the transaction with an ACK or not.

You can use these AT commands to send example CB messages to ModemManager:

• AT+SEND_SAMPLE_CB
• AT+SEND_RANDOM_CB
• AT+SEND_MULTIPART_CB

These won't be delivered through the chat, but will be sent as if they were real broadcast messages from the network. ModemManager, in its current state, should discard them, but if you're running in debug mode you should see them.

These are in GSM-7 encoding, do you want me to add UCS-2 too?

[CoderThomasB]

@Biktorgj Thanks! I'll get round to testing them with my PinePhone soon.

What I suspect is happening in ModemManager it that somewhere in the code it is checking for the message type and gets the value for a CB, and it doesn't know how to handle it, so it just discards the message (not before logging it, though). The next step is to find that check and add a print statement, so we know what "type" value the CB has, and then a new case to the switch / if that tries to handle CBs.

In the end, we probably want ModemManager to expose a signal on it's dbus API that Phosh or other shell/apps and hook into and display alerts "properly" rather than text messages.

[CoderThomasB]

@Biktorgj Ok, I just try it and got this error from Modem Manager:

error parsing PDU (0): Invalid date: year: 2011, month: 52, day: 46, hour: 17, minute: 100, second: 104

That month is clearly not valid and the g_date_time_new does not like it, hence the error. Do you know what is happening here?