UDT short data messages have wrong CRC16 (DMR tier III)

[kantooon]

For private short data messages sent on the TSCC, the output of dsd-fme is:

---

Slot 1 Data Header - Indiv - Unified Data Transport (UDT) - Response Requested - Source: 2268018 Target: 2260319
SAP 00 [UDT Data] - FMT 4 [ISO8 Char] - PDn 12 - BLOCKS 0 SF 0 - PF 0 OP 1A
Slot 1 - Multi Block Control Message CRC16 ERR 687A - A4B6

---

And for group short data messages:

---

Slot 1 Data Header - Group - Unified Data Transport (UDT) - Source: 2268018 Target: 13167
SAP 00 [UDT Data] - FMT 4 [ISO8 Char] - PDn 12 - BLOCKS 0 SF 0 - PF 0 OP 1A
Slot 1 - Multi Block Control Message CRC16 ERR CA72 - 6BE

---

The radios decode the message just fine. I've tried looking at the code but could not find the issue. I suspect a CRC mask might be applied although the last UDT continuation block CRC16 is specified to have no CRC mask, see TS 102 361-1: 8.2.2.5 Unified Data Transport (UDT) last data block structure

[lwvmobile]

Yeah, probably the CRC Mask, I can already see if you run an XOR on 0xA4B6 and 0xCCCC you get 0x687A, same for the second sample, so without the mask, the comparison would be equal.

[lwvmobile]

Can you send a sample of that so I can work it over?

[kantooon]

I can do that. How do you want me to capture the samples? Format, sample rate? I can use an RTL-SDR for that.

[lwvmobile]

Just use DSD-FME to capture it, you can either make a symbol capture bin file with the -c capture.bin option, or make a wav file of the captured audio with -6 audio.wav . If its off of a repeater, the symbol capture bin should be fine, if its simplex, the wav file will probably work better for me.

You can upload your file to a file host and send the link, I usually just recommend using https://ufile.io/ since it doesn't require registration or log in to upload or download.

[kantooon]

Let's see if it gets through this time. Just rename them to .bin

[lwvmobile]

Just for future reference, its probably easier (and safer) to just zip your bin and wav files up first before uploading them somewhere. Don't name them as PDF files, considering those have been known to carry malware payloads and seeing links like that is normally red flag for me.

I ran the files, I'll look into it. It looks like its trying to run a message with 0 len block value, so I'll dig into it a bit later on and see what I figure out.

[kantooon]

Oh damn I forgot to tell you, I think UAB is wrong too, it should be +1, I did a local patch on that but nuked it by mistake.

[kantooon]

Yeah, you're absolutely right about that, didn't occur to me, I was in a hurry sorry.

[lwvmobile]

Yeah, looks like there are some errors in the block assembly for UDT. Looking back at my notes in the code, it looks like I never had samples to test it on until now. Thinking back now, I seem to remember there being confusion on my end between 'Unified Data Transport' and 'Unified Single Block Data', and having to separate the code for the two according to the ETSI documentation, but otherwise, never have seeing either format before.

14:34:56 Sync: +DMR  [slot1]  slot2  | Color Code=01 | DATA 
 Slot 1 Data Header - Group - Unified Data Transport (UDT) - Source: 2268018 Target: 13167 
  SAP 00 [UDT Data] - FMT 4 [ISO8 Char] - PDn 10 - BLOCKS 1 SF 0 - PF 0 OP 1A BL: 1; BC: 0; 
 DMR PDU Payload [80][04][00][33][6F][22][9B][72][50][1A][CF][E7]
14:34:56 Sync: +DMR   slot1  [slot2] | Color Code=01 | CSBK
 Protected Control Signalling Block(s) (CRC ERR) 
 DMR PDU Payload [FF][83][DF][17][32][09][0E][D1][E7][CD][8A][91]
14:34:56 Sync: +DMR  [slot1]  slot2  | Color Code=01 | UDTC  BL: 1; BC: 1; 
 Slot 1 - Multi Block Control Message
  [80][04][00][33][6F][22][9B][72][50][1A][CF][E7]
  [54][45][53][54][3B][FF][FF][FF][FF][FF][9B][5D]
   Unified Data Transport - ASCII: TEST;     

I got it assembling now, but had to do a few things, including a +1 on the uab appended block value, not sure why its zero for on the header when there is clearly 1 block following. Maybe it means there is an appended data header block, and we only need to look at the last block flag instead? ETSI sure likes to make it confusing at times.

BTW, does your system's opposite slot go into Protected CSBK mode every time UDT data is put out there? Or is that just some decoding errors from signal related issues? Maybe your radio too close to the SDR and nuking that time slot?

[kantooon]

Nicely done! The zero value in UAB is because there can be no data header without at least one block of appended data, so they saved one bit by setting 11 to be 4 blocks.

The CRC error on slot 2 is due to the fact that my radios don't like tuning to a channel sending IDLE bursts when receiving a grant, they simply go back to the CC. So as a workaround the IDLE burst has a data type of CSBK, that keeps them happy. Gotta find a more elegant solution. That and the RTL-SDR being literally near the radios which overload it a little. I should probably disconnect the antenna when capturing.

[lwvmobile]

Nicely done! The zero value in UAB is because there can be no data header without at least one block of appended data, so they saved one bit by setting 11 to be 4 blocks.

Its still a work in progress though, I may have to re-write some parts of UDT handling, I'm beginning to wonder if UAB is for appended header blocks, and then you get your data blocks on top of that after the appended blocks. Honestly, makes no sense. Could you send a longer text message by chance over UDT? Something that'll make it use more blocks, just want to see what the behavior of uab bits are, if it means 0,1,2,3 is really 1,2,3,4, or if we just want the last bit block and to collect x number of blocks until its received.

The CRC error on slot 2 is due to the fact that my radios don't like tuning to a channel sending IDLE bursts when receiving a grant, they simply go back to the CC. So as a workaround the IDLE burst has a data type of CSBK, that keeps them happy. Gotta find a more elegant solution. That and the RTL-SDR being literally near the radios which overload it a little. I should probably disconnect the antenna when capturing.

Oh okay, didn't know you could do something like that, makes sense though, I guess. What vendor radios/repeaters are these?

[kantooon]

Its still a work in progress though, I may have to re-write some parts of UDT handling, I'm beginning to wonder if UAB is for appended header blocks, and then you get your data blocks on top of that after the appended blocks. Honestly, makes no sense. Could you send a longer text message by chance over UDT? Something that'll make it use more blocks, just want to see what the behavior of uab bits are, if it means 0,1,2,3 is really 1,2,3,4, or if we just want the last bit block and to collect x number of blocks until its received.

0,1,2,3 is really 1,2,3,4. There can be no data header without at least one data payload. But I can send longer messages just so you can test, it's no problem. Maximum number of characters in ISO 8 bit is 46. Handling ISO 7 bit or 16 bit unicode might be more tricky for you.

Oh okay, didn't know you could do something like that, makes sense though, I guess. What vendor radios/repeaters are these?

Oh I can do whatever I want, the only question is if it's correct and readable by the radios :) Radios are Hytera, repeater is based on MMDVM on a LimeNet-Micro SDR. Homebrew.

[lwvmobile]

0,1,2,3 is really 1,2,3,4. There can be no data header without at least one data payload. But I can send longer messages just so you can test, it's no problem. Maximum number of characters in ISO 8 bit is 46. Handling ISO 7 bit or 16 bit unicode might be more tricky for you.

Okay, was probably just overthinking it. For now, just try to do one with all ISO8 characters filled in, if possible. I'll have to redo handling to do the ISO7 and UTF16 formats another time, there are several locations that need that treatment, just never implemented it properly from the get go, but have found that UTF16 almost always just carries ISO8 characters with an extra 0x00 in front of it if its just English Alphanumeric, so unless its in Chinese or uses diacritical characters, should still work. ISO7, probably not so much.

Oh I can do whatever I want, the only question is if it's correct and readable by the radios :) Radios are Hytera, repeater is based on MMDVM on a LimeNet-Micro SDR. Homebrew.

Okay, that makes more sense now.

[kantooon]

message.bin.tar.gz

Here is a maximum length message, 4 blocks.

[lwvmobile]

Okay, that seems to be working okay now. Now entirely sure if the completed CRC is okay or not, but I have a feeling this message might have had a bad block, and thus the CRC Error. Reading the text message, it looks like it tried to censor itself lol.

08:53:42 Sync: +DMR  [slot1]  slot2  | Color Code=01 | DATA 
 Slot 1 Data Header - Indiv - Unified Data Transport (UDT) - Source: 16776901 Target: 16777215 
  SAP 00 [UDT Data] - FMT 4 [ISO8 Char] - PDn 0 - BLOCKS 4 SF 0 - PF 0 OP 1A
 DMR PDU Payload [00][04][FF][FF][FF][FF][FE][C5][03][1A][86][CA]
08:53:42 Sync: +DMR   slot1  [slot2] | Color Code=01 | CSBK
 Protected Control Signalling Block(s) (CRC ERR) 
 DMR PDU Payload [FF][83][DF][17][32][09][4E][D1][E7][CD][8A][91]
 SLC_C_SYS_PARMS - Tiny - Net ID: 2 Site ID: 1.2 - Reg Req: 1 - CSC: 1 
 SLCO Completed Block [20][02][60][1F][B0]

08:53:42 Sync: +DMR  [slot1]  slot2  | Color Code=01 | UDTC 
 DMR PDU Payload [4C][6F][72][65][6D][20][69][70][73][75][6D][20]
08:53:42 Sync: +DMR   slot1  [slot2] | Color Code=01 | CSBK
 Protected Control Signalling Block(s) (CRC ERR) 
 DMR PDU Payload [FF][83][DF][17][32][09][4E][D1][E7][CD][8A][91]
08:53:42 Sync: +DMR  [slot1]  slot2  | Color Code=01 | UDTC 
 DMR PDU Payload [64][6F][6C][6F][72][20][6F][6D][67][20][74][68]
08:53:42 Sync: +DMR   slot1  [slot2] | Color Code=01 | CSBK
 Protected Control Signalling Block(s) (CRC ERR) 
 DMR PDU Payload [FF][83][DF][17][32][09][4E][D1][E7][CD][8A][91]
 SLC_C_SYS_PARMS - Tiny - Net ID: 2 Site ID: 1.2 - Reg Req: 1 - CSC: 1 
 SLCO Completed Block [20][02][60][1F][B0]

08:53:42 Sync: +DMR  [slot1]  slot2  | Color Code=01 | UDTC 
 DMR PDU Payload [69][73][20][69][73][20][73][6F][20][62][6F][72]
08:53:42 Sync: +DMR   slot1  [slot2] | Color Code=01 | CSBK
 Protected Control Signalling Block(s) (CRC ERR) 
 DMR PDU Payload [FF][83][DF][17][32][09][4E][D1][E7][CD][8A][91]
08:53:42 Sync: +DMR  [slot1]  slot2  | Color Code=01 | UDTC 
 Slot 1 - Multi Block Control Message CRC16 ERR 8C23 - AC07
 Slot 1 - Multi Block Control Message
  [00][04][FF][FF][FF][FF][FE][C5][03][1A][86][CA]
  [4C][6F][72][65][6D][20][69][70][73][75][6D][20]
  [64][6F][6C][6F][72][20][6F][6D][67][20][74][68]
  [69][73][20][69][73][20][73][6F][20][62][6F][72]
  [69][6E][67][20][73][69][74][20][61][6D][8C][23]
  MBC/UDT Blocks CRC ERR  Unified Data Transport - ASCII: Lorem ipsum dolor omg this is so boring sit am
 DMR PDU Payload [69][6E][67][20][73][69][74][20][61][6D][8C][23]

The CRC is working on the shorter samples, all but one that also had an error in the text message.

[kantooon]

Interesting. My radios don't complain about the CRC so I think it's correct. If the CRC is wrong the radio willl send C_NACK and sliently discard the packet.

[lwvmobile]

It could have just been a reception issue or signal going into DSD-FME related issue. If you ran it multiple times, it would probably be fine for the most part. Not to mention, radios are usually more resilient to signal issues and do a better job at demodulation, and probably do a better job at error correction as well.

Anyways, I pushed the code to date if you want to test it out.

[kantooon]

Also, one other interesting usage of UDT is radios registering with a TG subscription list (maximum 7 groups). This is uplink only, so the bit in the UDT header will be set to indicate uplink instead of downlink. Those UDT packets won't be repeated on the downlink. The TS looks at what talkgroups it will accept and sends back an ACKD with a 7 bit mask where 1 is accepted TG and 0 is denied TG. I'm currently working on this feature in the TS.

[kantooon]

It could have just been a reception issue or signal going into DSD-FME related issue. If you ran it multiple times, it would probably be fine for the most part. Not to mention, radios are usually more resilient to signal issues and do a better job at demodulation, and probably do a better job at error correction as well.

Anyways, I pushed the code to date if you want to test it out.

Thanks a lot! I'll give it a try tomorrow. dsd-fme is an invaluable tool for testing my own code.

[lwvmobile]

Thanks a lot! I'll give it a try tomorrow. dsd-fme is an invaluable tool for testing my own code.

Alright, just let me know, probably won't do any more advanced decoding of UDT for the time being, will probably wait until some time in the future, I don't know why I had assumed it was a CSBK-like format, I must have read that in the manual at some point but didn't investigate further. I'll need something to parse the format types and decode them properly. I swear, to even decode elements out of some portions of DMR, you need at least three of the same PDF open all to different pages to follow the flow from point a to point b to point c. The uniformity of Data Headers particularly is insanely poor.

[lwvmobile]

How easy is it for you to send iso7 and utf16 text over UDT? If its not a big deal, can you send multiple samples of each? You could probably just make one large capture bin file if its convenient for you to do so that way as well. I starting writing a little into the UDT decoder, and wanted to test those first, see how they work and what all needs to be fixed.

[kantooon]

Easy but a bit annoying as I have to fire up Windows to program the radio. Here you go:

iso7_unicode16.tar.gz

[lwvmobile]

Okay, well thanks for the effort, and I understand the annoyance of having to fire up the Windows VM, got to do it every time I do a stupid Cygwin build and release. Anyways, I had worked on a UDT handler for a little bit yesterday, and ran your samples, made a few tweaks, and just pushed the commit for the WIP on it. Its still missing NMEA and LIP Location Reports, which, to be honest, I'm pretty burnt out on GPS reports lately working on that stupid Harris one, but it seems to do okay on the ISO7, ISO8, and UTF16 Text modes, and will also do most all of the other formats as well in the ETSI manual besides dumping any manufacturer specific things.

if you want to run your symbol capture bins back through, just run

dsd-fme -fs -i capture.bin and it'll replay those files you created.

[kantooon]

Cheers man, thanks for your efforts, I know what you mean. I'm facing the same issues, tons of subfeatures in the standard and each manufacturer implements it in its own way. It's true what they say, good work is rewarded by more work.

[lwvmobile]

Alright, well, if everything is working well for you now on the UDT stuff, let me know. I went ahead and implemented the NMEA and LIP protocol stuff, so I think that just about covers UDT for the most part. If everything is going well, I'll go ahead and close this issue out as completed.

[kantooon]

Oh goodie, if you also implemented location types, I can try to test that for you. The thing is, it seems each manufacturer implements position reporting messages differently, so I've got to try and reverse engineer each of their messages for this service.

Other than that, you can close this issue if you want, great work here.

[lwvmobile]

Yeah, if you do test the location, just let me know. Its possible that it won't go over UDT or USBD, but rather as an embedded GPS during voice or maybe even as a PDU like LRRP or LOCN (nmea GPGGA or GPGLL, etc) reports go out. Not entirely sure, just depends on how things are set up. I don't know if you have options in your configurations for location reports as NMEA, LIP, or how and if/when to send them.

[alwinhb]

@kantooon can you share some open source references for AI protocol layer 1 and Voice and Generic Services layer implementations of DMR?

[kantooon]

Yeah, if you do test the location, just let me know. Its possible that it won't go over UDT or USBD, but rather as an embedded GPS during voice or maybe even as a PDU like LRRP or LOCN (nmea GPGGA or GPGLL, etc) reports go out. Not entirely sure, just depends on how things are set up. I don't know if you have options in your configurations for location reports as NMEA, LIP, or how and if/when to send them.

I need to experiment more to figure out this stuff. The Hytera seems to be asking for a packet data call channel to send message with location info, with a FID of 8. Clearly something manufacturer defined.

@alwinhb do you mean MS or BS? For the first one, I'd try gr-op25, for the second one MMDVM bu also gr-op25_repeater.

[alwinhb]

@kantooon do you have reference burst packets for data

for example:- 7abc3520240678e3a3436a8b55bdff57d75df5d55ed179b2304122624d0589a7bc

This is a BS-sourced data burst, It would be great if you can share some good references for MS Sourced data bursts