g4klx / MMDVMHost

The host program for the MMDVM
GNU General Public License v2.0
373 stars 271 forks source link

Support for D-Star DV Fast Data #470

Open timclassic opened 5 years ago

timclassic commented 5 years ago

I've been testing D-Star DV "Fast Data" as implemented on the Kenwood TH-D74. I tried sending data via a ZUMspot only to find that it gets corrupted on its way through (https://github.com/juribeparada/MMDVM_HS/issues/71 has an example of the corruption). MMDVMHost also reports a BER of ~12.6%.

I discovered that if I comment out all three calls to m_fec.regenerateDStar() in DStarControl.cpp, the data is able to pass through without corruption. It seems that all 72 bits of each voice frame are used for data and there is no FEC applied, which also probably explains the ~12.6% BER report.

Some things I don't know:

  1. Is there a way to tell when a voice frame contains "fast data" instead of voice+FEC?
  2. Do Icom and Kenwood implement the same fast data spec? (I hope so)
  3. Why is the FEC recalculated and reapplied in MMDVMHost? Is it so that the BER reports are effectively per-hop? Is there something important to the protocol going on?

To answer question 1, I'm planning to look for a difference in the packet that indicates whether voice or data is contained within the voice frames. I think there must be a way to tell the difference, because the demos I've seen of Icom radios doing fast data allow the user seamlessly transmit audio while a data transfer is ongoing and the radio degrades to slow data while sending voice frames. (The Kenwood does not seem to have this slow data capability)

I'm going to email Kenwood and Icom in an effort to answer question 2 (and maybe question 1?) and also ask if they have any specs to share.

I'm hoping that someone reading this can answer question 3 for me.

I'm interested in all this because I think that fast data + hotspots and the reflector system could be a powerful combination.

-TimS, KG4BXH

g4klx commented 5 years ago

Hi Tim

Fast data on normal D-Star radios is a relatively new thing and certainly wasn’t around when my software was first written. The only data mode being used then was the relatively high speed DD mode on 23cms.

Although I own two radios that do this mode, an Icom ID-51 and a Kenwood TH-D74, I have not tried it out. So my comments are based on understanding the protocol and code rather than direct experience of the mode.

So to your questions:

To create the traces you need to go into the [Modem] section of MMDVM.ini and set either Debug=1 or Trace=1 (or both, I forget, and I’m at work currently) and logging to level 1 for debug messages. You shouldn’t need much data.

There must be a way to distinguish the voice and data modes, so I await your data logging with interest. Jonathan G4KLX 

Sent from Yahoo Mail for iPhone

On Wednesday, November 7, 2018, 02:39, Tim Stewart notifications@github.com wrote:

I've been testing D-Star DV "Fast Data" as implemented on the Kenwood TH-D74. I tried sending data via a ZUMspot only to find that it gets corrupted on its way through (juribeparada/MMDVM_HS#71 has an example of the corruption). MMDVMHost also reports a BER of ~12.6%.

I discovered that if I comment out all three calls to m_fec.regenerateDStar() in DStarControl.cpp, the data is able to pass through without corruption. It seems that all 72 bits of each voice frame are used for data and there is no FEC applied, which also probably explains the ~12.6% BER report.

Some things I don't know:

To answer question 1, I'm planning to look for a difference in the packet that indicates whether voice or data is contained within the voice frames. I think there must be a way to tell the difference, because the demos I've seen of Icom radios doing fast data allow the user seamlessly transmit audio while a data transfer is ongoing and the radio degrades to slow data while sending voice frames. (The Kenwood does not seem to have this slow data capability)

I'm going to email Kenwood and Icom in an effort to answer question 2 (and maybe question 1?) and also ask if they have any specs to share.

I'm hoping that someone reading this can answer question 3 for me.

I'm interested in all this because I think that fast data + hotspots and the reflector system could be a powerful combination.

-TimS, KG4BXH

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

timclassic commented 5 years ago

Jonathan,

Now it totally makes sense why you would want to regenerate the FEC at each hop. Even though "error correction" is in the name, I was thinking of it as a mere checksum. I suppose it is lucky that the currently-installed Icom repeaters don't recalculate FEC, at least from the perspective of using fast data through them. I certainly don't have the experience to know how much it would improve repeater use if the repeaters also did the recalculation.

I will grab some traces and post the results here. Thank you for the quick guide. If I can find the flag, I should be able to put together a patch that disables the recalc only in the case that fast data is in use.

juribeparada commented 5 years ago

Yes, I know, I follow with attention

timclassic commented 5 years ago

Here are my findings so far.

In fast data mode, I've found that the TH-D74 sends the following bytes in the first 21 voice frames before starting to send the actual data:

EE C2 A1 C8 42 6E 52 51 C3

After the 21st frame, the contents of the voice frames begin to vary, presumably with the actual data I tried to send. Indeed, even when I send a single A followed by a LF, the same 21 frames appear.

Here is a trace from the log that was generated by a D-RATS transmission as it started up and advertised that I was online:

D: 2018-11-09 05:06:49.213 RX D-Star Header
D: 2018-11-09 05:06:49.213 0000:  E0 2E 10 40 00 00 4B 47 34 42 58 48 20 47 4B 47    *...@..KG4BXH GKG*
D: 2018-11-09 05:06:49.214 0010:  34 42 58 48 20 42 43 51 43 51 43 51 20 20 4B 47    *4BXH BCQCQCQ  KG*
D: 2018-11-09 05:06:49.214 0020:  34 42 58 48 20 20 44 37 34 20 19 8A 00 2F          *4BXH  D74 .../*
D: 2018-11-09 05:06:49.214 D-Star, raw RSSI: 47, reported RSSI: -47 dBm
M: 2018-11-09 05:06:49.215 D-Star, received RF header from KG4BXH  /D74  to CQCQCQ
M: 2018-11-09 05:06:49.221 Debug: Mode set to D-Star
D: 2018-11-09 05:06:49.232 RX D-Star Data
D: 2018-11-09 05:06:49.232 0000:  E0 11 11 EE C2 A1 C8 42 6E 52 51 C3 55 2D 16 00    *.......BnRQ.U-..*
D: 2018-11-09 05:06:49.232 0010:  2F                                                 */*
D: 2018-11-09 05:06:49.232 D-Star, raw RSSI: 47, reported RSSI: -47 dBm
D: 2018-11-09 05:06:49.249 RX D-Star Data
D: 2018-11-09 05:06:49.249 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 30 1B FA       *.......BnRQ.0..*
D: 2018-11-09 05:06:49.271 RX D-Star Data
D: 2018-11-09 05:06:49.271 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 1D 6F FA       *.......BnRQ..o.*
D: 2018-11-09 05:06:49.288 RX D-Star Data
D: 2018-11-09 05:06:49.289 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 31 21 B3       *.......BnRQ.1!.*
D: 2018-11-09 05:06:49.312 RX D-Star Data
D: 2018-11-09 05:06:49.313 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 3D 2E FD       *.......BnRQ.=..*
D: 2018-11-09 05:06:49.329 RX D-Star Data
D: 2018-11-09 05:06:49.329 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 32 27 F2       *.......BnRQ.2'.*
D: 2018-11-09 05:06:49.351 RX D-Star Data
D: 2018-11-09 05:06:49.351 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 04 3B F2       *.......BnRQ..;.*
D: 2018-11-09 05:06:49.373 RX D-Star Data
D: 2018-11-09 05:06:49.374 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 33 21 B3       *.......BnRQ.3!.*
D: 2018-11-09 05:06:49.391 RX D-Star Data
D: 2018-11-09 05:06:49.393 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 50 6F B3       *.......BnRQ.Po.*
D: 2018-11-09 05:06:49.411 RX D-Star Data
D: 2018-11-09 05:06:49.413 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.432 RX D-Star Data
D: 2018-11-09 05:06:49.433 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.451 RX D-Star Data
D: 2018-11-09 05:06:49.453 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.470 RX D-Star Data
D: 2018-11-09 05:06:49.472 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.489 RX D-Star Data
D: 2018-11-09 05:06:49.491 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.509 RX D-Star Data
D: 2018-11-09 05:06:49.510 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.529 RX D-Star Data
D: 2018-11-09 05:06:49.530 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.549 RX D-Star Data
D: 2018-11-09 05:06:49.550 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.573 RX D-Star Data
D: 2018-11-09 05:06:49.574 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.591 RX D-Star Data
D: 2018-11-09 05:06:49.593 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.610 RX D-Star Data
D: 2018-11-09 05:06:49.612 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.630 RX D-Star Data
D: 2018-11-09 05:06:49.631 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 16 29 F5       *.......BnRQ..).*
D: 2018-11-09 05:06:49.650 RX D-Star Data
D: 2018-11-09 05:06:49.652 0000:  E0 11 11 0E 31 ED 3E 66 0A 13 4E 0A 55 2D 16 00    *....1.>f..N.U-..*
D: 2018-11-09 05:06:49.654 0010:  2F                                                 */*
D: 2018-11-09 05:06:49.656 D-Star, raw RSSI: 47, reported RSSI: -47 dBm
D: 2018-11-09 05:06:49.669 RX D-Star Data
D: 2018-11-09 05:06:49.670 0000:  E0 0F 11 2D 6D AE 00 66 49 2D 0D 6B EC 14 C0       *...-m..fI-.k...*
D: 2018-11-09 05:06:49.689 RX D-Star Data
D: 2018-11-09 05:06:49.690 0000:  E0 0F 11 4D 71 A4 88 66 49 2D 38 0A EC 00 D1       *...Mq..fI-8....*
D: 2018-11-09 05:06:49.709 RX D-Star Data
D: 2018-11-09 05:06:49.710 0000:  E0 0F 11 0E 31 ED 41 66 75 6C 31 2A E4 31 ED       *....1.Aful1*.1.*
D: 2018-11-09 05:06:49.729 RX D-Star Data
D: 2018-11-09 05:06:49.730 0000:  E0 0F 11 71 4E 92 1B 66 31 22 72 76 E4 31 ED       *...qN..f1"rv.1.*
D: 2018-11-09 05:06:49.753 RX D-Star Data
D: 2018-11-09 05:06:49.754 0000:  E0 0F 11 2D 92 AE 00 66 49 2D 31 2E E4 14 C0       *...-...fI-1....*
D: 2018-11-09 05:06:49.772 RX D-Star Data
D: 2018-11-09 05:06:49.774 0000:  E0 0F 11 D0 0F AE 00 66 6C 26 77 1F E4 00 D1       *.......fl&w....*
D: 2018-11-09 05:06:49.792 RX D-Star Data
D: 2018-11-09 05:06:49.794 0000:  E0 0F 11 0E 0C C2 03 66 25 2E 61 55 E4 0D CB       *.......f%.aU...*
D: 2018-11-09 05:06:49.812 RX D-Star Data
D: 2018-11-09 05:06:49.813 0000:  E0 0F 11 0E 37 49 73 66 80 A2 FB E2 E4 07 ED       *....7Isf.......*
D: 2018-11-09 05:06:49.832 RX D-Star Data
D: 2018-11-09 05:06:49.833 0000:  E0 0F 11 00 9E 9E 32 66 78 64 E6 2F E4 83 D8       *......2fxd./...*
D: 2018-11-09 05:06:49.851 RX D-Star Data
D: 2018-11-09 05:06:49.853 0000:  E0 0F 11 4D 0F B9 CC 66 70 DE 6B 6E E4 1A 43       *...M...fp.kn..C*
D: 2018-11-09 05:06:49.870 RX D-Star Data
D: 2018-11-09 05:06:49.871 0000:  E0 0F 11 16 29 F5 26 66 12 0B 56 4D F3 00 D1       *....).&f..VM...*
D: 2018-11-09 05:06:49.889 RX D-Star Data
D: 2018-11-09 05:06:49.890 0000:  E0 0F 11 16 29 F5 26 66 12 0B 56 4D F3 12 F5       *....).&f..VM...*
D: 2018-11-09 05:06:49.909 RX D-Star Data
D: 2018-11-09 05:06:49.910 0000:  E0 0F 11 EE C2 A1 C8 42 6E 52 51 C3 55 55 55       *.......BnRQ.UUU*
M: 2018-11-09 05:06:49.916 Debug: DStarRX: Found end sync in Data
D: 2018-11-09 05:06:49.923 RX D-Star EOT
D: 2018-11-09 05:06:49.925 0000:  E0 03 13                                           *...*
M: 2018-11-09 05:06:49.926 D-Star, received RF end of transmission, 0.7 seconds, BER: 0.0%, RSSI: -47/-47/-47 dBm

So, my theory is that there is something special about EE C2 A1 C8 42 6E 52 51 C3, such that it won't be generated by AMBE+FEC, or perhaps it's just extremely unlikely to occur 21 times in a row. Further, I imagine that 9-byte sequence itself validates against FEC (just guessing, I don't yet know how to check this).

Anyway, if my theory is correct, we should be able to support fast data through hotspots (and repeaters?) by turning off FEC recalculation in the case that we see 21 of these in a row.

I wonder if there's a "re-enable voice" sequence that indicates we should re-enable FEC, supporting Icom's apparent ability to move between the two modes in a single transmission. (I say "apparent" because I think this is possible based on the Icom video that demos text and image sending while also being able to send voice during the transfer). Maybe that's what 16 29 F5 26 66 12 0B 56 4D is for?

timclassic commented 5 years ago

I should note that this dump was created by MMDVMHost with the FEC recalculation disabled, which is why you see a 0% BER.

Also, I said above that "I imagine that 9-byte sequence itself validates against FEC." If memory serves, these bytes contributed to the 12.6% BER in my original tests, so I think my statement is wrong. I'll double-check and confirm, though I don't think it affects my theory much overall.

g4klx commented 5 years ago

That is very very interesting. What you say makes sense. I think I can move forwards with what you have told me. If you could investigate the Icom mode switch, and also if Icom use the same system, I would appreciate it. I have some ideas and I'll try and implement them next week. Unfortunately the gateway will also need modifying which is not ideal, but necessary.

juribeparada commented 5 years ago

Hi, I have done some tests with the same file (sent.log) and my ID51 with Fast DV data. Unfortunately it looks different from Kenwood format:


D: 2018-11-11 23:54:54.411 RX D-Star Header
D: 2018-11-11 23:54:54.411 0000:  E0 2E 10 40 00 00 43 41 36 4A 41 55 20 47 43 41    *...@..CA6JAU GCA*
D: 2018-11-11 23:54:54.411 0010:  36 4A 41 55 20 43 20 20 20 20 20 20 20 45 43 41    *6JAU C       ECA*
D: 2018-11-11 23:54:54.411 0020:  36 4A 41 55 20 20 20 20 20 20 56 C0 00 04          *6JAU      V...*
M: 2018-11-11 23:54:54.411 D-Star, received RF header from CA6JAU  /     to        E
D: 2018-11-11 23:54:54.430 RX D-Star Data
D: 2018-11-11 23:54:54.430 0000:  E0 11 11 B2 4D 22 48 C0 16 28 26 C8 55 2D 16 00    *....M"H..(&.U-..*
D: 2018-11-11 23:54:54.430 0010:  04                                                 *.*
D: 2018-11-11 23:54:54.449 RX D-Star Data
D: 2018-11-11 23:54:54.449 0000:  E0 0F 11 AE CC 2A 78 E1 13 3C 67 C0 30 0E FD       *.....*x..<g.0..*
D: 2018-11-11 23:54:54.468 RX D-Star Data
D: 2018-11-11 23:54:54.468 0000:  E0 0F 11 CE 8E 2E 39 66 12 78 31 B0 14 36 B3       *......9f.x1..6.*
D: 2018-11-11 23:54:54.492 RX D-Star Data
D: 2018-11-11 23:54:54.492 0000:  E0 0F 11 AE CC 2A 78 E1 91 34 67 C0 31 1A E1       *.....*x..4g.1..*
D: 2018-11-11 23:54:54.511 RX D-Star Data
D: 2018-11-11 23:54:54.511 0000:  E0 0F 11 CA 8E 26 19 26 1E 4C A1 F0 19 2D F6       *.....&.&.L...-.*
D: 2018-11-11 23:54:54.529 RX D-Star Data
D: 2018-11-11 23:54:54.529 0000:  E0 0F 11 AE CC 2A 78 E1 13 3C 67 C0 32 6F BC       *.....*x..<g.2o.*
D: 2018-11-11 23:54:54.547 RX D-Star Data
D: 2018-11-11 23:54:54.547 0000:  E0 0F 11 BE 6E 22 48 23 1B 0C 77 80 50 06 D7       *....n"H#..w.P..*
D: 2018-11-11 23:54:54.572 RX D-Star Data
D: 2018-11-11 23:54:54.572 0000:  E0 0F 11 CE 8E 22 09 E4 1E 4C 20 F4 33 7A A2       *....."...L .3z.*
D: 2018-11-11 23:54:54.590 RX D-Star Data
D: 2018-11-11 23:54:54.590 0000:  E0 0F 11 AA CC 2A 78 E1 91 3C 67 C0 35 6F B3       *.....*x..<g.5o.*
D: 2018-11-11 23:54:54.612 RX D-Star Data
D: 2018-11-11 23:54:54.612 0000:  E0 0F 11 CE CE AC 31 47 1E 58 E0 F0 16 29 F5       *......1G.X...).*
D: 2018-11-11 23:54:54.629 RX D-Star Data
D: 2018-11-11 23:54:54.629 0000:  E0 0F 11 AE 8C 2E 68 22 1F 1C 37 84 16 29 F5       *......h"..7..).*
D: 2018-11-11 23:54:54.651 RX D-Star Data
D: 2018-11-11 23:54:54.651 0000:  E0 0F 11 50 26 E0 60 66 15 4D 44 4E E4 1B FB       *...P&.`f.MDN...*
D: 2018-11-11 23:54:54.668 RX D-Star Data
D: 2018-11-11 23:54:54.668 0000:  E0 0F 11 03 3B B3 2F 66 12 4D 44 43 E4 26 E0       *....;./f.MDC.&.*
D: 2018-11-11 23:54:54.691 RX D-Star Data
D: 2018-11-11 23:54:54.691 0000:  E0 0F 11 15 3D F4 25 66 1A 0E 49 0B E4 2A B3       *....=.%f..I..*.*
D: 2018-11-11 23:54:54.708 RX D-Star Data
D: 2018-11-11 23:54:54.708 0000:  E0 0F 11 12 3D FC 21 66 10 0E 51 58 E4 2A FE       *....=.!f..QX.*.*
D: 2018-11-11 23:54:54.732 RX D-Star Data
D: 2018-11-11 23:54:54.732 0000:  E0 0F 11 03 3B F6 2D 66 5A 4D 10 7F E4 3B B3       *....;.-fZM...;.*
D: 2018-11-11 23:54:54.751 RX D-Star Data
D: 2018-11-11 23:54:54.751 0000:  E0 0F 11 18 26 E0 60 66 1D 1E 10 44 E4 3C EA       *....&.`f...D.<.*
D: 2018-11-11 23:54:54.768 RX D-Star Data
D: 2018-11-11 23:54:54.768 0000:  E0 0F 11 11 6F E7 25 66 07 19 1E 26 FD 21 FF       *....o.%f...&.!.*
D: 2018-11-11 23:54:54.793 RX D-Star Data
D: 2018-11-11 23:54:54.793 0000:  E0 0F 11 7A 4F 93 40 66 74 6D 30 2B FD 36 B3       *...zO.@ftm0+.6.*
D: 2018-11-11 23:54:54.811 RX D-Star Data
D: 2018-11-11 23:54:54.811 0000:  E0 0F 11 CE 8E A0 01 C4 1E 4C 20 F4 16 29 F5       *.........L ..).*
D: 2018-11-11 23:54:54.829 RX D-Star Data
D: 2018-11-11 23:54:54.829 0000:  E0 0F 11 AA CC 2E 68 23 91 3C E6 C4 16 29 F5       *......h#.<...).*
D: 2018-11-11 23:54:54.852 RX D-Star Data
D: 2018-11-11 23:54:54.852 0000:  E0 0F 11 CE 8E 2E 39 66 12 78 31 B0 55 55 55       *......9f.x1.UUU*
D: 2018-11-11 23:54:54.852 RX D-Star EOT
D: 2018-11-11 23:54:54.852 0000:  E0 03 13                                           *...*
M: 2018-11-11 23:54:54.858 D-Star, received RF end of transmission, 0.5 seconds, BER: 0.0%
juribeparada commented 5 years ago

I've realized that not always starts with the same data sequence, after repeating the same transmission with a ID51. For example, if I transmit a single "A" (I "cleaned" the log for better understanding):

70 4F 93 40 64 74 6D 30 2B 55 2D 16       *...pO.@dtm0+U-.*
70 4F 93 40 64 74 6D 30 2B F2 0E 99       *...pO.@dtm0+...*
70 4F 93 40 64 74 6D 30 2B F2 4F 93       *...pO.@dtm0+.O.*
CE 8E 2E 39 66 12 78 31 B0 30 0E FD       *......9f.x1.0..*
AE CC 2A 78 E1 91 34 67 C0 14 36 B3       *.....*x..4g..6.*
CA 8E 26 19 26 1E 4C A1 F0 31 1A E1       *.....&.&.L..1..*
AE CC 2A 78 E1 13 3C 67 C0 19 2D F6       *.....*x..<g..-.*
BE 6E 22 48 23 1B 0C 77 80 32 6F BC       *....n"H#..w.2o.*
CE 8E 22 09 E4 1E 4C 20 F4 50 06 D7       *....."...L .P..*
AA CC 2A 78 E1 91 3C 67 C0 33 7A A2       *.....*x..<g.3z.*
CE CE AC 31 47 1E 58 E0 F0 35 6F B3       *......1G.X..5o.*
AE 8C 2E 68 22 1F 1C 37 84 16 29 F5       *......h"..7..).*
CE CE 2A 29 A5 1E 58 61 F4 16 29 F5       *.....*)..Xa..).*
AA CC 2A 78 E1 13 3C 67 C0 55 55 55       *.....*x..<g.UUU*

And if I repeat the same "A":

B2 4D 22 48 C0 16 28 26 C8 55 2D 16       *....M"H..(&.U-.*
AE CC 2A 78 E1 13 3C 67 C0 30 0E FD       *.....*x..<g.0..*
CE 8E 2E 39 66 12 78 31 B0 14 36 B3       *......9f.x1..6.*
AE CC 2A 78 E1 91 34 67 C0 31 1A E1       *.....*x..4g.1..*
CA 8E 26 19 26 1E 4C A1 F0 19 2D F6       *.....&.&.L...-.*
AE CC 2A 78 E1 13 3C 67 C0 32 6F BC       *.....*x..<g.2o.*
BE 6E 22 48 23 1B 0C 77 80 50 06 D7       *....n"H#..w.P..*
CE 8E 22 09 E4 1E 4C 20 F4 33 7A A2       *....."...L .3z.*
AA CC 2A 78 E1 91 3C 67 C0 35 6F B3       *.....*x..<g.5o.*
CE CE AC 31 47 1E 58 E0 F0 16 29 F5       *......1G.X...).*
AE 8C 2E 68 22 1F 1C 37 84 16 29 F5       *......h"..7..).*
70 4F 93 40 64 74 6D 30 2B F2 0E 99       *...pO.@dtm0+...*
70 4F 93 40 64 74 6D 30 2B F2 4F 93       *...pO.@dtm0+.O.*
CE CE 26 19 27 12 6C 70 B0 16 29 F5       *.....&.'.lp..).*
AE EC 2A 78 E1 13 3C 67 C0 16 29 F5       *.....*x..<g..).*
CE CE 2E 39 67 1E 58 E0 F0 55 55 55       *......9g.X..UUU*
juribeparada commented 5 years ago

I did more experiments, trying to decode the frames. Now I transmitted: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"

I got: fastdv_icom

My guess: A: a kind of data start?. It seems to me always the same, but for some transmissions is not there! B: sync flag C: data header?, it looks always the same and it is always there. It doesn't change if I modify D-Star fields, for example. D: Slow data, here has my TX message "Andy Uribe / ID51E " E: data payload

The data payload seems to be scrambled with: 70 4F 93 40 64 74 6D 30 2B E2 4F 93 (not sure about 64 and E2). If I de-scramble the payload:

45 46 47 48 02 49 4A 4B 4C 06 41 42                *EFGH.IJKL.AB*
4D 4E 4F 50 02 51 52 53 54 06 43 44                *MNOP.QRST.CD*
59 5A 61 62 02 63 64 65 66 06 55 56                *YZab.cdef.UV*
67 68 69 6A 02 6B 6C 6D 6E 06 57 58                *ghij.klmn.WX*
73 74 75 76 02 77 78 79 7A 06 6F 70                *stuv.wxyz.op*
30 31 32 33 02 34 35 36 37 06 71 72                *0123.4567.qr*
00 00 00 00 02 00 00 00 00 10 38 39                *..........89*
00 00 00 00 02 00 00 00 00 10 00 00                *............*
g4klx commented 5 years ago

It looks like the Kenwood data is the easiest to support. I think the data marker is transmitted multiple times to make sure that you receive it. You'll find that 21 frames occurs a lot in D-Star and is the distance between sync frames, so it sort of makes sense to use that as the basic unit of data transmissions. Once I have reliably detected the data mode, I'd just switch off the FEC code and let it continue. I'd probably also regenerate the data marker also for good measure. I'm still looking at the Icom version. In theory switching off the FEC will be fine if the system switches back to voice, you just won't get the benefit of the FEC for the rest of that transmission. I do wish that Kenwood and Icom had used a bit in the header instead or as well as, to make my life easier.

juribeparada commented 5 years ago

I am going to repeat the same tests with a ID-4100A and see if I get the same results. Something that I forget to mention: after all my tests with ID-51A, the first frame could be one of the following two possibilities: 1) The sequence: B2 4D 22 48 C0 16 28 26 C8 2) Directly an "almost" empty payload: 70 4F 93 40 64 74 6D 30 2B, where the following slow data will contain the first 4 bytes of the message.

Anyway, it seems to me that the large structure "C" is always in any Icom fast DV transmission.

timclassic commented 5 years ago

Unfortunately, I don't have any Icom hardware to test with :( I'm hoping to get an Icom ID-4100A by the end of the year, though.

I have received an initial response from my email to Kenwood:

The last point could mean the D74A implements a simpler subset, I suppose, since it doesn't support switching back and forth between fast and slow.

I haven't gotten a negative response from either Icom or Kenwood support about getting tech specs, so I'll update here when I have any answers there.

Maybe there is some shared property of the bytes I saw (EE C2 A1 C8 42 6E 52 51 C3) and block C of @juribeparada's dump, e.g. some checksum they both validate against, and Kenwood just implemented something simpler. But I'm just shooting in the dark here :)

Anyway, I plan to poke around a bit more with the TH-D7A this week and compare to the other posts here, in case I can discover anything.

Thanks for looking at this!

g4klx commented 5 years ago

I've added some code to the D-Star support in the host. It should detect the use of fast data mode, and log it. It should work on RF and network although the ircDDB Gateway needs to be modified before we can send fast data over any network. For now, if you can test/modify the code that I've added for the fast data mode.

juribeparada commented 5 years ago

I will test at night here

juribeparada commented 5 years ago

I had little time for testing today, but I could check: 1) ID-4100A sends the same frames as ID-51 in fast DV mode, with the two "identifiers". 2) The current MMDVMHost detects both Icom identifiers well, and works well with ID-51 and ID-4100A in fast DV data.

g4klx commented 5 years ago

I've added data marker regeneration on RF and network traffic. I've checked the ircDDB Gateway and it does no AMBE FEC regeneration so fast data should pass through it without problems. The only potential issue is the DTMF decoder which could trigger and blank the data. This will need testing to make sure what goes in, comes out unchanged. I'm hoping that we can get this finished soon.

g4klx commented 5 years ago

BTW in your data, I wonder if element C is your RF header, scrambled. The only way to be sure would be for someone else to generate some Icom fast data with a dissimilar set of callsigns and do a comparison.

timclassic commented 5 years ago

I will test the changes tonight.

juribeparada commented 5 years ago

It seems to me the block "C" doesn't change: I used different D-Star fields YOUR, RPT1, etc (except the callsign, I've used always the same), even I transmitted accidentally in simplex with the 4100, and I captured the same "C" structure, something curious. I will be out of the town for 10 days, then I will be not able to do more tests until come back.

timclassic commented 5 years ago

The changes look good to me so far. I was able to send and receive the same text using an echo test, and I get the Kenwood message in the logs.

Now I'm hanging out on REF030D in D-RATS if either of you are around and want to test.

timclassic commented 5 years ago

I've shared a file via D-RATS in case one of you decide to test via REF030D while I am away. I realize this may not work yet, but I thought it might be fun to try.

Goodnight!

-TimS, KG4BXH

g4klx commented 5 years ago

That's very pleasing. I am still worried about the DTMF processing in the gateway which could potentially corrupt the data if triggered by random data. I will look to switch it off in data mode. I also need to think about the older D-Star Repeater software, whether to upgrade it or not. It will be interesting if there is a switch in the Icom data mode to go back to audio, as things stand it should work anyway, but any following audio won't have the extra processing that audio gets now. Sorry I wasn't on D-RATS but I live in the UK and the time difference killed that, as well as other things.

timclassic commented 5 years ago

I suspected you were in the UK and it was a long shot :) Also, my pi-star decided to restart all services at 3:30am EST anyway, so that killed the test after a while... need to turn that off. So, to be clear, my D-RATS test is now over, heh.

Anyway, I'm still waiting on potential docs from Kenwood and Icom. That would be really nice if one of them could just tell us how it works. I'll see if I can find someone willing to test with me, given that I have only one radio.

phl0 commented 5 years ago

I think I could help out with testing. I have a ID51 at least but neeed to figure out how fast data works.

g4klx commented 5 years ago

I'm in the same boat Flo. I have a TH-D74 and an ID-51 and haven't used fast data on either of them. That's why it's never been added to my work before. I've updated the ircDDB Gateway to bypass all DTMF handling when fast data is detected, but it's on a branch because I can't build it currently. It'll be rolled into master when it's been compiled cleanly.

phl0 commented 5 years ago

Ok deal. I will try to set this up with the gateway from the branch and test.

juribeparada commented 5 years ago

In my case, I enabled Fast DV data from ID-51 menu (DV Set->DV Fast Data->Fast Data->ON), and then simply I send the data using the serial port of the radio at 38400 bps:

echo "Test DV fast data" > /dev/ttyXXX cat file.txt > /dev/ttyXXX

If fast data is OFF, all data will be sent as slow data as usual.

phl0 commented 5 years ago

I finally have my fast data equipment up and running. Sorry for the delay. Anyone up for a test (across reflectors)?

ravny commented 5 years ago

Hi, yes we can do the test just suggest which ref and time.

timclassic commented 5 years ago

@phl0 @ravny If anyone is around this weekend, I have D-RATS listening again on REF030D, and I'm serving up a text file if you browse my files.

ravny commented 5 years ago

@timclassic Hi, Im connected to REF030D

timclassic commented 5 years ago

@ravny I was offline for a few hours (not sure how many--the battery powering the LTE+D-Star hotspots finally ran out) but everything is set back up (on mains) and I've reconnected to REF030D.

I'll be in and out today, but I've left that file shared so someone connecting should be able to test even if I'm not at my desk.

phl0 commented 5 years ago

Just testing right now. But it won't connect for some reason. Checking d-rats settings ...

phl0 commented 5 years ago

bildschirmfoto vom 2018-11-18 21-06-26

phl0 commented 5 years ago

bildschirmfoto von 2018-11-18 21-41-46

phl0 commented 5 years ago

bildschirmfoto von 2018-11-18 21-45-36

timclassic commented 5 years ago

@phl0 Thanks for the test! I just turned off my hotspot and radio, because it appears my Kenwood got stuck transmitting--the PTT light was stuck on and a second radio monitoring my hotspot frequency showed a silent carrier, even after unplugging the hotspot. I've seen this happen outside of the test before, so it's likely either a TH-D74 bug or a D-RATS bug/misconfiguration.

I was able to see most of your messages, though some from your side appeared scrambled, and I was sad that I lost the transcript by restarting. However, the screenshot you just uploaded is very similar to what I saw, so I suspect it demonstrates the issue nicely.

Also, I'm not sure why you couldn't see me on the map. Maybe my position packets weren't making it through unscathed? Or maybe I'm not configured to transmit them?

timclassic commented 5 years ago

Oh, and regarding the scrambled messages: I saw at least one instance where MMDVMHost didn't switch into Icom data mode (perhaps the magic byte string didn't make it?) and other instances where it did switch to Icom data mode, yet the data still appeared garbled.

Unfortunately I didn't do a very good job preserving the logs, but I'll post evidence here if I can find it.

phl0 commented 5 years ago

Not sure. S55IAR came in after you went. He seems to broadcast GPS positions. I could immediately see him on the map as shown above. But it seems he cannot read my messages to the chat (yet).

timclassic commented 5 years ago

@phl0 Perhaps S55IAR is using an Icom that doesn't understand fast data?

phl0 commented 5 years ago

Lets see what @ravny says :)

ravny commented 5 years ago

HI Im not shure why I can not read messages currently I have connected ID51 and fast data ON and 38400 for the com port speed Any suggestion what to check ?

phl0 commented 5 years ago

Yepp. Set com port speed to 9k6. I wasn't able to communicate with this setting at 38k4.

ravny commented 5 years ago

OK and on the radio you leave FAST DATA ON ?

phl0 commented 5 years ago

yes. That is my setting with an ID-51 Anniversary Edition.

ravny commented 5 years ago

Now I have fast data on and 9,6k com port speed I can niot read anything image

timclassic commented 5 years ago

@ravny I sent a few messages, but messages I see from your end appear garbled. Do you see the "D-Star, switching to data mode (Icom)" message in your MMDVMHost logs?

ravny commented 5 years ago

Yes u right MMDVM wont switch to data mode.

timclassic commented 5 years ago

If you're sure that you are running the patched version that is capable of the data mode switch, it would be interesting to see a dump of your transmission. Maybe there is an additional sequence of bytes that needs to be added to the code, or something.

I had to set the following to MMDVM.ini:

[Log]
FileLevel=1

[Modem]
Trace=1
Debug=1

I'm not sure that Debug=1 is necessary, but the other two are.

g4klx commented 5 years ago

It seems that the support for Icom fast data is not quite there yet. This thread of tweets may help us: https://twitter.com/isithran/status/1063164783342641152