search

JvanKatwijk / dab-cmdline

DAB decoding library with example of its use
GNU General Public License v2.0
57 stars 29 forks source link

Main TII of 0x16 decodes as 0xff #91

Closed srcejon closed 1 year ago

srcejon commented 1 year ago

A user in France reported a main TII of what should be 0x14 decoding as 0xff. I've found the same with a TII of 0x16 in the UK on 215.072MHz near London.

Here's a dump of some of the data computed in TII_Detector::processNULL

hulpTable[] = {0.033322,0.032229,0.017495,0.003001,0.015740,0.005635,0.005190,0.014116,0.015232,0.017211,0.071607,0.050573,0.001726,0.006226,0.030202,0.027032,0.013247,0.015970,0.046357,0.003054,0.036548,0.014234,0.041270,0.013134,0.048222,0.534342,0.036374,0.013863,0.017658,0.316324,0.054635,0.013134,0.007199,0.006602,0.006587,0.018351,0.021258,0.024930,0.020262,0.006384,0.024990,0.046117,2.992398,0.112000,0.039069,0.004858,0.037524,0.046304,0.009315,0.007271,0.024175,0.039459,0.012557,0.030397,0.018799,0.031979,0.021991,0.011365,0.016714,0.006930,0.004808,0.026931,0.015936,0.027846,0.028176,0.017205,0.015438,0.023799,0.004915,0.028892,0.031529,0.036843,0.019947,0.453491,0.092666,0.023731,0.071105,0.268657,0.052420,0.006335,0.016436,0.008851,0.021607,0.034120,0.016978,0.037421,0.102918,0.015863,0.014173,0.146825,3.385903,0.068001,0.030084,0.007933,0.005267,0.008046,0.112886,0.615365,0.024125,0.006107,0.063406,0.058142,0.047537,0.046570,0.016502,0.031814,0.043064,0.038356,0.007928,0.006931,0.046983,0.075101,0.008154,0.136534,1.859743,0.079350,0.044245,0.035166,0.048226,0.037586,0.042892,0.016159,0.019534,0.018744,0.002744,0.022694,0.044072,0.036351,0.009865,0.053853,0.052355,0.012978,0.023112,0.010522,0.043566,0.052566,0.013896,0.016469,0.005868,0.019383,0.012732,0.021551,0.047526,0.013804,0.017613,0.038883,0.019552,0.007354,0.020700,0.031689,0.012262,0.007234,0.025951,0.017694,0.005677,0.012981,0.032183,0.053012,0.039971,0.005391,0.010107,0.004228,0.010894,0.012265,0.009924,0.030246,0.022637,0.008334,0.013652,0.611818,0.033609,0.004411,0.032916,0.070446,0.026055,0.037627,0.012676,0.037599,0.019006,0.005473,0.013352,0.014759,0.054502,0.005546,0.007376,0.062512,2.749999,0.063616,0.012755,0.003153,0.019487,0.038988,}; avgTable[] = {0.022098, 0.185391, 0.020553, 0.204533, 0.145409, 0.025552, 0.019033, 0.164639, }; C_table[] = {0.000000, 0.615365, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 10.988042, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, }; D_table[] = {0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, }; maxIndex 18 x[] = {0.046357, 2.992398, 0.015438, 3.385903, 1.859743, 0.005868, 0.010894, 2.749999, }; pattern 0x59 89 invtable[89] 0xff

pattern is repeatedly 0x59

The sub ID is correct.

JvanKatwijk commented 1 year ago

Any chance that I get a recording giving the questionable output?

Op ma 27 feb 2023 om 12:42 schreef srcejon @.***>:

A user in France reported a main TII of what should be 0x16 decoding as 0xff. I've found the same in the UK on 215.072MHz near London.

Here's a dump of some of the data computed in TII_Detector::processNULL

hulpTable[] = {0.033322,0.032229,0.017495,0.003001,0.015740,0.005635,0.005190,0.014116,0.015232,0.017211,0.071607,0.050573,0.001726,0.006226,0.030202,0.027032,0.013247,0.015970,0.046357,0.003054,0.036548,0.014234,0.041270,0.013134,0.048222,0.534342,0.036374,0.013863,0.017658,0.316324,0.054635,0.013134,0.007199,0.006602,0.006587,0.018351,0.021258,0.024930,0.020262,0.006384,0.024990,0.046117,2.992398,0.112000,0.039069,0.004858,0.037524,0.046304,0.009315,0.007271,0.024175,0.039459,0.012557,0.030397,0.018799,0.031979,0.021991,0.011365,0.016714,0.006930,0.004808,0.026931,0.015936,0.027846,0.028176,0.017205,0.015438,0.023799,0.004915,0.028892,0.031529,0.036843,0.019947,0.453491,0.092666,0.023731,0.071105,0.268657,0.052420,0.006335,0.016436,0.008851,0.021607,0.034120,0.016978,0.037421,0.102918,0.015863,0.014173,0.146825,3.385903,0.068001,0.030084,0.007933,0.005267,0.008046,0.112886,0.615365,0.024125,0.006107,0.063406,0.058142,0.047537,0.046570,0.016502,0.031814,0.043064,0.038356,0.007928,0.006931,0.046983,0.075101,0.008154,0.136534,1.859743,0.079350,0.044245,0.035166,0.048226,0.037586,0.042892,0.016159,0.019534,0.018744,0.002744,0.022694,0.044072,0.036351,0.009865,0.053853,0.052355,0.012978,0.023112,0.010522,0.043566,0.052566,0.013896,0.016469,0.005868,0.019383,0.012732,0.021551,0.047526,0.013804,0.017613,0.038883,0.019552,0.007354,0.020700,0.031689,0.012262,0.007234,0.025951,0.017694,0.005677,0.012981,0.032183,0.053012,0.039971,0.005391,0.010107,0.004228,0.010894,0.012265,0.009924,0.030246,0.022637,0.008334,0.013652,0.611818,0.033609,0.004411,0.032916,0.070446,0.026055,0.037627,0.012676,0.037599,0.019006,0.005473,0.013352,0.014759,0.054502,0.005546,0.007376,0.062512,2.749999,0.063616,0.012755,0.003153,0.019487,0.038988,}; avgTable[] = {0.022098, 0.185391, 0.020553, 0.204533, 0.145409, 0.025552, 0.019033, 0.164639, }; C_table[] = {0.000000, 0.615365, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 10.988042, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, }; D_table[] = {0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, }; maxIndex 18 x[] = {0.046357, 2.992398, 0.015438, 3.385903, 1.859743, 0.005868, 0.010894, 2.749999, }; pattern 0x59 89 invtable[89] 0xff

pattern is repeatedly 0x59, but presumably should be 0x55 to map to Main Tii of 0x16, which suggests x[4] and x[5] are the wrong way around.

— Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/dab-cmdline/issues/91, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCPHQGCAAYRCE3TEEKSJCLWZSHJNANCNFSM6AAAAAAVJJHS5M . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Jan van Katwijk

srcejon commented 1 year ago

Sure: https://sdrangel.org/downloads/dab_tii_2023-02-28T13_02_10_458.zip

JvanKatwijk commented 1 year ago

There is a slight problem, I cannot read the file. Very strange, VLC can read it (but that does not help) and neither the windows version of qt-dab, the fedora version nor the ubuntu version recognize the file as legitimate wav file

Op di 28 feb 2023 om 14:06 schreef srcejon @.***>:

Sure: https://sdrangel.org/downloads/dab_tii_2023-02-28T13_02_10_458.zip

— Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/dab-cmdline/issues/91#issuecomment-1448146981, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCPHQDPYW4QTD7EHH7X353WZXZ4LANCNFSM6AAAAAAVJJHS5M . You are receiving this because you commented.Message ID: @.***>

-- Jan van Katwijk

andimik commented 1 year ago

Confirmed, very strange.

Qt-DAB even says, "File not found" when I change the extension to *.sdr (assuming it's wav).

Mediainfo says,

Complete name                            : /home/andreas/Downloads/test.sdr
Format                                   : Wave
File size                                : 37.9 MiB
Duration                                 : 4 s 856 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 65.5 Mb/s
FileExtension_Invalid                    : act at9 wav

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 4 s 856 ms
Bit rate mode                            : Constant
Bit rate                                 : 65.5 Mb/s
Channel(s)                               : 2 channels
Sampling rate                            : 2 048 kHz
Bit depth                                : 16 bits
Stream size                              : 37.9 MiB (100%)

eti-cmdline says

./eti-cmdline-wavfiles -F ~/Downloads/test.sdr 
/home/andreas/Downloads/test.sdr is not a recorded dab file, sorry
Installing device failed, fatal25
srcejon commented 1 year ago

It"s a 16-bit 2.048MSa/s IQ .wav file recorded in SDRangel. I'll have to build qt-dab to see what it needs.

JvanKatwijk commented 1 year ago

can you just build a "raw" file when using a dabstick? I know there are some(times) issues with the distributions of the libsndfile lib.

Op wo 1 mrt 2023 om 19:55 schreef srcejon @.***>:

It"s a 16-bit 2.048MSa/s IQ .wav file recorded in SDRangel. I'll have to build qt-dab to see what it needs.

— Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/dab-cmdline/issues/91#issuecomment-1450692315, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCPHQHMKJENBAKBN7GT44TWZ6LTHANCNFSM6AAAAAAVJJHS5M . You are receiving this because you commented.Message ID: @.***>

-- Jan van Katwijk

andimik commented 1 year ago

Oh, indeed, it's a 16bit 2ch wave file, Jan.

I checked under Windows. It's very difficult to lock. Left and right there are very strong signals.

Needed to use a bw filter.

I get

0x59 22 18
0x59 22 01 Screenshot (389)

    QIRX 4.0.7.0    TII Logger started at 2023_03_01 19:07:17 UTC
    All times in the log are UTC as received from DAB!

Date/Time               Chn EId     Label               MER     Thrsh   Patt    M_Id    S_Id    Stren   km abs  km rel  Bearing Own Lat     Own Long    OwnMSL  Valid
----------------------  --- ----    ------------------  ----    -----   ----    ----    ----    -----   ------  ------  ------- ---------   ---------   ------  -----
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  9.1 0.24    0x59    22  18  0.872   1178.0  0.0 299.2   0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  9.9 0.13    0x59    22  18  0.869   1178.0  0.0 299.2   0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  9.9 0.13    0x59    22  1   0.157   1192.5  14.5    298.1   0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  7.8 0.24    0x65    26  5   0.645               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  7.8 0.24    0x93    39  8   0.610               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  7.8 0.24    0xac    50  0   0.558               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  7.8 0.24    0xac    50  1   0.524               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  7.8 0.24    0xf 0   22  0.479               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0x65    26  5   0.645               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0x93    39  8   0.610               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xac    50  0   0.558               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xac    50  1   0.524               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xf 0   22  0.479               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.7 0.24    0x65    26  5   0.645               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.7 0.24    0x93    39  8   0.610               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.7 0.24    0xac    50  0   0.558               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.7 0.24    0xac    50  1   0.524               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.7 0.24    0xf 0   22  0.479               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.7 0.28    0x59    22  18  0.894   1178.0  0.0 299.2   0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  9.8 0.14    0x59    22  18  0.874   1178.0  0.0 299.2   0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.0 0.24    0x65    26  5   0.645               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.0 0.24    0x93    39  8   0.610               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.0 0.24    0xac    50  0   0.558               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.0 0.24    0xac    50  1   0.524               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  8.0 0.24    0xf 0   22  0.479               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0x65    26  5   0.645               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0x93    39  8   0.610               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xac    50  0   0.558               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xac    50  1   0.524               0.000000    0.000000    -999    False
2023-02-28  13:02:00 Z  NN  C1B2    (Herts Beds Bucks)  4.9 0.24    0xf 0   22  0.479               0.000000    0.000000    -999    False
andimik commented 1 year ago

These are the transmitters, right?

Screenshot (390)

---------------------------
Hemel Hempstead/Pimlico
---------------------------
Id: 2800259

Latitude: 51.7285

Longitude: -.4261

Altitude: 137m

Name: Hemel Hempstead/Pimlico

DAB Block: 10D

EId: C1B2

Main, SubId: 22 18

Frequency: 215.072MHz

Ant. Height: 69m

Power: 4KW

Polarisation: v
andimik commented 1 year ago

@softsyst

Dear Clem, assuming (= I don't know the spec, but even my muxes here have 2 hex numbers) that the pattern always contains of 2 hex numbers, Qirx shows it as 0xf (one digit) as well.

Any idea?

srcejon commented 1 year ago

Yes, those are the transmitters. I updated the original comment to use the correct decimal number!

So actually, it looks like pattern is calculated properly as 0x59 / 89 / 0131

In table[]

0131,       // 0 1 0 1 1 0 0 1      22

However, is the bug here:

for (i = 0; i < 70; ++i) 
    invTable [table [i]] = i;
for (i = 71; i < 256; i ++)
   invTable [i] = -1;

The first for loop sets invTable[89] to 22, then the second for loop sets it back to -1.

srcejon commented 1 year ago

If I change the code to:

for (i = 0; i < 256; i ++)
   invTable [i] = -1;
for (i = 0; i < 70; ++i) 
    invTable [table [i]] = i;

The TII is now correctly decoded as 0x16 (22).

andimik commented 1 year ago

@srcejon

but do you mean 0xff or 0xf?

srcejon commented 1 year ago

I was always getting a main TII of 0xff (which makes sense, as that table entry is set to -1 in the code above)

I'm not quite sure what you are referring to with regards to Qirx and 0xf. I can't see that in the screenshot you posted.

andimik commented 1 year ago

The 8th line shows 0xf, not 0xff

srcejon commented 1 year ago

The 8th line shows 0xf, not 0xff

Ah, I see, in the text below the image. I presume that is because a pattern of 0xf maps to a M_Id of 0:

0017,       // 0 0 0 0 1 1 1 1      0

I was definitely getting a pattern of 0x59, which was mapping to a TII of 0xff.

I think it is a bug just by inspecting the code above. invTable is corrupted.

softsyst commented 1 year ago

@softsyst

Dear Clem, assuming (= I don't know the spec, but even my muxes here have 2 hex numbers) that the pattern always contains of 2 hex numbers, Qirx shows it as 0xf (one digit) as well.

Any idea?

Hello Andreas,

Please have a look at the DAB Standard ETSI EN 300 401 V2.1.1, p. 114, Table 26:

The column p is the TII Main Id, the pattern is the column ab(p). The correspondence is 1:1. The pattern is just a binary number, which is usually converted into hexadecimal. The pattern 00001111 (binary) is in hex 0x0f or simple f. 0x0f and f and 00001111 are all identical, decimal 15.

Thus, if the pattern f (=0x0f=00001111) is deocded, this indicates a Main Id of 0 (p value in that row of the table).

As you see by inspecting the table, there is no pattern 0xff, as the table ends at 0xf0 (binary 11110000). The rule guiding the pattern construction is that there MUST be exactly four binary 1's in the pattern, otherwise it is no longer unambiguous (I describe the drawback of this construction principle in detail in my "TII collision" tutorial).

The raw file which I also downloaded is near to unusable, as it is only a few seconds short. This is not enough (at least in qirx) to let all the TII carriers come to a stable state, to allow a correct decoding. As you see from the CIR spectrum, there are probably 5 transmitters contributing to the spectrum, as there are 5 clear peaks in the sepctrum. Due to the short time in my run only one is decoded, being 22/18. This is also the reason why most of the entries in the TII logging file presented above are crap, they indicate mostly noise.

As correctly mentioned by @srcejon (in his revised table), the ETSI table shows for the Main Id 22 the binary pattern 01011001, which is hex 0x59, or decimal 89.

If a somewhat longer raw file were possible, I am pretty confident that at least three transmitters - perhaps even five - could be decoded, as the carriers are rather clearly showing up in the TII spectrum.

Best, Clem

Here is the decoding with CIR and TII spectra. image

andimik commented 1 year ago

@softsyst

Many thanks, Clem for your very detailed answer! Appreciating that a lot!

softsyst commented 1 year ago

You're very welcome, as always!

JvanKatwijk commented 1 year ago

whow, that is good thinking. Of course I tried to optimize here, and again, it shows that "optimization kills"

Thanks for pointing this out. It also needs to be changed in Qt-DAB. I'll make the change in dab-cmdline as well

Op wo 1 mrt 2023 om 20:41 schreef srcejon @.***>:

Yes, those are the transmitters. Update the original comment to use the correct decimal number!

So actually, it looks like pattern is calculated properly as 0x59 / 89 / 0131

In table[]

0131, // 0 1 0 1 1 0 0 1 22

However, is the bug here:

for (i = 0; i < 70; ++i) invTable [table [i]] = i; for (i = 71; i < 256; i ++) invTable [i] = -1;

The first for loop sets invTable[89] to 22, then the second for loop sets it back to -1.

— Reply to this email directly, view it on GitHub https://github.com/JvanKatwijk/dab-cmdline/issues/91#issuecomment-1450748521, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCPHQDUFWAEKHDBRVJWVYDWZ6Q7VANCNFSM6AAAAAAVJJHS5M . You are receiving this because you commented.Message ID: @.***>

-- Jan van Katwijk

srcejon commented 1 year ago

Thanks for the fix!