search

AlbrechtL / welle.io

A DAB/DAB+ Software Defined Radio (SDR)
https://www.welle.io
Other
632 stars 118 forks source link

Support for DAB SPI #558

Closed nickpiggott closed 1 month ago

nickpiggott commented 4 years ago

Is there interest in supporting decoding of DAB SPI?

This would support:

There isn't an open source DAB EPG decoder, so this might need to be a "from scratch" activity.

AlbrechtL commented 4 years ago

Can you explain more in detail what a DAB SPI is?

nickpiggott commented 4 years ago

DAB SPI was previously known as DAB EPG. (TS 102 818)

SPI is (usually) transmitted as a packet data service, one per multiplex, and contains

DAB SPI is implemented in car receivers (BMW, Audi, etc.) and was previously integrated into receivers like the PURE Evoke-3, iRiver B20, Roberts MP-SOUND 41.

SPI can also be acquired over IP, using a RadioDNS Lookup. When acquired over IP, it usually contains:

The HRadio Project has implemented an SPI over IP package in Java, which they've used in their Android app, and it appears to have been integrated into DAB-Z too.

As welle.io implements DAB Slideshow (with an MOT Decoder), implementing DAB SPI would need a EPG Binary Decoder and an XML parser. Implementing SPI over IP needs an XML parser. The output XML files are identical between DAB and IP, so can be rendered using an single UI element.

AlbrechtL commented 4 years ago

Thanks for the explanation.

As far as I know there is no EPG Binary Decoder specification public available.

Do you know if DAB SPI is already in use?

andimik commented 4 years ago

Yes, there are lots of muxes all over Europe using SPI

The only software which can display it (but only the preview, no export of the thumbnails possible) is the Noxon Software.

Qt-DAB can export the content to xml files. But I could not find a suitable parser for them.

nickpiggott commented 4 years ago

Ben Poor (formerly from Global Radio) has a binary encoder which gives the mappings:

https://github.com/magicbadger/python-dabepg

The also has an "unmarshall" method that I've never tried:

def unmarshall(i): """Unmarshalls a PI or SI binary file to its respective :class:Epg or :class:ServiceInfo object
:param i: String or File object to read binary from :type i: str, file """

Might that help?

AlbrechtL commented 4 years ago

The best case would be to have a C/C++ decoder that we can just integrate.

But as a first step we should consider to create files that we can put into python-dabepg.

@nickpiggott Do you know which kind of input data python-dabepg needs?

nickpiggott commented 4 years ago

I'll talk with @magicbader to be certain, but I think it's the binary payload of an MOT Object received over DAB. So if you have an MOT decoder already that extracts the PNG/JPG images for Slideshow, you can use that

It needs to decode:

Nick

On Thu, 2 Jul 2020 at 09:46, Albrecht Lohofener notifications@github.com wrote:

The best case would be to have a C/C++ decoder that we can just integrate.

But as a first step we should consider to create files that we can put into python-dabepg https://github.com/magicbadger/python-dabepg.

@nickpiggott https://github.com/nickpiggott Do you know which kind of input data python-dabepg needs?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AlbrechtL/welle.io/issues/558#issuecomment-652875653, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYF3WUEHMOV7DDZ6XOKFEDRZRCNBANCNFSM4NGYUHFA .

-- Nick

andimik commented 4 years ago

Please correct me if I am wrong, but Qt-DAB also can handle it.

The only problem there is that I couldn't find a program which displays the xml files in a comfortable format. Of course a text editor shows it.

But Qt-DAB can extract the slides (not 100% accurate, does not work on all muxes) and the content.

nickpiggott commented 4 years ago

I think Qt-DAB may only support Slideshow. SPI is a different application, but both of them use MOT as a transport protocol.

SPI requires MOT Directory to be supported (which Slideshow does not), and SPI consists of binary-encoded XMI files as well as image files of differing resolutions (usually representing station logos).

So it's important not to confuse Slideshow with SPI, even though they both use MOT as transport.

On Fri, 3 Jul 2020 at 14:56, andimik notifications@github.com wrote:

Please correct me if I am wrong, but Qt-DAB also can handle it.

The only problem there is that I couldn't find a program which display the xml files.

But Qt-DAB can extract the slides (not 100% accurate, does not work on all muxes) and the content.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AlbrechtL/welle.io/issues/558#issuecomment-653560620, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYF3WQBFXQU4O7MJN75CH3RZXPRJANCNFSM4NGYUHFA .

-- Nick

andimik commented 4 years ago

This is what you get from Qt-DAB when you for instance use SWR Mux (feed 7 deg East)

handle_MOT: type 700, name ee010ea.EIB dir = 1
handle_MOT: type 203, name radio_swr-dasding_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-dasding_128x128v.png
handle_MOT: type 203, name radio_swr-swr-aktuell_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-swr-aktuell_128x128v.png
handle_MOT: type 203, name radio_swr-swr1_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-swr1_128x128v.png
handle_MOT: type 203, name radio_swr-swr2_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-swr2_128x128v.png
handle_MOT: type 203, name radio_swr-swr3_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-swr3_128x128v.png
handle_MOT: type 203, name radio_swr-swr4_128x128v.png dir = 1
going to write file /tmp/qt-pictures/radio_swr-swr4_128x128v.png
handle_MOT: type 701, name w20200325dd301c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd3a2c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd3a3c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd3a5c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd3a6c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd804c0.EHB dir = 1
handle_MOT: type 701, name w20200325dd904c0.EHB dir = 1
handle_MOT: type 701, name w20200325ddc04c0.EHB dir = 1
handle_MOT: type 701, name w20200325ddd04c0.EHB dir = 1

radio_swr-swr4_128x128v radio_swr-swr3_128x128v radio_swr-swr2_128x128v radio_swr-swr1_128x128v radio_swr-swr-aktuell_128x128v radio_swr-dasding_128x128v

The newer versions don't write the EHB files any more out of the box.

You have to change the path in radio.cpp to

    epgPath     = dabSettings -> value ("epgPath", "/tmp/"). toString ();
    filePath    = dabSettings -> value ("filePath", "/tmp/"). toString ();

epg.zip

<epg system="DAB">
 <schedule version="1">
  <scope stopTime="2020-03-29T00:00:00+01:00" startTime="2020-03-28T00:00:00+01:00">
   <serviceScope id="40.e0.10"/>
  </scope>
  <programme recommendation="no" shortId="2269703" broadcast="on-air">
   <mediumName>Die Nacht</mediumName>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.5"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <location>
    <time duration="PT5H" time="2020-03-28T00:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269721" broadcast="on-air">
   <mediumName>Guten Morgen Bad</mediumName>
   <longName>Guten Morgen Baden-Württemberg </longName>
   <mediaDescription>
    <shortDescription>5:57 - 6:00 Anstöße
6:57 - 7:00 Anstöße</shortDescription>
   </mediaDescription>
   <genre type="main" href="urn:tva:metadata:cs:FormatCS:2005:2.1.2"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.14"/>
   <location>
    <time duration="PT3H" time="2020-03-28T05:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269738" broadcast="on-air">
   <mediumName>Schmidts Samstag</mediumName>
   <genre type="main" href="urn:tva:metadata:cs:FormatCS:2005:2.1.2"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.14"/>
   <location>
    <time duration="PT4H" time="2020-03-28T08:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269751" broadcast="on-air">
   <mediumName>Aktuell um zwöl</mediumName>
   <longName>Aktuell um zwölf</longName>
   <genre type="main" href="urn:tva:metadata:cs:FormatCS:2005:2.1.2"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.14"/>
   <location>
    <time duration="PT1H" time="2020-03-28T12:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269776" broadcast="on-air">
   <mediumName>Arbeitsplatz</mediumName>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.1.3.3"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <location>
    <time duration="PT2H" time="2020-03-28T13:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269796" broadcast="on-air">
   <mediumName>Stadion</mediumName>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.2"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.14"/>
   <location>
    <time duration="PT3H" time="2020-03-28T15:00:00+01:00"/>
   </location>
  </programme>
  <programme recommendation="no" shortId="2269839" broadcast="on-air">
   <mediumName>Der Abend</mediumName>
   <mediaDescription>
    <shortDescription>18:40 - 18:42 Sport kompakt</shortDescription>
   </mediaDescription>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.5"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.6.4.1"/>
   <genre type="main" href="urn:tva:metadata:cs:ContentCS:2005:3.1.1.16"/>
   <location>
    <time duration="PT6H" time="2020-03-28T18:00:00+01:00"/>
   </location>
  </programme>
 </schedule>
</epg>
andimik commented 4 years ago

Can we reuse these files?

https://github.com/MichalLeonBorsuk/drm-code/tree/622cc259e15f25ec5432ec3aa8176147c0a7f94d/branches/dream-980/src/datadecoding

AlbrechtL commented 4 years ago

@nickpiggott Thanks for your explanation.

I checked an exemplary audio service. There the slideshow is coming via X-PAD and the MOT protocol. But there is no MOT dictionary included. Thus I assume that the MOT dictionary is transmitted inside a data service. But how do I know which is the corresponding data service to an audio service? Inside the my ensemble there are multiple data services.

nickpiggott commented 4 years ago

@AlbrechtL

I will refer to EPG as SPI, as SPI is the new name for the application.

The SPI service will always use MOT, and it may be transmitted:

X-PAD - with Slideshow and SPI

There will be ONE FIG0/13 entry referencing the current audio service (SId / SCId). It will have "Number of User Applications = 2"

(* this value is created by the encoder - typically 12 for Slideshow, 16 for EPG)

When decoding X-PAD (EN 300 401 s 7.4.2.0 and EN 301 234 s 5.2.2) the Content Indicator flag will be set (CI Flag = 0b1), and each X-PAD field will start with at least TWO 1 byte Contents Indicators (s 7.4.4):

(* this value as advertised in FIG0/13 above)

Use the same process to rebuild Data Groups from the X-PAD fields as you do for Slideshow, but keep the two applications (X and Y) separate. Each complete Data Group creates an MOT Segment, and a complete MOT Object is rebuilt from one or more MOT Segments. (EN 301 234 s5.1).

MSC Packet Data Channel with SPI

You need to use FOUR FIGs to locate an SPI service:

So locate the SubChID that has TMId= 11 (Packet Data), DSCTy = 60 (MOT), User Application Type 0x07 (SPI). Extract the packets from that SubChID that match Packet Address.

Concatenate those packets to make MSC Data Groups -> MOT Segments -> MOT Objects

Currently you can't tell which services are included in an SPI service without decoding it. (There is a debate about how to signal this, such as making the SPI MSC Packet Data service a component of the audio service, but this isn't established). So it's usual to just decode the SPI service(s) and cache them on the receiver.

Common Process

When decoding SPI, you will receive MOT Bodies and ONE MOT Directory. The MOT Directory contains the headers for all the MOT Bodies (referenced by their Transport ID). Therefore you can't decode any of the MOT Objects until you have the MOT Directory.

Once you've decoded the MOT Objects, you will have:

Final step - you need to binary decode the SI and PI files, which you can push through a binary decoder, that will create SI.xml and PI.xml files. The SI has the service information, and the links to logos will refer to ContentName values from the MOT Objects. The PI files have programme schedule information, usually one file per service per calendar day.

EXAMPLE ETI FILE This ETI file contains a snapshot of the Small Scale DAB Trial in Bristol UK (taken 2020-07-12 at 09:50 UTC - Updated to replace previously corrupted file).

There is a MSC Packet Data service with SId = e1c000d4. It's an 8kbit/s channel, so there is 1 x 24 byte packet per Common Interleaved Frame. The Packet Address is 0x01.

The SPI service contains:

The total SPI service spans 20,216 packets, so it takes just over 8 minutes to decode normally.

(Not all services in the ensemble are represented in the SI file).

andimik commented 4 years ago

Thanks for the file, just fyi: Qt-DAB does not find anything in the SPI service (not even a png file), so you cannot rely on Qt-DAB's code here.

nickpiggott commented 4 years ago

My apologies @andimik - that ETI file was corrupted. I've uploaded a new version here which is only 176MB long, and I've tested with DAB-Scout 3 to be working.

The MOT Directory appears close to the end of the file, so it had to loop in DAB scout to be able to decode everything.

Could you try again with Qt-DAB please?

andimik commented 4 years ago

Are you sure this file is NOT corrupt? I even get no audio in the today's file whereas yesterday it worked.

grafik grafik

Have checked it with dablin_gtk.

BTW: Please zip the eti before uploading, it will save a lot.

nickpiggott commented 4 years ago

I recorded that file from the test multiplexer and didn't configure any audio inputs. It did decode correctly in DAB-Scout 3.

You should see a MSC Packet Data channel with SID e1c000d4, which has the SPI application in it (packet address 1). If you're looking at FIGs, 0/2, 0/3, 0/8 and 0/13 should all reference to it.

Nick

On Sun, 12 Jul 2020, 12:01 andimik, notifications@github.com wrote:

Are you sure this file is NOT corrupt? I even get no audio in the today's file whereas yesterday it worked.

Have checked it with dablin_gtk.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AlbrechtL/welle.io/issues/558#issuecomment-657206468, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYF3WRM7YOODEW6E44AK6TR3GJYBANCNFSM4NGYUHFA .

andimik commented 4 years ago

Oh I see, I thought it was a live capture ...

I get this for example

`FIG: 0/3 Length: 6 OE: 0 C/N: 0 P/D: 0 Decoding:

   SCId: 0x0
   CAOrg flag: 0 CAOrg field absent
   DG flag: 0
   data groups are used to transport the service component
   DSCTy: 60 Multimedia Object Transfer (MOT)
   SubChId: 0xE
   Packet address: 0x1

Stream Number: value: 13 SCID: value: 14 desc: Sub-channel Identifier SAD: value: 828 desc: Sub-channel Start Address TPL: desc: Sub-channel Type and Protection Level EEP: value: 34 desc: Equal Error Protection Level: 3-A Rate: 1/2 CUs: 6 STL: value: 3 desc: Sub-channel Stream Length bitrate: value: 8 desc: kbit/s `

I can see the service "EPG" in all my tools, but the Fraunhofer Software is running for lots of minutes without showing the SPI content.

bristol_spi_fraunhofer

Qt-DAB says

Data Services
program name;;serviceId;subchannelId;start address;length (CU); bit rate; FEC; prot level; appType ; subService ; 

EPG             ;United Kingdom;E1C000D4;14;828;6;8;0;EEP 3-A;7;no;;

That's why I even fear this new ETI is corrupt. Because I don't get the CU overview in Andy's DAB Player.

These are the screenshots of yesterday's file. bristol_andi_player_fic bristol_andi_player

FIC analysis on https://www.telder.com/fic/upload_new_fic.php says (filtered to E1C000D4)

2020-07-11 18:23:07 000058 FIG 0/8 SId=E1C000D4 SCIdS=0 SCId=0 2020-07-11 18:23:07 000023 FIG 1/5 SId=E1C000D4 Data service label EPG 2020-07-11 18:23:07 000009 FIG 0/2 SId=E1C000D4 [ MSC Packet data Primary SCId=0 ] 2020-07-11 18:23:07 000002 FIG 0/13 SId=E1C000D4 SCIdS=0 UAtype=0x07=[ EPG ] [ 01 00 ]

nickpiggott commented 4 years ago

Hello

That signalling looks correct. I am able to decode the file using DAB Scout.

The SPI uses about 22,000 frames to receive, so probably about 10-12 minutes. DAB Scout is looping the file to get all the objects because the MOT Directory comes last. Does the Fraunhofer decode indicate that it's found any MOT objects? An MOT Directory?

I'll make a file twice as long, so the SPI loops twice and see if that works.

Nick

On Sun, 12 Jul 2020, 13:05 andimik, notifications@github.com wrote:

Oh I see, I thought it was a live capture ...

I get this for example

FIG: 0/3 Length: 6 OE: 0 C/N: 0 P/D: 0 Decoding: - SCId: 0x0 CAOrg flag: 0 CAOrg field absent DG flag: 0 data groups are used to transport the service component DSCTy: 60 Multimedia Object Transfer (MOT) SubChId: 0xE Packet address: 0x1 Stream Number: value: 13 SCID: value: 14 desc: Sub-channel Identifier SAD: value: 828 desc: Sub-channel Start Address TPL: desc: Sub-channel Type and Protection Level EEP: value: 34 desc: Equal Error Protection Level: 3-A Rate: 1/2 CUs: 6 STL: value: 3 desc: Sub-channel Stream Length bitrate: value: 8 desc: kbit/s

I can see the service "EPG" in all my tools, but the Fraunhofer Software is running for lots of minutes without showing the SPI content.

[image: bristol_spi_fraunhofer] https://user-images.githubusercontent.com/24510556/87245743-3428cd00-c448-11ea-92ad-a65304f4f08e.png

Qt-DAB says

Data Services program name;;serviceId;subchannelId;start address;length (CU); bit rate; FEC; prot level; appType ; subService ;

EPG ;United Kingdom;E1C000D4;14;828;6;8;0;EEP 3-A;7;no;;

That's why I even fear this new ETI is corrupt.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/AlbrechtL/welle.io/issues/558#issuecomment-657212853, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACYF3WS7FT427VLACVNUFYDR3GRHFANCNFSM4NGYUHFA .

andimik commented 4 years ago

Yes, please make it much longer (and zip it ...)

Does the Fraunhofer decode indicate that it's found any MOT objects? An MOT Directory?

It will do after the progress bar is 100% and all frames have been successfully received.

andimik commented 4 years ago

BTW: Don't use your mail tool, please log on to github as you missed my updated comments.

andimik commented 4 years ago

Does your license allow you to increase the bitrate? There are 30 CUs free.

nickpiggott commented 4 years ago

I'm trying not to change too many variables, and I'm not going to change the configuration of the on-air service.

In this recording I've halved the payload size by omitting the 320x240px logos - it should not be received in just over 8,000 frames (3.5 minutes), and there are at least three full rotations of the carousel in the file.

There is audio on the service Ujima Radio.

andimik commented 4 years ago

Now it loads it completely, but there is no preview in the next week available. Should I find events in this SPI?

no_epg_bristol

At least I find these pointers in the Fraunhofer Software (hidden folder .mmp\cache\spi in user directory)

$ cat /media/andreas/DOS/db.mmp 
MMP-STORAGE�_
�����������������   ��_
������
      ����
          Sam FMSam FM BristolSam FM Bristol����"Sam_FM_112_32.pngimage/png Sam_FM_32_32.pngimage/png$Sam_FM_128_128.pngimage/png��ɇ_
������
      ��ɇ
         BStoke�Bradley Stoke��������"BStoke_112_32.pngimage/png BStoke_32_32.pngimage/png$BStoke_128_128.pngimage/png����_
������
      ���BCfBCfm��������BCfm_112_32.pngimage/pngBCfm_32_32.pngimage/png BCfm_128_128.pngimage/png����_
������
      ����
NOODSNOODS Radio�������� NOODS_112_32.pngimage/pngNOODS_32_32.pngimage/png"NOODS_128_128.pngimage/png����_
������
      ����BFBS GrkBFBS Gurkha��������&BFBS_Grk_112_32.pngimage/png$BFBS_Grk_32_32.pngimage/png(BFBS_Grk_128_128.pngimage/png����_
������
      ����HUBHub Radio��������HUB_112_32.pngimage/png�HUB_32_32.pngimage/pngHUB_128_128.pngimage/png��û_
������
      ��û
         BreezeBreeze Bristol$The Breeze Bristol����"Breeze_112_32.pngimage/png Breeze_32_32.pngimage/png$Breeze_128_128.pngimage/png����_
������
      ����
          SomerVSomer Valley��������"SomerV_112_32.pngimage/png SomerV_32_32.pngimage/png$SomerV_128_128.pngimage/png����_
������
      ����
Ujima
Ujima�������� Ujima_112_32.pngimage/pngUjima_32_32.pngimage/png"Ujima_128_128.pngimage/png��

but I could not see any png so far.

You wrote

1 x SI file (20200711_e1.c1c8.SI.xml) 6 x PI files (3 days of Programme Information for each of services c3bb and cf92)

Strange, cannot see that.

nickpiggott commented 4 years ago

That db.mmp looks like the contents of the SPI SI.xml file that has been decoded - you can see the references to the station names and logo files and so on.

There should be programme schedules for THREE days (12th, 13th, 14th July 2020) for:

There should be logos at 32 x 32, 112 x 32 and 128 x 128 pixels for

All those are decoding correctly in DAB Scout - I can see programme schedules, and I can see the logos correctly decoded in the cache file. (DAB Scout 3 doesn't show the logos in the UI).

Is there any insight into what the Fraunhofer analyser doesn't like? It can clearly decode the MOT Objects as it has decoded the MOT Directory at the very least.

nickpiggott commented 4 years ago

From DAB Scout 3

image

and the data directory

image

and I've attached the decoding of the MOT Directory which DAB Scout reformats to XML

_motDirectory.xml.txt

AlbrechtL commented 4 years ago

I will refer to EPG as SPI, as SPI is the new name for the application.

The SPI service will always use MOT, and it may be transmitted:

  • In the X-PAD of the radio service it relates to (Not very common) AND/OR
  • As an MSC Packet Data service carrying information on one or more radio services (Commonly implemented)

Thanks for the very detailed description! To me it looks like that this is very interesting to implement.

andimik commented 4 years ago

Dear Albrecht and Nick,

Jan has implemented a simple SPI decoder (still in progress!!!) and can show the SPI in a separate window.

nickpiggott commented 4 years ago

That's great news! Is there a build ready to test with, and I can check it out with some UK / Bristol services I know are broadcasting SPI?

andimik commented 4 years ago

I forgot to mention that it's inside Qt-DAB

First of all add these lines to ~/.qt-dab.ini

epgPath=/tmp/qt-dab/epg/
filePath=/tmp/qt-dab/mot/

Any other path is fine, too.

Then you have to select the SPI service and wait till the EHB files have been written (check the console for messages)

For example 

starting a backend for EPG              (E0D010C9)
handle_MOT: type 701 (7), name w20200411dd804c0.EHB dir = 1
epg
currentSID = D804
Version 1
epg file 
handle_MOT: type 701 (7), name w20200411dd904c0.EHB dir = 1
epg
currentSID = D904
Version 1
epg file

Then you select one service (which has an SPI preview) and click on detail which will open a window.

There you should see a new button "timeTable".

Then another window with the SPI preview appears (only time, no date information).

This also works when you play recordings, so it's not like Fraunhofer software which only accepts SPI in the present or future.

It looks like this (some bugs have been fixed in the meantime)

https://www.rundfunkforum.de/download/file.php?id=5419&mode=view

AlbrechtL commented 3 years ago

Cool! Thanks for the hint. Since welle.io bases on Jan codes the integration of Jans SPI decoder shouldn't difficult.

andimik commented 2 years ago

Hint: cQIRX (Qirx Console) can write all slides and has an XML Parser for the EIB File.

https://qirx.softsyst.com/LinuxCmdLineInstall

But it's not Open Source.

The Linux version allows to select even data services which are not detected as SPI (Like in Slovenia, where the EPG/SPI Service is called Spored).

The Windows Version works with most of the available muxes. This is the only software (besides of Fraunhofer) that uses 32x32 thumbnails for the audio services.

nickpiggott commented 2 years ago

Thanks for that suggestion @andimik - I really find it useful to see what's being broadcast on SPI.

Why is the Slovenian service not being detected as SPI? If it's being signalled as SPI (DAB EPG) application, it shouldn't matter what the service label is?

andimik commented 2 years ago

Without having read the standard, Jan and Clem told me they have to identify SPI by their service label names.

Is there really a flag which could help identifying SPI?

nickpiggott commented 2 years ago

I did a long walk-through of how you "unthread" the configuration of data services from their various FIGs.

In summary, it's identified from the User Application Type transmitted in FIG0/13, which should have a value 0x07 for SPI. If it's not being transmitted with that, it's not compliant with the specification, and its unlikely any compliant receiver will identify and decode it.

(There were, back in the year 200x, some receivers that speculatively decoded packet data services with MOT Directory, to see if they contained files ending EIB. But that was never part of the standard, and is not at all reliable. The current standard, for example, does not require objects to be named ending EIB - they're identified by their MOT Headers).

AlbrechtL commented 2 years ago

Stupid question: Has Jan with Qt-DAB some progress in SPI decoding?

andimik commented 2 years ago

No, although there are lots of progresses regarding other features, the SPI is still in same experimental stage...

It even does not show the date, just the time.

AlbrechtL commented 2 years ago

Does anymore knows how Andreas Gsinn DAB Player is getting the SPI? Does he has its own implementation?

andimik commented 2 years ago

On this German page he writes, this is in preparation, but the same text is online for a very long time now.

There is no support for SPI in DAB Player.

https://www.ukwtv.de/cms/downloads-aside/281-dab-player-von-andreas-gsinn.html

AlbrechtL commented 2 years ago

Oh, you are right. I thought he implemented it but I mixed up something, obviously.

AlbrechtL commented 2 years ago

@AlbrechtL

I will refer to EPG as SPI, as SPI is the new name for the application.

The SPI service will always use MOT, and it may be transmitted:

* In the X-PAD of the radio service it relates to _(Not very common)_
  AND/OR

* As an MSC Packet Data service carrying information on one or more radio services _(Commonly implemented)_

X-PAD - with Slideshow and SPI

There will be ONE FIG0/13 entry referencing the current audio service (SId / SCId). It will have "Number of User Applications = 2"

* User Application 1 = 0x02 (Slideshow), X-PAD AppTy = X* (MOT), DSCTy = 60 (MOT)

* User Application 2 = 0x07 (EPG), X-PAD AppTy = Y* (MOT), DSCTy = 60 (MOT)

(* this value is created by the encoder - typically 12 for Slideshow, 16 for EPG)

When decoding X-PAD (EN 300 401 s 7.4.2.0 and EN 301 234 s 5.2.2) the Content Indicator flag will be set (CI Flag = 0b1), and each X-PAD field will start with at least TWO 1 byte Contents Indicators (s 7.4.4):

* X-PAD AppTy = 1 (MOT Data Group Length Indicator)

* X-PAD AppTy = X or X+1 (Slideshow MOT Data Group Start / Continuation)
  OR

* X-PAD AppTy = 1 (MOT Data Group Length Indicator)

* X-PAD AppTy = Y or Y+1 (EPG MOT Data Group Start / Continuation)

(* this value as advertised in FIG0/13 above)

Use the same process to rebuild Data Groups from the X-PAD fields as you do for Slideshow, but keep the two applications (X and Y) separate. Each complete Data Group creates an MOT Segment, and a complete MOT Object is rebuilt from one or more MOT Segments. (EN 301 234 s5.1).

MSC Packet Data Channel with SPI

You need to use FOUR FIGs to locate an SPI service:

* FIG0/2 with P/D = 1, TMId = 0b11 (MSC Packet Data Channel). Provides:  32bit SId and a 12bit SCId.

* FIG0/3 with P/D =1 , SCId (matching FIG0/2), DSCTy = 60 (MOT). Provides: SubChId, Packet Address

* FIG0/8 with P/D = 1, 32bit SId, 12 bit SCId (which match FIG0/2). Provides: 4 bit SCIdS value.

* FIG0/13 with P/D =1, 32bit SId (matching FIG0/2), and SCIds (matching FIG0/8). Provides: User Application Type

So locate the SubChID that has TMId= 11 (Packet Data), DSCTy = 60 (MOT), User Application Type 0x07 (SPI). Extract the packets from that SubChID that match Packet Address.

Concatenate those packets to make MSC Data Groups -> MOT Segments -> MOT Objects

Currently you can't tell which services are included in an SPI service without decoding it. (There is a debate about how to signal this, such as making the SPI MSC Packet Data service a component of the audio service, but this isn't established). So it's usual to just decode the SPI service(s) and cache them on the receiver.

Common Process

When decoding SPI, you will receive MOT Bodies and ONE MOT Directory. The MOT Directory contains the headers for all the MOT Bodies (referenced by their Transport ID). Therefore you can't decode any of the MOT Objects until you have the MOT Directory.

* MOT Directory is transported in MSC Data Group type 6

* Compressed MOT Directory is transported in MSC Data Group type 7 - you have to pass the payload through GZIP to expand it.

* MOT Bodies are transported in MSC Data Group type 4.

Once you've decoded the MOT Objects, you will have:

* ONE SI file - Content Type = 7 and Content SubType = 0

* ZERO or more PI files - ContentType = 7 and Content SubType =1

* ZERO or more other files (representing the logos usually) - with content types per EN 301 234 / TS 101 757 Table 17.

Final step - you need to binary decode the SI and PI files, which you can push through a binary decoder, that will create SI.xml and PI.xml files. The SI has the service information, and the links to logos will refer to ContentName values from the MOT Objects. The PI files have programme schedule information, usually one file per service per calendar day.

EXAMPLE ETI FILE This ETI file contains a snapshot of the Small Scale DAB Trial in Bristol UK (taken 2020-07-12 at 09:50 UTC - Updated to replace previously corrupted file).

There is a MSC Packet Data service with SId = e1c000d4. It's an 8kbit/s channel, so there is 1 x 24 byte packet per Common Interleaved Frame. The Packet Address is 0x01.

The SPI service contains:

* 1 x SI file (20200711_e1.c1c8.SI.xml)

* 6 x PI files (3 days of Programme Information for each of services c3bb and cf92)

* 36 x PNG files (4 logos for each of 9 services)

The total SPI service spans 20,216 packets, so it takes just over 8 minutes to decode normally.

(Not all services in the ensemble are represented in the SI file).

@nickpiggott Today I started in taking a look into it. So my first question is: How do I know which services is carrying the SPI data? In my case I have 5 data services. I took a look into the DAB standards but I didn't find a corresponding data service announcement or signal.

2022-03-14T21:18:56.426 Debug: RadioController: Found data service  E0D110BC "EPG Deutschland "
2022-03-14T21:18:56.471 Debug: RadioController: Found data service  E0D010BC "TPEG            "
2022-03-14T21:18:56.506 Debug: RadioController: Found data service  E0D310BC "TPEG_MM         "
2022-03-14T21:18:56.554 Debug: RadioController: Found data service  E0D210BC "PPP-RTK-AdV     "
2022-03-14T21:18:56.787 Debug: RadioController: Found data service  E0D00250 "DRadio Daten    "
nickpiggott commented 2 years ago

Hello,

Let me give you an updated ETI file. This has 13 audio services in it; the SPI service has data on 12 services - 58 objects (plus the MOT Directory). There's no audio on the audio channels (to keep the ZIP file very small).

Adapting my earlier comment specifically for this example file (and using the output of etisnoop)

So it's complicated to discover, but you start with the

  1. FIG0/2 which says you have an MSC Packet Data channel, giving you SId = 0xE1C000D4and SCId = 0x00.
  2. FIG0/3 which says you have Service Component with SCId = 0x00, giving you DSCTy = 60 (MOT), SubChannel = 0xE and Packet Address = 0x1
  3. FIG0/8 which says you have SId = 0xE1C000D4and SCId = 0x00, giving you SCIds = 0x0
  4. FIG0/13 which says you have SId = 0xE1C000D4 and SCIds = 0x00, giving you User Application = 7 (EPG) and User Application Data = 0x01 (Extended Profile)

From that Subchannel 0xE (14) you should get a series of 24 byte packets (8kbit/s packet channel).

These python-dabmsc and python-dabmot might help with that.

The decoded MOT Objects will be a mix of

The last two files have to be decoded from a compacted binary format - you can either decode directly to memory objects or you can decode to XML and then parse XML. There's a "unmarshall" component in the SPI Python Libraries (https://github.com/Opendigitalradio/python-hybridspi/blob/master/src/spi/binary/__init__.py - line 1242 onwards), which will decode the binary in to a Python object (of type ProgrammeSchedule or ServiceInfo).

Does that help move things on a few steps?

AlbrechtL commented 2 years ago

Thanks. This makes it a bit more clear what is actual to do.

Now, I realized that I may didn't say clearly what my current issue is.

I have a number of audio services which may not contain SPI information.

2022-03-15T19:48:59.238 Debug: RadioController: Found service  D210 "Dlf             "
2022-03-15T19:48:59.638 Debug: RadioController: Found service  1A64 "ERF Plus        "
2022-03-15T19:48:59.921 Debug: RadioController: Found service  D220 "Dlf Kultur      "
2022-03-15T19:49:00.004 Debug: RadioController: Found service  D240 "DRadio DokDeb   "
2022-03-15T19:49:00.157 Debug: RadioController: Found service  D75B "KLASSIK RADIO   "
2022-03-15T19:49:00.207 Debug: RadioController: Found service  15DC "sunshine live   "
2022-03-15T19:49:00.238 Debug: RadioController: Found service  D01C "radio horeb     "
2022-03-15T19:49:00.288 Debug: RadioController: Found service  100D "Schwarzwaldradio"
2022-03-15T19:49:00.321 Debug: RadioController: Found service  1A45 "ENERGY          "
2022-03-15T19:49:00.358 Debug: RadioController: Found service  10C3 "SCHLAGERPARADIES"
2022-03-15T19:49:00.404 Debug: RadioController: Found service  17FA "Absolut relax   "
2022-03-15T19:49:00.438 Debug: RadioController: Found service  15DD "RADIO BOB!      "
2022-03-15T19:49:00.487 Debug: RadioController: Found service  D230 "Dlf Nova        "

I expected to have a link from audio service to a SPI contained data service. Just guessing from the name D210 "Dlf "can be related with E0D00250 "DRadio Daten ". But where is the link?

Data services:


2022-03-14T21:18:56.426 Debug: RadioController: Found data service  E0D110BC "EPG Deutschland "
2022-03-14T21:18:56.471 Debug: RadioController: Found data service  E0D010BC "TPEG            "
2022-03-14T21:18:56.506 Debug: RadioController: Found data service  E0D310BC "TPEG_MM         "
2022-03-14T21:18:56.554 Debug: RadioController: Found data service  E0D210BC "PPP-RTK-AdV     "
2022-03-14T21:18:56.787 Debug: RadioController: Found data service  E0D00250 "DRadio Daten    "

Or is the case in just looking to every data service for SPI?

In my first step I wanted to get station images because we already have a MOT decoder.

nickpiggott commented 2 years ago

You have to decode the SPI service to find the ServiceInformation file.

Within that Service Information file is the information for each service in that SPI - from that you can link back to the station.

There is no signalling (officially) in the DAB FIC that links an Audio service to the Packet Data service carrying it's SPI information.

(off-topic - there was a LONG discussion about making the SPI service a secondary component of the audio channel to which it related, but this was never formalised).

AlbrechtL commented 1 month ago

I'm closing this issue because the DAB data services are not in the focus right now.

If somebody provides a PR, that would be fantastic! To have station logs would be really nice.