oe-alliance / AutoBouquetsMaker

Automatically build and update bouquets from the DVB stream.
GNU General Public License v3.0
23 stars 59 forks source link

Sky Deutschland #283

Open yannydodo12 opened 8 months ago

yannydodo12 commented 8 months ago

Hi can the sat_192_sky_deutschland.xml be updated now that we can use the normal bouquets. I did try update it myself lol but doesn't seem to work right..

<sections>
        <section number="100">Entertainment</section>
        <section number="200">Sport</section>
        <section number="300">Film</section>
        <section number="500">Kids</section>
        <section number="600">Musik</section>
        <section number="700">News & International</section>
        <section number="800">Regional</section>
        <section number="900">Marktplatz</section>
        <section number="1000">LCN nicht bekannt</section>
    </sections>

I think this is the new line up. https://www.sky.de/hilfe/s/article/senderliste-sky-satellitenreceiver Thanks

AbuBaniaz commented 8 months ago

Before we update the sections, the channel order needs to be correct.

Here is a log: 192.168.1.179-20231103-183646.log

Example file:

EXAMPLE_hd_sat_192_sky_deutschland_CustomLCN.zip

yannydodo12 commented 8 months ago

Yeah that doesn't seem right to me. And seems to be missing sky showcase and probably some others. I've ended up using e-channelizer to make the bouquet lol..

forgot above lol it is there... i thought it would be something like this. I really don't understand it correctly lol userbouquet.abm.sat_192_sky_deutschland.main.zip

AbuBaniaz commented 8 months ago

We use a custom lcn file to place the channel in order. If @huevos @miberian can't find a way to get the channels in order, we will have to edit the lcn file.

sky de channel list.txt

Huevos commented 8 months ago

I have looked for this a long time. Can't find LCN on any of the transponders. So how are the boxes tuning? Do these Sky DE boxes even put the channels in an order?

yannydodo12 commented 8 months ago

I have looked for this a long time. Can't find LCN on any of the transponders. So how are the boxes tuning? Do these Sky DE boxes even put the channels in an order?

Looking at this video it does seem so https://www.youtube.com/watch?v=0mjYQd8BCbg

Huevos commented 8 months ago

Yes, but to correctly order the channels we need to find where that LCN data is stored on the transport stream (if at all). On all the transport streams I have seen that carry Sky DE channels that datais either not available or hidden in some non conventional place.

We really need an expert in SI tables to look at this, e.g. @LraiZer

yannydodo12 commented 8 months ago

Yes, but to correctly order the channels we need to find where that LCN data is stored on the transport stream (if at all). On all the transport streams I have seen that carry Sky DE channels that datais either not available or hidden in some non conventional place.

We really need an expert in SI tables to look at this, e.g. @LraiZer

Not sure it's worth the hassle mate . Not sure if anything of value is available here. All above my pay grade 😂 https://www.satellite-calculations.com/DVB/19east_all.php??

Huevos commented 8 months ago

No LCN listed there, and it must come from the transport stream, so unless someone knows where it is hidden, it is unlikely it will ever be fixed.

LraiZer commented 8 months ago

OpenTV LCN is normally hidden HERE I don't know of anyone who has found any LCN for DE on the TS yet, i've looked in the past and not found any. No harm in having another look though, as its been a while since i last looked.

Huevos commented 8 months ago

In the past I spent a lot of time looking. Maybe it is only list on one transponder. If someone has one of these boxes maybe the home transponder is listed in the setup, like on Sky UK boxes.

AbuBaniaz commented 8 months ago

It is the first time I have seen a list from their website, I was always redirected to the UK site. Some of the channels on that list do not show up in the debug log. Probably because they are by another provider.

Is there an easy way to see the log with lines starting on each service name instead of having multiple services on one line?

yannydodo12 commented 8 months ago

It is the first time I have seen a list from their website, I was always redirected to the UK site. Some of the channels on that list do not show up in the debug log. Probably because they are by another provider.

Is there an easy way to see the log with lines starting on each service name instead of having multiple services on one line?

On here it shows the lcn, maybe we can make an custom lcn with the details here . https://www.lyngsat.com/packages/Sky-Deutschland.html

AbuBaniaz commented 8 months ago

ABM uses data on the satellite, not websites.

AbuBaniaz commented 8 months ago

Someone has confirmed that the channel list on link in first comment is correct.

A raw stream recording of the frequency ABM uses for Sky DE is uploaded on link below. Obtained using a Zgemma H9S SE. Not checked as I don't have Transedit anymore.

https://www.mediafire.com/file/wmn36cooqz5one5/19.2e_12031500.ts/file

LraiZer commented 8 months ago

So maybe test this, and ask dad for the lcn.. ;)

Follow the paper trail, to an app-level configuration file on the stream containing the dvb:URL

12031 H 27500 9/10 QPSK 19.2e

GFD Service (0xdad)

dvbsnoop -n 10 0x08ae >/tmp/bingo.txt

stream targetUri:

100=142=dvb://85.B.8E;1;1 101=147=dvb://85.2.93;1;1 103=23=dvb://85.4.17;1;1 104=110=dvb://85.D.6E;1;1 105=14=dvb://85.2.E;1;1

STB lamedb5: s:008e:00c00000:000b:0085:25:0:0,"Sky Showcase HD",p:SKY,C:0 s:0093:00c00000:0002:0085:25:0:0,"Sky One HD",p:SKY,C:0

AbuBaniaz commented 8 months ago

Thanks. How did you export the data in this format please?

100=142=dvb://85.B.8E;1;1
101=147=dvb://85.2.93;1;1
103=23=dvb://85.4.17;1;1
104=110=dvb://85.D.6E;1;1
105=14=dvb://85.2.E;1;1
Huevos commented 8 months ago

dvbsnoop -n 10 0x08ae >/tmp/bingo.txt We need to capture this with the dvbreader.c and output it to Python for processing. So how do we modify the reader to output that data?

Huevos commented 8 months ago

100=142=dvb://85.B.8E;1;1 What is 142?

LraiZer commented 8 months ago

dvbsnoop -b -n 100 0x08ae >/tmp/bingo.txt

The data already contains CR/LF, so just output the data as binary data with above command and open with notepad to view.

There is all sort of junk in there along with lcn data, like some other useful data such as bouquet names etc. but whether this data is of any use other than looking at it, may remain a mystery how to reliably parse this lot. bingo.txt

LraiZer commented 8 months ago

100=142=dvb://85.B.8E;1;1 What is 142?

I would guess from the data, that it is assigned as Serien? or is it a channelID? 0x02=100 0x02=101 0x02=110 0x02=123 0x02=124 0x02=126 0x02=127 0x02=13 0x02=136 0x02=14 0x02=142 0x02=147 0x02=23 0x02=772 0x02=8652 0x02=8676 0x02=Serien

Huevos commented 8 months ago

So somehow we need the dvbreader.c to output this data.

LraiZer commented 8 months ago

Looks like a different ETag for many different data files. ETag: "1a5-600aaef11d8ef" looks like sky de bouquets file.

ETag: "1a5-600aaef11d8ef" data is currently sliced across section 2,3,4,5 of 58 sections on table_id_extension 85.

Table_ID: 158 (0x9e) [= User private]

table_id_extension: 85 (0x0055)

Section_number: 2 (0x02) Section_number: 3 (0x03) Section_number: 4 (0x04) Section_number: 5 (0x05)

Last_section_number: 58 (0x3a)

Specific "User_Defined-decoding" would be needed to be coded from scratch in a reader.

yannydodo12 commented 8 months ago

Sorry for opening a can of worms. I've edited the sat_192_sky_deutschland.xml In the custom folder and got what scanned in in the correct order for now 😁 If you want to use it until ye get her working with all the channels. here it is..

custom sat_192_sky_deutschland.xml.zip

LraiZer commented 8 months ago

dvbsnoop -n 10 0x08ae >/tmp/bingo.txt We need to capture this with the dvbreader.c and output it to Python for processing. So how do we modify the reader to output that data?

So can't a new protocol be coded for this in the python reader code that uses current read_ts function in the dvbreader..c code which is already coded to dump the required sections header info and content data?

Just read all sections first, concatenate the sections content data sequentially 0 to last_section. Then usie the ETag as a delimer to determine each config file beginning and end data.

Huevos commented 7 months ago

The read_ts project was never finished. We need to see what that outputs. It is so long ago I really can't remember what is was supposed to do.

LraiZer commented 7 months ago

Maybe the read_ts code could eventually be recoded and turned into the skyq config reader for this?

Had another quick look at the data, to see what is hidden and to work out how we put back together.

So looks like we have to read section 0 first to get the filenames in the order they are sent, then read the rest of the sections sequentially to get the .txt file data which is followed by the .headers file data.

Q-skyTVServiceGenres.txt Q-skyTVServiceGenres.txt.headers Q-kbwHDTVchannels-1.txt Q-kbwHDTVchannels-1.txt.headers Q-kdgTVFavouriteChannels-2.txt Q-kdgTVFavouriteChannels-2.txt.headers Q-kbwTVServiceGenres.txt Q-kbwTVServiceGenres.txt.headers Q-umgTVFavouriteChannels-32.txt Q-umgTVFavouriteChannels-32.txt.headers Q-umgHDTVchannels-32.txt Q-umgHDTVchannels-32.txt.headers Q-skyIPTVchannels-123.txt Q-skyIPTVchannels-123.txt.headers Q-SkyDNonRecordableServices.txt Q-SkyDNonRecordableServices.txt.headers Q-deuTVFavouriteChannels-4.txt Q-deuTVFavouriteChannels-4.txt.headers Q-ausTVFavouriteChannels.txt Q-ausTVFavouriteChannels.txt.headers Q-skyTVFavouriteChannels-120.txt Q-skyTVFavouriteChannels-120.txt.headers Q-TVServicesGenresLabels.txt Q-TVServicesGenresLabels.txt.headers Q-kbwTVFavouriteChannels-1.txt Q-kbwTVFavouriteChannels-1.txt.headers Q-kdgTVServiceGenres.txt Q-kdgTVServiceGenres.txt.headers Q-pknHDTVchannels-8.txt Q-pknHDTVchannels-8.txt.headers Q-skyHDTVchannels-256.txt Q-skyHDTVchannels-256.txt.headers Q-aucTVFavouriteChannels-8.txt Q-aucTVFavouriteChannels-8.txt.headers EMMacquisition.txt EMMacquisition.txt.headers Q-skyIPTVchannels-120.txt Q-skyIPTVchannels-120.txt.headers Q-kbwTVFavouriteChannels.txt Q-kbwTVFavouriteChannels.txt.headers Q-pknTVServiceGenres.txt Q-pknTVServiceGenres.txt.headers Q-kdgTVFavouriteChannels.txt Q-kdgTVFavouriteChannels.txt.headers Q-skyHDTVchannels-4.txt Q-skyHDTVchannels-4.txt.headers Q-deuTVFavouriteChannels.txt Q-deuTVFavouriteChannels.txt.headers Bmails.xml Bmails.xml.headers Q-umgTVServiceGenres.txt Q-umgTVServiceGenres.txt.headers Q-bdl0x1000D.txt Q-bdl0x1000D.txt.headers Q-pknTVFavouriteChannels.txt Q-pknTVFavouriteChannels.txt.headers Q-aucTVFavouriteChannels.txt Q-aucTVFavouriteChannels.txt.headers Q-telecolumbusHDTVchannels-16.txt Q-telecolumbusHDTVchannels-16.txt.headers Q-ausTVFavouriteChannels-4.txt Q-ausTVFavouriteChannels-4.txt.headers Q-pknTVFavouriteChannels-8.txt Q-pknTVFavouriteChannels-8.txt.headers Q-umgTVFavouriteChannels.txt Q-umgTVFavouriteChannels.txt.headers Q-kdgHDTVchannels-2.txt Q-kdgHDTVchannels-2.txt.headers Q-aucTVServiceGenres.txt Q-aucTVServiceGenres.txt.headers Q-liwestHDTVchannels-64.txt Q-liwestHDTVchannels-64.txt.headers Q-skyIPTVchannels-128.txt Q-skyIPTVchannels-128.txt.headers

ETag: "5cdafe0c12960-7a" ETag: "5ce199cc9c544-10e1" ETag: "5f4bed7769d7b-10cd" ETag: "5f6d96c029d00-269" ETag: "5f6d96c029d00-272" ETag: "5fefb737ed3b7-35" ETag: "5ffd0a9e21600-1304" ETag: "5ffd0a9f15840-12ad" ETag: "600aaef1 1d8ef-1a5" ETag: "600aaef1 31d20-1a5" ETag: "600aaef1 6fd50-18d" ETag: "600aaef1 c3549-18d" ETag: "601ec00816c24-a1" ETag: "604842495549 a-1ac" ETag: "604842495549 f-9588" ETag: "6048424 9a84b8-1a3" ETag: "6048424 9fa55b-76" ETag: "6048424 a4bde6-365e" ETag: "6048424 a6021d-e63" ETag: "6048424 a9de66-906" ETag: "6048424 ab2685-306e" ETag: "6048424 af04f1-1ae" ETag: "6048424 b18754-9514" ETag: "6048424 b6a3ea-195" ETag: "6048424 b951a2-269" ETag: "6048424 bbbadc-272" ETag: "6048424 be6fff-1a3" ETag: "6048424 c38c96-195" ETag: "60771b6a238d1-1a3" ETag: "607d27ff 849c1-c8b" ETag: "607d27ff b010f-8c5" ETag: "607d27ff dc40c-adc" ETag: "609914d e87529-6573" ETag: "609914d e9b58c-3658" ETag: "609914d edb8bf-332f" ETag: "609914d eef158-35f1" ETag: "609914d f3005a-285e"

For example in data bellow:

46=8680
47=5503
48=9002
49=377
50=99
51=8678  ETag: "1ae-6048424af04f1"
Content-Length: 430
0x02=Serien
0x03=Sport
0x04=Filme
0x05=Dokus
0x06=Kids
0x07=Musik
0x08=News & Intl.
0x09=Regional
0x0A=Marktplatz
0x0B=Public Value
0x0C=Unterhaltung
.Ø@5žð-  Á   ÿ             ï¿ÿbcc_XXSI ñæwžÿÌ Uí:ÿ     ETag: "a1-601ec00816c24"
Content-Length: 161
 1=826
2=1110
3=2003
4=7500

Would give us the two files: Q-TVServicesGenresLabels.txt Q-TVServicesGenresLabels.txt.headers

Q-TVServicesGenresLabels.txt contents would be: 0x02=Serien 0x03=Sport 0x04=Filme 0x05=Dokus 0x06=Kids 0x07=Musik 0x08=News & Intl. 0x09=Regional 0x0A=Marktplatz 0x0B=Public Value 0x0C=Unterhaltung

Q-TVServicesGenresLabels.txt.headers contents would be: ETag: "a1-601ec00816c24" Content-Length: 161

Etag comes after its related data. First part of Etag "a1-" is the hex value of content-length, ie a1=161 Also seen is that the byte before the file name Q-TVServicesGenresLabels.txt when we need to read the file names gives us the file name length.

LraiZer commented 7 months ago

Had a little look at this some more, but don't know if anyone else is looking into this also, so just sharing where i am at. I will be out of country soon, and too busy over xmas to look at it again until maybe sometime next year.

Still looking at working out how to parse the data reliably. Parsing filenames still looks tricky, but there is some more new parsing info now in the debug notes.

I've made a little config dump tool to read the stream data. It can be used instead of this dvbsnoop binary dump command, as the data output is more or less the same. The config dump tool outputs only the required sections content data, and in correct order.

dvbsnoop 0x8ae -b -n 60 -f 0x9e.00.55.00 -m 0xff.00.ff.00 >/tmp/skyq_config-binary-dump.txt

skyq_config_dump.zip

Huevos commented 7 months ago

Hi, thanks for your work on this.