Add Bis TV on Hot Bird 13°E

[ghost]

This provides a bouquet for the French subscription service Bis TV on Hot Bird at 13°E. A slightly different version of the package is also provided on Eutelsat 5 West A.

[AbuBaniaz]

Are you sure this is correct? Slightly out compared to this list. (The list may be wrong) http://www.expertmultimedia.ch/satellite/bis-televisions/bis-tv-liste-des-chaines

[ghost]

That list is indeed wrong, as is every other Bis TV channel list I've ever found online. Even the list on the official Bis TV web site is inaccurate. Since no two online sources of information agree, I think the best we can do is trust the Bouquet Allocation Table we get from the satellite.

The bouquet ID is correct, but you're right that there was a problem. Due to a technical issue in the ABM code (more a design flaw than a bug), an ABM scan would only occasionally generate the complete bouquet. Most of the time it would omit most of the channels. I've updated the pull request with a fix for this.

Here's the technical explanation:

The Bis TV bouquet on Hot Bird spans two separate network IDs. NID 319 belongs to Bis TV and contains most of their primary subscription channels, but several of the subscription channels are part of NID 318 which also contains many FTA channels that are also advertised as part of the Bis TV offer. The BAT (published on a TS in network 319) correctly lists all of these channels in both networks.

On Hot Bird you will find multiple different NIT-other tables being published in a single transport stream, and that is the case here: the TS where we get the Bis TV BAT contains NIT-other tables for six additional networks besides the one we want. Unfortunately, the ABM scanning code assumes there would only be one NIT-other table, so it simply grabs the first one it sees in the transport stream. Because of this, successful generation of the Bis TV bouquet would depend on which NIT-other table was the first to be broadcast after the scan was started. If it was the table for network 318, the complete bouquet would be generated. If it was for any of the other six networks, the network 318 channels would effectively not exist and the bouquet would contain only the network 319 channels.

I've added the ability to explicitly specify which NIT-other table should be used, so the complete Bis TV bouquet can now be reliably generated. This leaves in place the limitation that only one NIT-other table can be used. A better (and more difficult) fix would be to redesign the relevant code to process all of the NIT-other tables, but until someone actually discovers a bouquet that needs 3+ networks I'm sure it's not worth the effort!

[AbuBaniaz]

This crashes on other providers because they do not have the newly added "nit_other_network_id". Bis TV was more complete though.

Can you edit it so that it bypasses this if not in provider file. It crashed on Freeview UK and Virgin UK

<   933.808>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 487, in doScan
<   933.809>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 217, in read
<   933.809> KeyError: 'nit_other_network_id'
<   933.809> [ePyObject] (CallObject(<bound method AutoBouquetsMaker.doScan of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

[ghost]

Sorry about that. I only have DVB-S so I can't test DVB-C or DVB-T, but I believe this update should fix the problem.

[AbuBaniaz]

Hello again, still crashing.

For cable this is what happens

<  1495.968> [Manager] Reading cable_uk_virgin (hd_41050_dvbc_uk)...
<  1495.969> [Tools] Transponder provider name cable_uk_virgin
<  1496.121> [DvbScanner] Adapter 0
<  1496.121> [DvbScanner] Demuxer 1
<  1496.122> [DvbScanner] Frontend 2
<  1496.122> [DvbScanner] DVBType dvbc
<  1496.122> [DvbScanner] NIT pid: 0x10
<  1496.123> [DvbScanner] NIT current table id: 0x40
<  1496.123> [DvbScanner] NIT other table id: 0x41
<  1496.123> Traceback (most recent call last):
<  1496.123>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 487, in doScan
<  1496.124>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 217, in read
<  1496.124> KeyError: 'nit_other_network_id'
<  1496.124> [ePyObject] (CallObject(<bound method AutoBouquetsMaker.doScan of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

For terrestrial

<  1813.538> [Manager] Reading terrestrial_uk_freeview (crystal_palace_____)...
<  1813.539> [Tools] Transponder provider name terrestrial_uk_freeview
<  1813.850> [Tools] 3 custom transponders found for that region.
<  1813.851> [DvbScanner] Adapter 0
<  1813.851> [DvbScanner] Demuxer 0
<  1813.851> [DvbScanner] Frontend 3
<  1813.852> [DvbScanner] DVBType dvbt
<  1813.852> [DvbScanner] NIT pid: 0x10
<  1813.852> [DvbScanner] NIT current table id: 0x40
<  1813.853> [DvbScanner] NIT other table id: 0x0
<  1813.853> Traceback (most recent call last):
<  1813.853>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/main.py", line 487, in doScan
<  1813.853>   File "/usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/scanner/manager.py", line 217, in read
<  1813.854> KeyError: 'nit_other_network_id'
<  1813.854> [ePyObject] (CallObject(<bound method AutoBouquetsMaker.doScan of <class 'Plugins.SystemPlugins.AutoBouquetsMaker.scanner.main.AutoBouquetsMaker'>>,()) failed)

[Huevos]

You need to look for another way to do it without hacking the Python code. For example, dependent provider.

[ghost]

There is a problem in the Python code that makes it unable to correctly process the network's BAT, and it cannot be worked around with a dependent provider.

[Huevos]

If there is a problem with the BAT, why are you touching the NIT code.

[ghost]

There isn't a problem with the BAT. The BAT is complete and correct. There is a problem with the way the Python code processes the BAT, or rather with the way the Python code deals with the other network ID referenced in the BAT which is triggered by the unusual way that Eutelsat publishes multiple different NIT-other tables in a single TS. Typically there is only one NIT-other in a given TS so the Python code currently assumes that will be the case and breaks when it is not.

[Huevos]

According to the DVB standard, "Any sections of an NIT which refer to a network other than the actual network shall take a table_id value of 0x41 and the network_id shall take the value allocated to the other network in TS 101 162"

[ghost]

And that's exactly the case, except that here there are multiple NIT-other sections referring to multiple other networks - all of them with a table ID of 0x41 and the network ID correctly set to that of another network. Look at this dvbsnoop excerpt from the TS where the Bis TV BAT is published:

# dvbsnoop 0x10 | grep -A 5 'Table_ID: 65'
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 44 (0x002c)
Network_ID: 272 (0x0110)  [= >>ERROR: not (yet) defined... Report!<<]
--
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 650 (0x028a)
Network_ID: 318 (0x013e)  [= Eutelsat Satellite System 13.0E / Satellite / ETSO]
--
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 44 (0x002c)
Network_ID: 702 (0x02be)  [= >>ERROR: not (yet) defined... Report!<<]
--
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 44 (0x002c)
Network_ID: 64511 (0xfbff)  [= --> please lookup at http://www.dvb.org]
--
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 44 (0x002c)
Network_ID: 176 (0x00b0)  [= >>ERROR: not (yet) defined... Report!<<]
--
Table_ID: 65 (0x41)  [= Network Information Table (NIT) - other network]
section_syntax_indicator: 1 (0x01)
reserved_1: 1 (0x01)
reserved_2: 3 (0x03)
Section_length: 44 (0x002c)
Network_ID: 200 (0x00c8)  [= >>ERROR: not (yet) defined... Report!<<]

There are six different Network_IDs there. (The same six keep then coming down on a loop.) The BAT is for network 319, and includes some additional services that are part of network 318. If the ABM scan is run at the right moment so that when it goes to look for an NIT-other table 318 happens to be the next one to be sent, everything's fine and the bouquet gets generated correctly. But ABM currently looks for one and only one NIT-other table - the first one transmitted after the moment it goes looking. If any of the five other NIT-other tables happens to be the next one sent, the bouquet generation fails.

[Huevos]

Does your providers file work with a pre-scan?

[ghost]

If you mean what I think you do, no. The root of the problem here is that because ABM hasn't scanned network 318, it basically thinks that those channels don't exist so when it sees them in the BAT it doesn't assign them LCNs. (Except of course on those occasional scans where I'm lucky enough to hit the 318 jackpot.)

The more I look at other providers, the more I think the really unique factor in this case is that the Bis TV BAT includes channels with two different network IDs. All of the other BAT providers I've looked at contain only one network ID, so NIT-other is irrelevant. In fact many of them also have multiple NIT-other sections being published, it's just that it doesn't matter because the entire BAT is covered by NIT-actual.

I don't claim my proposed fix is optimal. It's really more of a workaround than a proper fix. The optimal solution would be for ABM to process all of the NIT-other tables being sent, but doing that would require a much more significant redesign of the code and since to date there seems to be exactly one provider ever discovered where this is a problem I think I'd personally say it's not worth doing. But if the "only one NIT-other" assumption remains, the only way I can think of to get this bouquet working is to be able to specify which NIT-other should be used.

[AbuBaniaz]

On an image with no channel data, this is what I get running with your changes.

Shame that the terrestrial/cable crashes.

[ghost]

That's what you should be getting. Without the changes, a failed scan will omit 20-25 and basically everything above 58 or so.

It's occurred to me that the second crash @AbuBaniaz saw (the one after I thought I had fixed the problem) might be because of old data persisting in providers.cache. AbuBaniaz, if you're reading this, would you be able to confirm whether or not you still get the crash after providers.cache has been removed?

[Huevos]

Should be able to do what you are attempting just by editing lines 213-226 of dvbscanner.py without all the other changes.

[AbuBaniaz]

You were correct about the providers cache.

Problem is worse after the changes. If I delete the cache, the dvb-c and dvb-t providers do not show. Nothing in debug log showing why they are rejected.

[ghost]

@Huevos: What do you have in mind? If that bit of code isn't going to just grab the first NIT-other it sees, it needs to know which NIT-other it should be looking for. All the other changes exist to make it possible to pass a configuration value in to to provide that information. The self-contained alternative would be a more extensive modification to the code to make it process all of the NIT-other sections received rather than stopping at only one. Is that what you're thinking?

@AbuBaniaz: Thanks for that, I'll get it fixed.

[AbuBaniaz]

If I revert the changes, perform a conventional scan on 13e and use the previous provider file without the NIT-other specified, the bouquets created are the same. Having to scan before running ABM, is not ideal.

[Huevos]

The way I see it, it is possible to see how many sections each network has, but not possible to tell how many networks the complete table has. If that is correct the only way to know all networks have come out it to run it until a timeout like DVBSnoop does. But that is just going to make the plugin run slow.

If code is going to be added to workaround this table style with multiple networks it really needs to be able to accept more than one network, after all even on this transponder there are several.

[ghost]

I absolutely agree that a timeout should be avoided.

In all of the cases I've come across where there are multiple other networks, the NIT-other sections are transmitted in a repeating loop with the same order each time: A-B-C-A-B-C. We could collect every NIT-other section we get until we see a repetition (same network ID and section version) of one we've already got. Then we'd know we'd covered a full loop cycle.

[Huevos]

I can see problems on a few other providers. Tivusat seems to have the same bug. And other providers are looking for the other table where it does not exist. So need table id set to 0x0. Looks like NIT other could do with a bit of a rethink.

[ghost]

I've modified the NIT-other handling code to process all available sections along the lines I described above. I'll submit a new pull request with the changes.

[Huevos]

Can you just zip and attach for the time being to give us a chance to test the changes before committing them.

[Huevos]

Can you submit a new pull request with the NIT all other code please.

[ghost]

6

[AbuBaniaz]

I have merged your other request. It results in the same bouquet creation as pictured above.

You have an e-mail about the provider file.

[ghost]

Now that the NIT-other tables are being processed correctly, I've resubmitted the Bis TV (Hot Bird) provider as #7.