Norwegian DVB-T channels wanted

[robinsmidsrod]

Hi everyone!

I tried using AutoBouquetsMaker on OpenATV 6.2 on a Dreambox DM7020HD, and ended up being blocked because there wasn't any option to find Norwegian channels distributed over DVB-T. I couldn't actually find any Norwegian providers at all in the configuration (not even for DVB-S2 or DVB-C).

What must be done to get those supported? Is there anything I can do to help things along?

[AbuBaniaz]

They have not been requested, so not added yet.

Have you tried installing and using the Terrestrialscan plugin? If that creates the bouquet in same as official order it will be relatively straight forward.

You may need raw stream recordings from some of your frequencies. Details here https://github.com/oe-alliance/AutoBouquetsMaker/tree/tools/scripts%20and%20tools

[robinsmidsrod]

@AbuBaniaz Where can I find the Terrestrialscan plugin? Is it supposed to be in the OpenATV extension list on my tuner, or do I need to look somewhere else?

[robinsmidsrod]

@AbuBaniaz If you mean a normal automatic scan with DVB-T, I've done that and all of the channels available do show up, but not in the order they're shown on the provider's homepage. The provider is rikstv.no. The channel layout is here: https://www.rikstv.no/produkter/kanalliste/

[AbuBaniaz]

There is a plugin called Terrestrial scan. Please use it and report back

[AbuBaniaz]

It will be under system plugins in order for you to install it. Once installed, it's in menu with tuning/scanning items

[AbuBaniaz]

screenshot if you need it

[robinsmidsrod]

@AbuBaniaz I've installed the terrestrialscan plugin and performed a scan. It detected 152 channels. Which file do you need from my installation to figure out the next step? I noticed an XML file was generated/updated. Where is it located?

[Huevos]

When you used TerrestrialScan did it create a bouquet? Was the bouquet in the correct order?

[robinsmidsrod]

@Huevos Yes, it created a bouquet called "Last Scanned" which contained all of the channels (with the same channels listed multiple times, probably different DVB-T towers, I know I can receive at least two different ones), but not in the order listed on rikstv.no. It also stated that it create an XML file called terrestrial.xml, but it didn't mention where it was saved.

[AbuBaniaz]

There should have been a terrestrial bouquet too. Please double check.

The terrestrial.xml file will have been created in /etc/enigma2 (it says this in the description text). Can you attach it as well as your lamedb file which is in the same location.

Also, what frequency is NRK1 for you?

[robinsmidsrod]

There is no terrestrial bouquet. I even checked with the bouquetEditor in the web interface. I only have one called Favorites (TV) and one called Favorites (Radio).

For some reason there wasn't any terrestrial.xml file in /etc/enigma2, it was straight in /etc. I've attached it below.

terrestrial-xml.txt

I had both lamedb and lamedb5 in /etc/enigma2, so I gave you both.

lamedb.txt lamedb5.txt

NRK1 is at frequency 562MHz. I've attached some screenshots with the service info.

[Huevos]

That xml file is nothing to do with this plugin. You need to enable bouquet creation and xml file creation in the plugin config screen.

If a bouquet is not created, that means there are no LCNs in the standard DVB location.

Please use the DVB Stream recorder and make stream recordings of all transport streams that are active in your area. About 30-40 seconds of each transport stream should be sufficient.

Tool to make the recordings is here: https://github.com/oe-alliance/AutoBouquetsMaker/tree/tools/scripts%20and%20tools/Stream%20recording

What receiver is it?

[AbuBaniaz]

What receiver is it?

Dreambox DM7020HD

[AbuBaniaz]

The terrestrial.xml file you uploaded is the stock file. Please follow Huevos' instructions on running the plugin.

Anyway, the frequencies I got from your lamedb file are as follows ( for reference purposes only)

<locations>
    <terrestrial name="REGION NAME HERE (Europe DVB-T/T2)" flags="5">
        <transponder centre_frequency="474001000" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="498001000" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="562001125" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="578001125" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="610001250" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="642001250" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="658001375" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="666001375" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="714001375" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="738001500" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="778001500" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
        <transponder centre_frequency="786001625" bandwidth="0" constellation="3" code_rate_hp="0" code_rate_lp="0" guard_interval="4" transmission_mode="2" hierarchy_information="4" inversion="2" />
    </terrestrial>
</locations>

[robinsmidsrod]

Here is the terrestrial.xml file from the TerrestrialScan plugin.

terrestrial.xml.txt

When I was previously trying to run it I got a "Cannot get the NIM" error because I was currently streaming from the receiver.

This was with the following settings:

Tuner: Automatic Band: UHF/VHF Europe Clear before scan: Yes Only free scan: No Restrict search to single ONID: No Create terrestrial bouquet: Yes Create terrestrial xml file: Yes

No bouquet was created. Not sure why.

My physical location is "Norway / Vestfold", but for some reason my antenna is directed against and reaching the signals for "Norway / Østfold" better than Vestfold.

[robinsmidsrod]

If a bouquet is not created, that means there are no LCNs in the standard DVB location.

What is the definition of "LCN"?

Please use the DVB Stream recorder and make stream recordings of all transport streams that are active in your area. About 30-40 seconds of each transport stream should be sufficient.

What do you mean by "each transport stream"? Does that mean each of the 11 frequencies in the terrestrial.xml file, or each and every found channel?

[Huevos]

LCN = Logical Channel Numbers. This is what we are looking for. If present this is used to sort the channel order in the bouquet.

If no LCNs are present there is nothing for us to sort the list of channels found.

No bouquet will be created if no LCNs are found.

Your provider may have the LCN list not in the standard place or no LCN list at all.

So we can try to locate where the LCN data is we need a 30.40 second recording from each of the 11 frequencies. Please use the tool that we provided a link to.

If your provider does not have LCN data in the SI tables it will be impossible to create an ordered ABM provider. It still may be possible to create a fast scan provider file with channels ordered alphabetically, but that depends what data is available on the 11 transport streams, which is why we need those recordings.

[robinsmidsrod]

I've managed to capture the transport stream for 562MHz (NRK1 Østfold) using dvbstream, as mentioned in the instructions above. Here is a link to the file: https://www.dropbox.com/s/ague224f6woo0rz/562mhz_nrk1_ostfold.ts?dl=0

If this is correct, do I need to capture more of the frequencies?

Hopefully this is useful to you in finding out more.

[AbuBaniaz]

Can you obtain all the other frequencies please?

[robinsmidsrod]

@AbuBaniaz Working on it. Did this one contain anything useful?

[robinsmidsrod]

@AbuBaniaz @Huevos Okay, here are all the recordings from all the different frequencies, and also the terrestrial.xml file:

https://www.dropbox.com/sh/rec9kcyezk7hh7m/AADHAKxmMRtqjLkznk3ovQfVa?dl=0

Hopefully you're able to get something out of those files. Some of the frequencies had somewhat poor reception, so they might not contain clean information. Hopefully it is enough to find the proper metadata.

It would be helpful if you briefly explain what process you're using to figure out what information is in the files.

[AbuBaniaz]

Did this one contain anything useful?

No, that is why we asked for more. Thanks for doing so, we will have a look at it

[Huevos]

We just look at each transport stream with software that can read the SI tables, e.g. Transedit or DVB Inspector.

[robinsmidsrod]

@Huevos After installing TransEdit and DVB Inspector I was able to find logical channel lists under PSI -> NIT (using DVB Inspector). It seems like the DVB-T provider is using the NorDig private data serialization format. You can see channel lists using the Nordig Logical Channel Descriptor V2 (0x87). I notice that I can receive both Østfold (0x3405) (562mhz_nrk1.ts) and Buskerud, Tele, Vestfold (0x3403) (786mhz_nrk_tegnsprak.ts) networks, and they should be different bouquets in the final scan results.

Carefully looking at the channel lists, it seems like channel below LCN 200 are TV, while channels in the 2xx LCN range are radio, and those channels in the 9xx LCN range are TV, but special things like sign-language transmissions and the like.

Here is a screenshot from DVB Inspector showing the LCN for one of the files:

[Huevos]

Those tables are pretty severely broken. They say every transport stream is on 794 MHz which obviously is impossible. We should be able to create a work around for this.

We already have 0x87 in the reader for satellite channels. https://github.com/oe-alliance/AutoBouquetsMaker/blob/master/AutoBouquetsMaker/lib/dvbreader.c

We need to check this is compatible with your DVB-T services, but for this we need some debug logs.

Please use the attached file (remove the .txt extension) and do an ABM scan with the "extra debug" option switched on.

File goes in: /usr/lib/enigma2/python/Plugins/SystemPlugins/AutoBouquetsMaker/providers

Post the debug logs.

terrestrial_no.xml.txt

[robinsmidsrod]

@Huevos I added the file in the folder as mentioned, but I wasn't able to find it in the providers list. Then I noticed after running xmllint on the file that there was an extra double-quote after the word "unknown". Fixing that made the provider show up. Once I did a scan I got some bouquets, but they were all messed up, all four of them.

I got some extra debug info in the logs, but I was unable to figure out where on the filesystem the log file was actually stored, so I had to do the cumbersome method of screenshotting each part of it and adding it below. Would be nice to know where the log file is for any future troubleshooting.

[AbuBaniaz]

On OpenATV

Menu > System > log settings Enable debug logs = yes You can set the location in that menu. Default is /home/root/logs

You must restart the GUI/Enigma2 for this to take effect. No separate file is written for ABM, its should be added to the system logs

Do you use Skype?

[robinsmidsrod]

Here is the debug log from the ABM scan:

Enigma2-debug-20190724_23-12-23.log

I have Skype and some other messaging systems (like Telegram). You can find more contact info here: https://robin.smidsrod.no/contact/

[Huevos]

Don't worry if the bouquets look a mess. There is currently some code missing from Python this provider will need. But looks promising.

[ABM-DvbScanner] NIT content {'logical_channel_number': 1, 'channel_list_name': 'LCN Norway', 'original_network_id': 8770, 'channel_list_id': 13, 'country_code': 'nor', 'service_id': 1005, 'descriptor_tag': 135, 'transport_stream_id': 510, 'visible_service_flag': 1}

This scan was from 562 MHz.

Can you change the frequency in the provider file to "498" MHz and scan again.

This line... configuration key="unknown"" frequency="562000000">Norway terrestrial</configuration

Also you seem to be missing 2 transport streams. On 562 MHz it shows 5 transport streams, and on 498 it shows 9 transport streams, That gives 13, but your scan only shows 11, maybe due to weak signal.

[Huevos]

Also, what area/city are we working on?

[robinsmidsrod]

@Huevos I'm located in the city of Tønsberg in Vestfold county of Norway. My roof antenna is directed against Østfold, so the frequencies broadcasted from that tower are stronger than the ones from "Buskerud, Tele, Vestfold". You are probably correct that some transport streams are missing because of a weak signal.

I'll get back to you with a scan for 498MHz in a little while.

[robinsmidsrod]

@Huevos Here is the debug log for 498MHz (Buskerud, Tele, Vestfold network). This is from a tower where I know there is a weak signal, so data corruption is highly likely.

Enigma2-debug-20190725_08-35-59_498mhz.log

[robinsmidsrod]

@Huevos Just FYI, the DM7020HD has two tuners. Tuner A is a DVB-S2 tuner connected to Thor satellite at 0.9W (no DISEQ switch) and tuner B is DVB-T connected to old roof antenna with amplifier, directed at Halden signal tower in Østfold.

I do notice in the bouquets generated that they also contain channels from the Thor satellite (Canal Digital transmissions / Telenor), so some filtering might be needed to get this to work properly.

[Huevos]

Thanks.

Do you think there is any kind of public document that shows the frequencies on each mast?

We really need to know this because the frequencies listed in the tables are wrong.

[robinsmidsrod]

@Huevos There is a coverage map available here (in Norwegian): http://ntv.no/sjekk-dekningskart/

It only has traditional UHF/VHF channel numbers, not frequencies. I'll look more to see if I can find a table with details.

[robinsmidsrod]

@Huevos As mentioned on http://ntv.no/ (a little bit down on the page) it says

Norges televisjon AS (NTV) har konsesjon for utbygging og drift av det digitale bakkenettet for TV, og frekvenstillatelse for frekvensspektrumet 470-790 MHz. NTV leverer distribusjonskapasitet til sine kunder NRK og RiksTV, og sørger for stabil utsending av TV- og radiokanaler i det digitale bakkenettet.

It also says (on ntv.no/om-ntv) the following:

FREKVENSER Norges televisjon as har konsesjon for å opprette og drive et digitalt trådløst bakkebasert senderanlegg for kringkasting. Konsesjonen gjelder for bruk i 470-790 MHz-båndet. Frekvensbruken skal være i samsvar med de tekniske spesifikasjonene i Recommendation ITU-R BT.1306 og ITU-R BT1368.

Translated it says that they have permission to use frequencies from 470-790MHz. It should be used in accordance with the ITU-R BT.1306 and ITU-R BT1368 specifications.

This means that they transmit on channels in Band IV and V, from channel 21 to channel 60.

Hopefully this should be enough to create a complete frequency table.

[Huevos]

Frequency table is simple. What we need is frequency table that correlates frequency to transport stream ID on each mast.

[Huevos]

Try the attached file in TerrestrialScan plugin. (Remove ".txt" in file name.)

Goes in: /usr/lib/enigma2/python/Plugins/SystemPlugins/TerrestrialScan.

Send file to receiver by FTP. Restart receiver. Do a scan with TerrestrialScan plugin.

See how the bouquet looks. MakeBouquet.py.txt

[robinsmidsrod]

@Huevos I'll try to do that. While I was looking through the files some more, I noticed that the frequency 474MHz was found in the 474mhz_disney_junior.ts file using DVB Inspector. It was under PSI -> NIT -> network 0x3405 -> transport stream 0x213 -> frequency list 0x62. Hopefully this can be used in some way. See attached image.

[robinsmidsrod]

@Huevos I've added in the file and scanned using TerrestrialScan. It gives me a bouquet that is mostly correct, but it mixes channels from both the masts that I can receive, so some of the channels have weak signal and some have strong. Is there some way to create multiple bouquets based on the network_id, that way you should be able to get consistent signal strength in a single bouquet? It would also be nice to split the radio channels into a specific radio bouquet.

The channels that don't match against the list on https://www.rikstv.no/produkter/kanalliste/ are the following:

channel 47 is eurosport 1 (list says DR1) channel 96-99 are swedish (not in list on rikstv.no) channel 198 is nrk p1 buskerud (not in list) channel 199 is nrk p1 vestfold (not in list) channel 995 is nrk3 lydtekst (list says nrk1 lydtekst) channel 997 is nrk1 lydtekst (list says nrk3 lydtekst)

[Huevos]

Where the plugin can see 2 transport streams with the same TSID it will automatically select the stronger one.

Which frequency is Eurosport on? Which frequency is DR1 on? Which frequency are the Swedish channels on?

[robinsmidsrod]

@Huevos Looking over the details of the generated bouquet, it is a mix of streams from multiple signal towers. If I look at NRK1 and NRK3 they are from different frequencies and different towers. They are also using different TSIDs and frequencies. What I've noticed is that the 86% signal is good, but the 54% is unusable (video/audio is breaking up).

LCN 1 NRK1 Østfold is 562MHz TSID: 0x1fe SNR 86% (25.12dB) (network: Østfold) LCN 7 NRK3/Super is 786MHz TSID: 0x0136 SNR 54% (15.75dB) (network: Buskerud, Tele, Vestfold)

With regards to the frequencies of those channels you asked about:

LCN 47 (Eurosport 1) is 490MHz TSID: 0x03fd DR1 is nowhere to be found, maybe stale channel list on website? LCN 96 (SVT1 Värmland) is 490MHz TSID: 0x03fd LCN 97 (SVT2 Värmland) is 490MHz TSID: 0x03fd LCN 98 (SVT Barn/SVT24) is 490MHz TSID: 0x03fd LCN 99 (Kunskapskanalen) is 490MHz TSID: 0x03fd

From what I can understand those channels above are coming from a Swedish signal tower, considering they have a different TSID/ONID than the other Norwegian channels.

After looking at the information about the two towers within my antenna's reach it seems like each tower is transmitting on 5 distinct frequencies (the 5 frequencies cover the entire channel package from NTV). Based on that information it seems reasonable to make one bouquet for each signal tower, as it would ensure the most consistent signal reception depending on the direction of the receiver's antenna.

[Huevos]

Hard to say without seeing a copy of 490 MHz

[robinsmidsrod]

@Huevos I've added a dump file for 490MHz in https://www.dropbox.com/sh/rec9kcyezk7hh7m/AADHAKxmMRtqjLkznk3ovQfVa?dl=0 (same as earlier) which does indeed indicate that these 5 channels (plus two more with weak signal) are found by TerrestrialScan. They are indeed Swedish.

[Huevos]

I don't think it is realistic trying to create an ABM provider for this situation. The tables are just too broken and to add to the problem you are receiving other muxes from outside Norway.

We should be able to get TerrestrailScan producing a correct bouquet though. I'll post back later with a new version.

[robinsmidsrod]

@Huevos That's too bad. If you were able to write up exactly what is wrong with the tables and what should be changed for them to be accurate (according to the standard) then I could at least try to send it to NTV and see how they respond. Most likely they'll just ignore us, but it is at least worth a try, just to see how they respond...

Getting some way of getting the standard channel list generated automatically would be a huge improvement, because right now I have to assemble it manually. If you're able to get TV and radio into their separate bouquets that would be really great, and separation according to network_id then I'd be really grateful. Thanks for all the help with tracking this down to the bottom, so far.

[Huevos]

plugin.py.txt

MakeBouquet.py.txt

[robinsmidsrod]

@Huevos Thank you for the files! I've run through a scan now and I do get a useful bouquet out of it with the settings you suggested above. The only problem is that too many of the channels in the bouquet end up coming from the weaker tower, and I have no idea why that is. Maybe my tuner is not able to get the correct signal strength fast enough, so the wrong frequency is used by the bouquet. If it was possible to in some way force the frequencies from one mast to always have priority over another when generating the bouquet I think I would be super happy and things would work out exactly as I need them to. Right now, I'd still need to manually make a bouquet because too many of the channels are unwatchable.

Also, you have a typo in the help message, it is written "Scandinavian".

[Huevos]

The problem is the signals are not marked in any way which mast they are coming from.

The plugin waits for signal strength to stabilize before storing it. If this is wrong it is a problem with the signal strength measurement coming from the drivers on your hardware... Or a signal strength algorithm problem in frontend.cpp in the image distro you are using.

OpenViX has a different algorithm but we don't build your hardware. Catch 22.

Most likely is your distro is showing decent signal for both masts even though one is weak.

The plugin is only aware of signal through the measurement taken by enigma and if this is wrong the plugin will get it wrong too.

[robinsmidsrod]

@Huevos Actually, they are. When you go into PSI->NIT you see the network_id (in DVB Inspector). That number/name you get in there represents the geographical region (or most likely mast) you're receiving.

In DVB Inspector it is listed as table_id_extension, and the one I get the strongest signal from is labeled Østfold (0x3405), which has an ONID of 8770 and a TSID of 510. If you look at the NTV channel to frequency map.txt file in my previously shared Dropbox folder, you can see that both 474MHz and 562MHz both come from Halden, Høyås, which is where my antenna is pointed at (and I have the strongest signal).

If you open up 498Mhz and 578MHz, which are both coming from Frodeåsen, Tele, that is the tower that has the weaker signal. Both of those files have the network_id 0x3403 (Buskerud, Tele, Vestfold) and the ONID 8770 and TSID 335, because they come from the same mast. That is the way you can differentiate them, as far as I can see.

Also, if you open up 490mhz you can see it has a different network_id (0x31a5, Teracom_Mux_1), and a different ONID (8945) and TSID (1021). That is the Swedish tower occasionally receive, but only in good weather. ;)

Are you able to do anything with this information?