search

bebo-dot-dev / m3u-epg-editor

a python m3u / epg optimizer
120 stars 27 forks source link

Enhancement: Generate channel numbers #6

Closed jobrien2001 closed 6 years ago

jobrien2001 commented 6 years ago

Is there any possibility to auto generate ascending channel numbers? and perhaps a starting value argument.

This would be useful for importing into tvheadend.

With that functionality we could achieve results like numbering US channels starting at 1, Canadian starting at 300, movies at 400, sports at 500... etc

bebo-dot-dev commented 6 years ago

I'm sure something like this will be possible but not at the moment because as it stands this script simply acts as an m3u "rewriter" using the fields from the original m3u that are supported in the regex search criteria defined here

I don't know a lot about tvheadend to be honest but I have a feeling that a tvh-chnum number attribute might be significant on each #EXTINF:-1 line in the m3u file for this feature to work. If you can confirm what tvheadend looks at to make this work I'll take a look to see what's possible to introduce the feature.

jobrien2001 commented 6 years ago

I just tested the attribute.

Here is a sample of a working channel number for tvh using the tvh-chnum attribute:

#EXTM3U
#EXTINF:-1 tvh-chnum="22" tvg-id="I285.65342.schedulesdirect.org" tvg-name="ID HD" tvg-logo="https://imgur.com/cFvRXD6.png" group-title="united states",ID HD
http://supersecretstreams.com/test.ts
bebo-dot-dev commented 6 years ago

ok it appears that a tvh-chnum attribute is indeed what's needed. The question now is how to go about getting it populated with a sequential number value in a way that works reasonably well for tvheadend / for what you need.

I could just start at 1 and increment up within a group and then use some new offset argument value to start the next group. Does that sound sensible?

jobrien2001 commented 6 years ago

Hello,

I was thinking about running your script once for every group I wanted, so only a starting value would be needed at each run... then combine the m3u files.

You suggestion is better as a whole but it might be more work? For example, lets say I wanted the US channels,Movies, and Live Events (in that order). how would the offset work?

I appreciate your time in this.

bebo-dot-dev commented 6 years ago

No problem, I think it's probably best to attempt to implement the best possible solution that can work over multiple groups.

To use your suggested group names as an example I think it could work like this:

  1. specify the groups to keep as normal in the --groups / -g as per existing behaviour i.e. -g="'us channels','movies','live events'"
  2. the groups are sorted in the order that was specified in the -g parameter to ensure you get the group order you want
  3. channels within groups are sorted as normal as per existing behaviour via the sort order specified in the --sortchannels / -s argument
  4. a new optional --tvhoffset / -t argument is introduced that supports an integer value being passed in i.e. -t=100
  5. if the --tvhoffset / -t argument is present / was given a value, the tvh-chnum attribute will be tacked onto each #EXTINF row in the m3u, incrementing up
  6. when the group changes the --tvhoffset / -t value will be applied to the running total to ensure that the incrementing tvh-chnum value in the next group starts at the correct value

..I think that sounds about right, it would be a non-breaking change and is a pretty flexible solution.

jobrien2001 commented 6 years ago

Ok I understand the offset. Would it possible to make the a dynamic offset to every hundred? So each group would start at 1,101, 201, 301

Lets say the US channels are 66 channels, so at row 66 calculates the current hundred like floor (66/100)=0, then the next hundred would be 100(0+1)+1=101 which would be the next value for movies. Lets say movies has 150 channels, so at channel 251 it will do the math again floor(251/100)=2, 100(2+1)+1=301. So the next group live events would start at 301

That would give it a cable like experience in kodi.

Sorry for the edits want to make sure my idea is clear.

bebo-dot-dev commented 6 years ago

Yes that's the idea, if the offset is specified as 100. I'll give it a go and I'll update here again when there's something to test in a branch.

bebo-dot-dev commented 6 years ago

There's a version ready for a test in the https://github.com/jjssoftware/m3u-epg-editor/tree/tvhoffset branch if you'd like to give a try. The new argument is called --tvh_offset / -t and the readme has been updated.

I'll hold on merging to master until we work out any kinks with it.

jobrien2001 commented 6 years ago

On the new branch, with or without the -t switch produces the same result.

root@ubuntu:~# python ./m3u-test.py -m="file:///root/vget.m3u" -e="file:///root/p2.xml.gz" -g="'Sports,United States'" -d="/root" -f="output"
2018-04-06T16:52:24.249473 performing HTTP GET request to file:///root/vget.m3u
2018-04-06T16:52:24.250757 saving retrieved m3u file: /root/original.m3u8
2018-04-06T16:52:24.251184 parsing m3u into a list of objects
2018-04-06T16:52:24.259988 m3u contains 543 items
2018-04-06T16:52:24.260066 keeping channel groups in this set(['Sports,United States'])
2018-04-06T16:52:24.261018 filtered m3u contains 0 items
2018-04-06T16:52:24.261317 saving new m3u file: /root/output.m3u8
Traceback (most recent call last):
  File "./m3u-test.py", line 505, in <module>
    main()
  File "./m3u-test.py", line 133, in main
    save_new_m3u(args, m3u_entries)
  File "./m3u-test.py", line 328, in save_new_m3u
    group_title = m3u_entries[0].group_title
IndexError: list index out of range
bebo-dot-dev commented 6 years ago

ok there is something for me to fix there related to no entries but I think there's no entries because your -g argument looks a bit off. Try this: -g="'sports','united states'"

jobrien2001 commented 6 years ago

I am using the following command: python ./m3u-test.py -m="file:///root/vget.m3u" -e="file:///root/p2.xml.gz" -g="'sports','united states'" -t=100 -d="/root" -f="output"

Producing:


#EXTM3U
#EXTINF:-1 tvh-chnum="1" tvg-name="A&E HD" tvg-id="I265.10035.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20a%26e%20hd%20.png" group-title="United States",A&E HD
#EXTINF:-1 tvh-chnum="2" tvg-name="ABC HD" tvg-id="I396.11259.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20abc%20hd.png" group-title="United States",ABC HD
#EXTINF:-1 tvh-chnum="3" tvg-name="AMC HD" tvg-id="I254.59337.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20amc%20hd.png" group-title="United States",AMC HD
#EXTINF:-1 tvh-chnum="4" tvg-name="AXS TV HD" tvg-id="I340.28506.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20axs%20tv%20hd.png" group-title="United States",AXS TV HD
#EXTINF:-1 tvh-chnum="5" tvg-name="American Heroes Channel HD" tvg-id="AmericanHeroesChannel.us" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20american%20heroes%20channel%20hd.png" group-title="United Sta$
#EXTINF:-1 tvh-chnum="6" tvg-name="Animal Planet HD" tvg-id="I282.57394.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20animal%20planet%20hd.png" group-title="United States",Animal Plan$
#EXTINF:-1 tvh-chnum="101" tvg-name="At The Races" tvg-id="I415.30505.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/at%20the%20races.png" group-title="Sports",At The Races
#EXTINF:-1 tvh-chnum="201" tvg-name="BBC America HD" tvg-id="I264.64492.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20bbc%20america%20hd.png" group-title="United States",BBC America HD
#EXTINF:-1 tvh-chnum="202" tvg-name="BET HD" tvg-id="I329.63236.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20bet%20hd.png" group-title="United States",BET HD
#EXTINF:-1 tvh-chnum="301" tvg-name="BT Sports 1" tvg-id="I413.82447.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%201.png" group-title="Sports",BT Sports 1
#EXTINF:-1 tvh-chnum="302" tvg-name="BT Sports 1 HD" tvg-id="I413.82447.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%201%20hd.png" group-title="Sports",BT Sports 1 HD
#EXTINF:-1 tvh-chnum="303" tvg-name="BT Sports 2" tvg-id="I414.82449.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%202.png" group-title="Sports",BT Sports 2
#EXTINF:-1 tvh-chnum="304" tvg-name="BT Sports 2 HD" tvg-id="I414.82449.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%202%20hd.png" group-title="Sports",BT Sports 2 HD
#EXTINF:-1 tvh-chnum="305" tvg-name="BT Sports 3" tvg-id="I417.95771.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%203.png" group-title="Sports",BT Sports 3
#EXTINF:-1 tvh-chnum="306" tvg-name="BT Sports 3 HD" tvg-id="I417.95771.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%203%20hd.png" group-title="Sports",BT Sports 3 HD
#EXTINF:-1 tvh-chnum="307" tvg-name="BT Sports ESPN" tvg-id="I426.46594.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%20espn.png" group-title="Sports",BT Sports ESPN
#EXTINF:-1 tvh-chnum="308" tvg-name="BT Sports ESPN HD" tvg-id="I426.46594.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bt%20sport%20espn%20hd.png" group-title="Sports",BT Sports ESPN HD
#EXTINF:-1 tvh-chnum="309" tvg-name="BeIN Sports 11 HD" tvg-id="beINSports11.qa" tvg-logo="http://vod6.vaders.tv:8080/logos/bein%20sports%2011%20english.png" group-title="Sports",BeIN Sports 11 HD
#EXTINF:-1 tvh-chnum="310" tvg-name="BeIN Sports 12 HD" tvg-id="beINSports12.qa" tvg-logo="http://vod6.vaders.tv:8080/logos/bein%20sports%2012%20english.png" group-title="Sports",BeIN Sports 12 HD
#EXTINF:-1 tvh-chnum="311" tvg-name="BeIN Sports Espanol HD" tvg-id="I467.76943.schedulesdirect.org" tvg-logo="https://imgur.com/JwkadXy.png" group-title="Sports",BeIN Sports Espanol HD
#EXTINF:-1 tvh-chnum="312" tvg-name="BeIN Sports USA HD" tvg-id="I620.76950.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/bein%20sports%20usa.png" group-title="Sports",BeIN Sports USA HD
#EXTINF:-1 tvh-chnum="313" tvg-name="Big Ten Network HD" tvg-id="I610.58321.schedulesdirect.org" tvg-logo="https://imgur.com/dXGOaNQ.png" group-title="Sports",Big Ten Network HD
#EXTINF:-1 tvh-chnum="401" tvg-name="Bloomberg HD" tvg-id="I353.71799.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20bloomberg%20hd.png" group-title="United States",Bloomberg HD
#EXTINF:-1 tvh-chnum="501" tvg-name="Boxnation" tvg-id="I437.73823.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/noxnation.png" group-title="Sports",Boxnation
#EXTINF:-1 tvh-chnum="601" tvg-name="Bravo HD" tvg-id="I237.58625.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20bravo%20hd.png" group-title="United States",Bravo HD
#EXTINF:-1 tvh-chnum="602" tvg-name="CBN HD" tvg-id="I267.49128.schedulesdirect.org" tvg-logo="http://vod6.vaders.tv:8080/logos/usa%20cbn%20hd.png" group-title="United States",CBN HD

The numbering and sorting seems to be off, Am i doing something wrong? (I removed the url)

bebo-dot-dev commented 6 years ago

No you're not doing anything wrong, this looks like a real bug related to no -s channel sort order being specified. I'm afraid it's too late to fix tonight though, I'll get to it tomorrow.

jobrien2001 commented 6 years ago

Works perfectly with sort by group title.

python ./m3u-test.py -m="file:///root/vget.m3u" -e="file:///root/p2.xml.gz" -g="'united states','canada','sports'" -t=100 -s="'group_title'" -d="/root" -f="output"

You could just add a note to readme and -h... or force sorting by group if tvh offset specified.

Thank you for time with this enhancement. I can test further tomorrow if you need me to. Good night.

jobrien2001 commented 6 years ago

Upon further inspection if i sort by group title it will sort the groups alphabetically, not the order the groups were entered.

if i use -g="'united states','canada','sports'" -s="'group-title'"

it will list the groups in the following order canada, sports, united states instead of united states,canada,sports.

bebo-dot-dev commented 6 years ago

Thanks for testing, it's really helpful. I've applied a couple of fixes to the tvhoffset branch and it looks like it's behaving a bit more sensibly now.

-s is no longer required for correct group sorting if you don't want to use / have no need to use -s channel sorting.

If -s channel sorting is used, an exact specific channel sort order within the desired group sort order should be achieved, otherwise channels will be alpha sorted within the desired group sort order. That's the plan ;)

haxcop commented 6 years ago

Really Nice Guys! @jjssoftware @jobrien2001 Love it!another point to notice here is tvg-id="I267.49128.schedulesdirect.org" <that string is really bad - as is difficult to sort the channels and map these correctly - It'll be better to replace it for the same tvg-name="CBN HD" for example into the same line... that needs to be also applied into the XML file for the Channel id="tvg-name="I267.49128.schedulesdirect.org" and Programme Channel="tvg-name="I267.49128.schedulesdirect.org" both for "CBN HD" that will help a lot and more exact map

bebo-dot-dev commented 6 years ago

Hi @haxcop, thanks.

I think I understand what you're saying about the channel with tvg-name="CBN HD" tvg-id="I267.49128.schedulesdirect.org". As far as I can see, this channel has no epg data in the vaders epg xml at the moment. It gets logged by the script to the no_epg_channels.txt file for this reason and I've performed a quick manual scan of the original m3u and epg files and that appears to confirm the situation.

It might be possible to further manipulate the m3u by some method but I'm struggling to see how mapping could be improved for this channel when there's no epg data available for this channel..unless it's epg data is identical to some other channel in the epg xml?

jobrien2001 commented 6 years ago

The changes works great but there are some details.

Some live Event channels are being sorted alphabetically (as they should). The problem is their names make numbering inconsistent.

Examples:

#EXTINF:-1 tvh-chnum="807" tvg-name="Channel #1" tvg-id="2499.vaders.tv" tvg-logo="" group-title="Live Events",Channel #1
#EXTINF:-1 tvh-chnum="808" tvg-name="Channel #10" tvg-id="2508.vaders.tv" tvg-logo="" group-title="Live Events",Channel #10
#EXTINF:-1 tvh-chnum="809" tvg-name="Channel #100" tvg-id="2648.vaders.tv" tvg-logo="" group-title="Live Events",Channel #100
#EXTINF:-1 tvh-chnum="958" tvg-name="MLB #1" tvg-id="39811.vaders.tv" tvg-logo="" group-title="Live Events",MLB #1
#EXTINF:-1 tvh-chnum="959" tvg-name="MLB #10" tvg-id="39835.vaders.tv" tvg-logo="" group-title="Live Events",MLB #10
#EXTINF:-1 tvh-chnum="960" tvg-name="MLB #11" tvg-id="None" tvg-logo="" group-title="Live Events",MLB #11
#EXTINF:-1 tvh-chnum="971" tvg-name="NBA #1" tvg-id="64243.vaders.tv" tvg-logo="" group-title="Live Events",NBA #1
#EXTINF:-1 tvh-chnum="972" tvg-name="NBA #10" tvg-id="64252.vaders.tv" tvg-logo="" group-title="Live Events",NBA #10
#EXTINF:-1 tvh-chnum="973" tvg-name="NBA #2" tvg-id="64244.vaders.tv" tvg-logo="" group-title="Live Events",NBA #2
#EXTINF:-1 tvh-chnum="981" tvg-name="NBC GOLD #1" tvg-id="75963.vaders.tv" tvg-logo="" group-title="Live Events",NBC GOLD #1
#EXTINF:-1 tvh-chnum="982" tvg-name="NBC GOLD #10" tvg-id="76001.vaders.tv" tvg-logo="" group-title="Live Events",NBC GOLD #10
#EXTINF:-1 tvh-chnum="983" tvg-name="NBC GOLD #11" tvg-id="76002.vaders.tv" tvg-logo="" group-title="Live Events",NBC GOLD #11
#EXTINF:-1 tvh-chnum="996" tvg-name="NFL #1" tvg-id="None" tvg-logo="" group-title="Live Events",NFL #1
#EXTINF:-1 tvh-chnum="997" tvg-name="NFL #10" tvg-id="None" tvg-logo="" group-title="Live Events",NFL #10
#EXTINF:-1 tvh-chnum="998" tvg-name="NFL #11" tvg-id="None" tvg-logo="" group-title="Live Events",NFL #11
#EXTINF:-1 tvh-chnum="1012" tvg-name="NHL #1" tvg-id="56622.vaders.tv" tvg-logo="" group-title="Live Events",NHL #1
#EXTINF:-1 tvh-chnum="1013" tvg-name="NHL #10" tvg-id="56652.vaders.tv" tvg-logo="" group-title="Live Events",NHL #10
#EXTINF:-1 tvh-chnum="1014" tvg-name="NHL #11" tvg-id="56653.vaders.tv" tvg-logo="" group-title="Live Events",NHL #11

Proposed solution:

Format live event channels with some number padding:

Channel #1  >>  Channel #001
NBC GOLD #1 >> NBC GOLD #01
NBA #1 >> NBA #01
NFL #1 >> NFL #01
MLB #1 >> MLB #01
NHL #1 >> NHL #01
bebo-dot-dev commented 6 years ago

I'm happy to hear that it's working as expected.

I'm not sure that it's a good idea to introduce channel renaming functionality but I do see that this would not be immediately apparent if the source m3u channel tvg-name values were already zero padded. It might be possible to introduce a general channel rename function (not specific to vaders) via some sort of regex rule in the future but I don't have the appetite for it at the moment.

The good news is I think there's an existing solution for this problem: the --sortchannels / -s argument enables customization of the channel sort order within groups to a specific sort order. If you were to list all of your channel tvg-name values in this argument in the exact order that you want them sorted into i.e. -s="'channel #1','channel #2','channel #3'" ..etc, then you should get the exact sort order that you're after.

This is basically what I said in this previous comment when I said: "If -s channel sorting is used, an exact specific channel sort order within the desired group sort order should be achieved, otherwise channels will be alpha sorted within the desired group sort order."

I think you're seeing natural alphabetic sorting by tvg-name at the moment. Give --sortchannels / -s a try, I think it fixes this problem.

haxcop commented 6 years ago

@jjssoftware @jobrien2001 I don't why you have the inconsistent channel ordering. but mine looks ok and follow like that until the end which is btw more than 100 Channels

bebo-dot-dev commented 6 years ago

Hi @haxcop, I think the thing that @jobrien2001 describes with channel ordering for channels in the Live Events group is a Tvheadend problem. I'm not a Tvheadend user myself so I can't see exactly what the problem is..but I use --sortchannels / -s to arrange the channel order into the exact order I want so I might never see this problem anyway so it's not a problem for me here.

If he comes back with anything we might know more.

jobrien2001 commented 6 years ago

The issues i had were due to Vader's m3u. I did some regex substitutions for all live events to add number padding. I also changed some channel's group-title (vader labels some sports as their country instead of sports).

The script works great. I cant find any problems.

Thanks for the enhancement

bebo-dot-dev commented 6 years ago

Thanks for the update. I've seen intermittent problems with vaders from time to time where the quality of data in their m3u and epg degrades and recovers without explanation. Sometimes it can look like a bug. I think it goes with the territory ;) Merging..

haxcop commented 6 years ago

Thanks Guys!.