search

bebo-dot-dev / m3u-epg-editor

a python m3u / epg optimizer
125 stars 28 forks source link

Troubleshooting, some channels have no epg #72

Closed halfagascan closed 1 year ago

halfagascan commented 1 year ago

Using the script to trim down a m3u/xml. Some channels have no epg data associated, they do not appear in no_epg_channels.txt. Using tv_validate_file providers.xml shows: Line 1 Invalid channel-id 'NBCEast_WNBC.us' Line 1 Invalid channel-id 'FoxEast_WNYW.us' Line 1 Invalid channel-id 'CBSEast_WCBS.us' Line 1 Invalid channel-id 'ABCEast_WABC.us' After using the script to process the downloaded xml, it reports this: Line 23 Invalid channel-id 'nbceast_wnbc.us' Line 47 Invalid channel-id 'foxeast_wnyw.us' Line 67 Invalid channel-id 'cbseast_wcbs.us' Line 75 Invalid channel-id 'abceast_wabc.us' Line 275 Invalid channel-id 'foxwest_kttv.us' Line 279 Invalid channel-id 'cbswest_kcbs.us' Line 283 Invalid channel-id 'nbcwest_knbc.us' Line 291 Invalid channel-id 'abcwest_kabc.us' Line 1067 Empty description . . Line 32729 Empty description

cat my_test.my.m3u8 | grep abcwest_kabc.us

EXTINF:-1 tvh-chnum="299" tvg-id="abcwest_kabc.us" tvg-name="ABC WEST" tvg-logo="https:/my.stream/backup/1052303.png" group-title="US-ENTERTAINMENT",ABC WEST

me@thunder:~/media/provider/wget$ cat my_test.provider.m3u8 | grep nbcwest_knbc.us

EXTINF:-1 tvh-chnum="303" tvg-id="nbcwest_knbc.us" tvg-name="NBC WEST" tvg-logo="https:/my.stream/backup/1052294.png" group-title="US-ENTERTAINMENT",NBC WEST

cat my_test.m3u8 | grep nbceast

EXTINF:-1 tvh-chnum="302" tvg-id="nbceast_wnbc.us" tvg-name="NBC EAST" tvg-logo="https:/my.stream/backup/1000100.png" group-title="US-ENTERTAINMENT",NBC EAST

NBC EAST is one channel there is no epg, FOX NEWS is another. I am wondering if the underscore in tvg-id=nbceast_wnbc.us, is causing the issue. I have edited the xml file and replaced nbceast_wnbc.us with nbceast.us, but it reverts back to having the underscore after using the script. In looking at the providers xml with tv_validate_file, there are many Line Empty description, there seems to be an , but no closing , anyway to just delete the empties? Anything you could suggest, been at this for awhile, and seems like I'm spinning my wheels. Thanks

.

halfagascan commented 1 year ago

Forgot some. tv_grep foxnews my_test.xml

FOX and Friends ᴸᶦᵛᵉ News, features and interviews. FOX and Friends ᴸᶦᵛᵉ News, features and interviews.

It has info, just not displaying

bebo-dot-dev commented 1 year ago

Hi there, thank you for your issue report.

fyi the way that the script matches m3u lines to epg entries is based on m3u line tvg-id attribute value to epg programme element channel attribute value matching.

My guess is that what you're experiencing is a data quality issue, this is the best that I have without being able to see the data that you're working with.

If you're willing to strip out any/all sensitive values from your m3u and epg files (passwords, hostnames etc) and attach the files here, I'll take a look to see if I can see anything obviously wrong. Being able to see your generated output process.log will help me understand how you're executing the script too.

halfagascan commented 1 year ago

Ok, here you go test_greyridge.zip

bebo-dot-dev commented 1 year ago

Thanks, will run some tests asap and will let you know what I find.

bebo-dot-dev commented 1 year ago

Hi there, I've taken a look at what you've supplied in your attached https://github.com/bebo-dot-dev/m3u-epg-editor/files/10290876/test_greyridge.zip.

As far as I can tell, you've supplied script generated m3u and epg xml files in this zip file, not sanitised source m3u and epg xml files prior to being processed by the script.

I need sanitised source m3u and epg xml files in the state that they're in prior to being processed by the script to stand any chance of reproducing the problem that you're seeing.

Back to you, thanks.

halfagascan commented 1 year ago

my.zip

bebo-dot-dev commented 1 year ago

Thanks, restarting analysis

bebo-dot-dev commented 1 year ago

Analysis result:

  1. I ran the script against your supplied https://github.com/bebo-dot-dev/m3u-epg-editor/files/10294741/my.zip data using your supplied greyridge.json configuration from the first supplied https://github.com/bebo-dot-dev/m3u-epg-editor/files/10290876/test_greyridge.zip file
  2. I then went looking for m3u lines in the newly generated m3u file and programme elements in the newly generated epg xml data file for channel "nbceast_wnbc.us" since your comment 1 above appeared to describe this as one of the missing/problematic channels
  3. I found one m3u file line in the newly generated m3u file and multiple epg xml file programme elements in the newly generated epg xml file for the "nbceast_wnbc.us" channel

I see no issue for this specific channel.

Perhaps your problem needs to be better described with more specific details.

halfagascan commented 1 year ago

I did see info in m3u/xml for nbc and foxnews. Let me redo my tvheadend and nextpvr dockers and see if that cures the problem, maybe there's a problem with those apps. On tvh web, a search for those two channels, nothing, nextpvr displays nothing at all. Using a wget m3u/xml, works fine, but then so many channels I don't want or need. Thanks for taking time to look at this. I follow up later

halfagascan commented 1 year ago

ok, clean nextpvr, still nothing but channel list, no epg data, error logs having issues with icons, cannot find. Looking at the xml, lots of termination locations, ie missing"", mostly at the end or incorrect addressing. Removed all icon references, no change. In looking at the m3u8, issues with incorrect icon references ie: https:/, missing the second /. Not sure if these errors are part of the cause or not. There was a program for editing m3u/xml, where you could select where the icons were used, m3u8 or xml, what do you think about an option to select xml, m3u8 or none? Thanks

bebo-dot-dev commented 1 year ago

Ok, addressing the sentences that you have written in your last comment one by one:

  1. Looking at the xml, lots of termination locations, ie missing"", mostly at the end or incorrect addressing
    • I don't follow. Please reference specifics in your supplied epg xml file and subsequently generated epg xml file that I am able to see.
  2. In looking at the m3u8, issues with incorrect icon references ie: https:/, missing the second /
    • Correct, I concur, I see this problem in the data files you have supplied, there are missing slashes present in your supplied original source m3u file. The python script in this repo has done nothing to introduce this problem, it only echoes out to the newly generated m3u file what is fed in. This is a source data quality issue, the python script in this repo doesn't do anything to fix data quality issues such as this one.
  3. There was a program for editing m3u/xml, where you could select where the icons were used, m3u8 or xml, what do you think about an option to select xml, m3u8 or none?
    • Again I don't follow, I don't know what program it is that you're referring to here so I have no response to this question.
halfagascan commented 1 year ago

Ok, still troubleshooting. I've checked several iptv xml documents using tv_validate_file. Most fail the validation check for one reason or another, one because the came before the , this is solved by moving the title into the correct position, the xml I included fails because of a blank <desc>, it is left open, no closing </desc>, the method I used to resolve, was to remove <desc> thru out the document. Most of the the missing </desc> applied to sports channels, that have no programming info at this time of year.</p> <ol start="3"> <li>The program I referred to is iptvBoss, a gui program to trim m3u/xml, it has the ability to reference a no icon file, ie it requests an http:// for the the icon file, if it is blank, it inputs "tvg-logo=""".<br /> I have submitted several issue reports to my provider, the response has not been favorable, the basic gist, m3u/xml is not supported, hasn't been for more than a year, only xc support is offered. What I have learned, thus far, reading the xmltv.dtd is very confusing, nothing appears to have a required declaration, most are recommended inputs, attempting to validate the xml according to the xmltv.dtd is, for the most part a waste of time, I have created my own xml file and, with a minimum configuration, it passes the validation check. I have little understanding of the xmltv, but at the least, how can you have an open <desc> with no closing </desc>, and have that as acceptable behavior? Comparing to html, would result in failure, yet in xmltv, it is only an error? Getting back on track, failure of some channels to show guide data, is what drove me to investigate likely causes, trimming, cutting, and eliminating obvious errors is my starting point. I'm unsure if an incorrect icon http:// link would be cause for failure, as this seems to be the major issue. Eliminating any <icon-logo> link, or the possibility to limit to either xml or m3u8 would be helpful. Thanks for taking the time to read the rant. I appreciate the help, Thanks</li> </ol> </div> </div> <div class="comment"> <div class="user"> <a rel="noreferrer nofollow" target="_blank" href="https://github.com/bebo-dot-dev"><img src="https://avatars.githubusercontent.com/u/536164?v=4" />bebo-dot-dev</a> commented <strong> 1 year ago</strong> </div> <div class="markdown-body"> <p>So far as EPG XML goes, it's a reasonably simple XML schema / standard, the DTD schema is documented and maintained here: <a href="https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd">https://github.com/XMLTV/xmltv/blob/master/xmltv.dtd</a> </p> <p>It sounds like you've read this spec and you've found that most of the elements and attributes within an EPG XML document are optional and in reality, EPG XML supplier/vendors generally only populate the bare minimum requirement into their EPG XML documents to make their doc valid and viable/usable.</p> <p>If you read the spec closely you'll see that the icon element within a channel element is a required element and an icon element is expected to contain a src attribute that points to a url (a channel icon image). There is no way to drop or ignore this icon element entirely because it's a required element and if the icon element wasn't present you could expect processing applications to not accept the EPG XML as valid. Similarly if the icon element contains an invalid src attribute url I think it would be fair to expect processing applications to not accept the EPG XML as valid for that channel and that channel not fully work as expected.</p> <p>Taking this further if you were to receive an EPG XML document and it happened to contain completely invalid XML i.e. unclosed elements / unclosed attribute quotes etc and you were to attempt to process that EPG XML document with the Python script in this repo, I would expect that to not work as expected. With the Python script as it stands, you'd probably end up with no programme elements or missing programme elements for some channels in the resulting EPG XML file.</p> <p>I'm not surprised that you're not getting far talking to your provider, they're all pretty much universally the same when it comes to customer support and they generally don't care :)</p> <p>Reading between the lines, I think what you're asking for / hoping for is for the Python script in this repo to somehow be able to patch and cope with the data quality issues that you're seeing in your m3u and epg files. If so, I'm afraid this isn't going to happen because attempting to code around any number of different data quality issue types that could occur is a path that I'm not willing to go down.</p> <p>There is no bug to address here and no viable feature request either because the issue that you have is a data quality issue.</p> </div> </div> <div class="comment"> <div class="user"> <a rel="noreferrer nofollow" target="_blank" href="https://github.com/bebo-dot-dev"><img src="https://avatars.githubusercontent.com/u/536164?v=4" />bebo-dot-dev</a> commented <strong> 1 year ago</strong> </div> <div class="markdown-body"> <p>One thing you could consider doing is preprocessing your m3u and epg files before having them processed by this Python script, one way to do this would be to use <code>sed</code>.</p> <p>If you were to do something like this you could attempt to fix up any data quality issues that you might have in your files in any way you like.</p> </div> </div> <div class="comment"> <div class="user"> <a rel="noreferrer nofollow" target="_blank" href="https://github.com/halfagascan"><img src="https://avatars.githubusercontent.com/u/43622819?v=4" />halfagascan</a> commented <strong> 1 year ago</strong> </div> <div class="markdown-body"> <p>I appreciate the help, and agree that its a provider issue. Interesting comments about the icon element, I've deleted those lines containing it and it validates fine, have yet to load into tvh or nextpvr, thats my project for today. I'll close this, but update on the icon logo pass or fail, for future reference. Again, thanks for the help and feed back.</p> </div> </div> <div class="comment"> <div class="user"> <a rel="noreferrer nofollow" target="_blank" href="https://github.com/halfagascan"><img src="https://avatars.githubusercontent.com/u/43622819?v=4" />halfagascan</a> commented <strong> 1 year ago</strong> </div> <div class="markdown-body"> <p>edit the m3u8 and remove all http links, I left this in place: tvg-logo="" edit the xml file and removed http links and left this: in place<icon src=""/> Both tvheadend and nextpvr work fine. Thanks edit: does not show open/close brackets around icon src, should be icon src= left arrow double quotes,slash, right arrow nothing inside quotes edit2 forgot to add, submitted trouble ticket on this issue, in short, its cheaper to refund the money than it is to fix the issue, they kicked me out, haha</p> </div> </div> <div class="comment"> <div class="user"> <a rel="noreferrer nofollow" target="_blank" href="https://github.com/bebo-dot-dev"><img src="https://avatars.githubusercontent.com/u/536164?v=4" />bebo-dot-dev</a> commented <strong> 1 year ago</strong> </div> <div class="markdown-body"> <p>That outcome sounds normal in my experience of any IPTV supplier that I've ever been near.</p> <p>There's never willingness to fix anything, if you report a problem you're an irritation that needs to be culled :)</p> </div> </div> <div class="page-bar-simple"> </div> <div class="footer"> <ul class="body"> <li>© <script> document.write(new Date().getFullYear()) </script> Githubissues.</li> <li>Githubissues is a development platform for aggregating issues.</li> </ul> </div> <script src="https://cdn.jsdelivr.net/npm/jquery@3.5.1/dist/jquery.min.js"></script> <script src="/githubissues/assets/js.js"></script> <script src="/githubissues/assets/markdown.js"></script> <script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.4.0/build/highlight.min.js"></script> <script src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.4.0/build/languages/go.min.js"></script> <script> hljs.highlightAll(); </script> </body> </html>