K-vanc / Tempest-EPG-Generator

The fastest XMLTV formatted EPG Generator with GUI and the lowest memory usage
https://hub.docker.com/r/kvanc/tempest_epg
Other
108 stars 19 forks source link

Its possible edit online xml or local file? #15

Closed Toei79 closed 1 year ago

Toei79 commented 1 year ago

It can be handy to do it

I have this one but I don't use all channels there several epgs from online channels, pluto,plex, samsung Its been working for years

https://i.mjh.nz/

K-vanc commented 1 year ago

I will have a look. For sure it can be done easily but based on my previous experience, they are not so stabile against burst for grabbing since server is just for hosting files

Toei79 commented 1 year ago

Ok great

I'm gonna ask the guy there if other options for it, fir the grabber part

Toei79 commented 1 year ago

this one? its someway similar lots of epgs https://epgshare01.online/

K-vanc commented 1 year ago

This can be done also But is there any reason for grabbing from already ready-to-use epg? You can directly copy/paste the link into your player. Anyway, i will try to do but currently my mind is full with the news from Turkey and will go there on this Thursday. So for the time, let's say soon.

Toei79 commented 1 year ago

Only useful there its pluto stuff online channels, I mean other can get with the other site files in tempest. Its kinda like the firts one its only online tv, firts one its more channels

Toei79 commented 1 year ago

If its hard then let like that

thanks it comes already with lots of options

K-vanc commented 1 year ago

It is not hard at least for me but I am leaving for vacation tomorrow and was fully loaded at work for things to be finished before go and others to be handed over. To be honest, couldnt find any time for Tempest since last week

K-vanc commented 1 year ago

Hi, i.mjh.nz has been added to Multi Nation folder with updated channel creation instruction.

Toei79 commented 1 year ago

Hi, i.mjh.nz has been added to Multi Nation folder with updated channel creation instruction.

HI Amazing thanks! im going to check i did the script but its easier this one.

i found this one can be useful kinda like zap2it for what i see https://www.tvpassport.com/tv-listings

they do descriptions.

any suggestions for a dummy epg? i did a few googling. i have one primarily that use with web grab plus then i modify.

and mix with this one, so its kinda like a epg but that always repeat just chance day times

https://www.reddit.com/r/IPTV/comments/la62s4/create_my_own_dummy_epg/

https://github.com/yurividal/dummyepgxml/blob/main/dummycreator.sh

this one also more limited but easy to build.

Toei79 commented 1 year ago

Hi, i.mjh.nz has been added to Multi Nation folder with updated channel creation instruction.

hi i read this https://github.com/K-vanc/Tempest-EPG-Generator/blob/main/Siteconfigs/Multi%20Nation/Info%20for%20tvguide_com_i_mjh_nz.md

and for example i texted Plex/

for Plex/ => Plex but keeps loading. image

but for tvguide multi loads are fine,

K-vanc commented 1 year ago

Hi. Tvpassport is already available in US and Canada folders.

I dont have idea for dummy scripts but a few days ago, another user asked the same and explained how can create siteconfig for dummy channels;

https://github.com/K-vanc/Tempest-EPG-Generator/issues/20#issuecomment-1451489983

You should enter only left side as user key so it will be only;

Plex/

If it is freezing while tvguide.com is running, this means you need to reset Linux ownership/permissions of new files after each copy/paste or has to set user groups properly to allow you copy/paste and read/write/execute Of same file from server.

Toei79 commented 1 year ago

ok gotcha maybe its that, that happened before, i gonna check it thanks

Toei79 commented 1 year ago

Hi. Tvpassport is already available in US and Canada folders.

I dont have idea for dummy scripts but a few days ago, another user asked the same and explained how can create siteconfig for dummy channels;

#20 (comment)

You should enter only left side as user key so it will be only;

Plex/

If it is freezing while tvguide.com is running, this means you need to reset Linux ownership/permissions of new files after each copy/paste or has to set user groups properly to allow you copy/paste and read/write/execute Of same file from server.

question if want use a different places for tvguide multi, i just need go to channel generator and tempest generate one for each? im guessing yes suppose.

K-vanc commented 1 year ago

With 1 userkey(zip/postal code etc.), you can create 1 channel list. You can select and save the channels you want from Configurator and create new channel list with another userkey. You can save multiple channels generated with diff userkeys from same siteconfig, into the same configuration file.

Toei79 commented 1 year ago

With 1 userkey(zip/postal code etc.), you can create 1 channel list. You can select and save the channels you want from Configurator and create new channel list with another userkey. You can save multiple channels generated with diff userkeys from same siteconfig, into the same configuration file.

question, about tvpassport, it gets descriptions or just title of epg? i just see there its descriptions on passporttv but when i get epg its just titles ,

K-vanc commented 1 year ago

No, all the information including desc, actor, director, presenter, showicon is available in the detail pages. If you didnt set "Index-Only" grabbing mode in your configuration, you should get all data. I tried to check with website but currently it is down. Also it would be great if you just write channel_id that you faced problem so I wouldnt try to guess it inside 8000 channels

Toei79 commented 1 year ago

No, all the information including desc, actor, director, presenter, showicon is available in the detail pages. If you didnt set "Index-Only" grabbing mode in your configuration, you should get all data. I tried to check with website but currently it is down. Also it would be great if you just write channel_id that you faced problem so I wouldnt try to guess it inside 8000 channels

common behavior its don't have details about programs its for what i see maybe im wrong from what you explaining now, but for sure tvguides comes like that without, its not actor stuff its whats the name of episode and description what im thinking, thats where zap2it its pretty good. i was thinking its only zap2it have the feature and be update, i dont touch nothing and i get some different types of stuff in the guide, i mean some countries i get detailed data about shows others just the titles, i gotta check index only if i have activated or not. i was thinking its a standard , i have to check if i have activited index only in my grabbing mode.

i know you cant get detailed for all channels, you lucky to get epg honestly sometimes or dont need any extra info in some cases just the title its fully fine. but maybe most watched channels have those. not all guides comes with it.

K-vanc commented 1 year ago

anyway, I will have a look when website is online again but all main US siteconfigs have all data as desc, actor, category etc. Maybe not for all channels and all shows but generally they are available.

K-vanc commented 1 year ago

I have checked tvguide.com now and it has almost all except cast information which is not exist in website. 123.txt

Toei79 commented 1 year ago

I have checked tvguide.com now and it has almost all except cast information which is not exist in website. 123.txt

image

man you right. probably and i do some wrong. i was checking tvguide from other countries where its not that type data but i move to my local one and there its, my bad, i was surprised, i not check on tempest i check there so i suppose tempest get what you see there, but i check other one.

still its more accurate detailed zap2it, and look icon stuff too. honestly i never tough that because tvguide its most famous, but i know zap2it its most used i guess you see lots scripts for it. icon stuff, zap2its its more detailed but still have lots ramdom ones like tvguide but fill more those than tvguide.

Toei79 commented 1 year ago

anyway i try again and run to get detailed stuff, sorry probably most stuff its in the guide, and i come ask you , just fear the guide lol because feels too technical still friendly , i can handle more tempest than webgrab plus , yours its more automatized easy to handle . i used webgrab for dummy ones. and no go father than that.

K-vanc commented 1 year ago

No worry, zap2it also in my to-do list. You will get it soon

K-vanc commented 1 year ago

epgshare01 also added to Multi Nation folder with channel creation instruction. I am closing this issue

ghost commented 1 year ago

i.mjh.nz is not grabbing data. output is "Request Error(url1): The requested URL returned error: 404"

K-vanc commented 1 year ago

hi. almost for every "NOT WORKING" comment, I am writing same thing as which channel/userkey etc. That source website has many different epg files with tons of channels. I cannot try 1 by 1 till I find your "not working" channel. I tried random 3 and it works. You have 404 error so seems you are requesting something wrong. Try to rechannel create and update new channels in your configuration file

image

ghost commented 1 year ago

I checked with (userkey - SamsungTVPlus) , 805 channel generated successfully, but not show grabbed for all channels.

K-vanc commented 1 year ago

userkey shall be;

SamsungTVPlus/

use the left side of codes.txt including all chars except " =>". after that you should generate 1242 channels, not 805. So please update your channels in config again after generating 1242 channels

Toei79 commented 1 year ago

I checked few ones and are getting channels fine , the one not working was the one with all sources in my case

Also check rights from the files thah happened me twice

There some channels that no one working by default, for example locomotion its not working i used this epg so i know its not despite to be there

ghost commented 1 year ago

I have checked with both "SamsungTVPlus/" and "SamsungTVPlus" both are generating same 1242 channels and 805 with duplicate channel remover. Previously its not grabbing data, now its grabbing data.

ghost commented 1 year ago

I think its source issue, they have epg time span of only 5 to 6 hours for SamsungTVPlus.

K-vanc commented 1 year ago

Not all channels have epg data in source. Tempest is not data provider but grabbing tool of existing data.

Good to hear. Now it is grabbing because of that "/" at the end. It is hardcoded into channel id and necessery for grabbing so be sure of userkey. It shall be same as shown on codes.txt file.

ghost commented 1 year ago

Yeah! Thanks for quick and supportive response.

Toei79 commented 1 year ago

No, all the information including desc, actor, director, presenter, showicon is available in the detail pages. If you didnt set "Index-Only" grabbing mode in your configuration, you should get all data. I tried to check with website but currently it is down. Also it would be great if you just write channel_id that you faced problem so I wouldnt try to guess it inside 8000 channels

what i do i mean i dont set Enable Index-Only Mode , but then it keep running forever , i let like two hours still not getting done, if i set its good to go in less 15 minutes. so im thinking the slow comes from one those siti files? in my case i use like 10 siti ini files, i tried zap2it alone and finish fast. with Index-Only Mode no activated

Toei79 commented 1 year ago

im going to do something i create a new one and add one by one i and see which one slow the grab process

K-vanc commented 1 year ago

Hi @Toei79 . we are categorizing siteconfigs into 2 as all-index and detail-page ones. All-index siteconfigs, contains all show data in its index page and when Tempest makes a request for day, it receives all information needed and directly start parsing. Zap2it is an example for all-index siteconfigs. if you make 5 days of request with it, Tempest will generate 5 urls for each day and parse data for xmltv elements. Lets say you grabbed 500 shows on 5 days for this channel and it all done with 5 url calls only.

Detail-page siteconfigs, contains only limited or minimum information(such as uri part of show etc) on its index page and all other information( as example description, cast info, episode details, showicon/fanarts etc.) stored in another url location and to gather it, you need to make seperate url requests. This is a way of websites to speed-up their server requests with less generic data. Tvtv.us is an example for detail-page siteconfigs. f you make 5 days of request with it, Tempest will generate 5 urls for each day and parse only existing data as start time, title, channel logo and uri part needed for other show details. At this point Tempest will create another url with uri part captured and call that url to get data of other details such as description, showicon, category etc. and parse data for xmltv elements. Lets say you grabbed 500 shows on 5 days for this channel and to get all shows with full information, Tempest made 5 url calls for each day + 1 additional url call for each show (1x500) and resulted as 505 url call for same channel and for same amount of shows.

if you consider 1 url response as 1 second average, you will finish grabbing in 5 seconds in all-index siteconfig while it will finish in 505 seconds (~8.5 mins) in detail-page siteconfig.

You may enable index-only mode to skip additional information grabbing and it will speed up while you will miss some details but for some siteconfigs, detail-page is mandatory since even show start/stop times are available only in detail pages.

Basicly, I am not telling index-only better since it is a matter of choice as usually detail page siteconfigs have much more rich epg data when compared with their index-only alternatives but for sure, they are faster. You have to decide which one is suitable for your needs. Also you need to consider, detail-page grabbing speed is directly related with your internet speed and multi-tasking capability of your device. As my real time example, I have 2 grabbing devices as Rpi3b+ and rpi0w which both are low-end devices but Rpi3b+ has much better multi-tasking than rpi0w. When I run my daily config with 293 channels/3days with several detail-page channels(%25 of total);

Rpi3b+ => finishes on 11-11,5mins rpi0w => finishes on 23-25 mins

basicly, main difference is processor speed (since Tempest using same amount of RAM on both devices) with multi-tasking ability and result is more than x2

Toei79 commented 1 year ago

Im pretty sure Tvtv.us slow down things i replaced with zap2it and speed up

siteconfigs into 2 as all-index and detail-page ones.

There some code on those to separate it? right im guessing sorry i just run the thing and skip guides pretty sure its there

Anyway tempest feel light weight in the way it runs.

K-vanc commented 1 year ago

Hi. Tvtv.us was one of the best siteconfig as all index with full epg data once I made it first and published. Couple of months ago, they made a full change on their server structure and changed to detail-page. It is not only detail-page problem for them since there are lots of detail-page siteconfigs but grabbing is relatively fast. Usually, we consider as;

average grabbing speed < 1second per show

on detail page siteconfigs as good. Tvtv.us server's respond speed also low(as average 2seconds per show) and this makes it terrible especially for several days (+2 and more) grabbing.

I dont understand your question here;

There some code on those to separate it? right im guessing sorry i just run the thing and skip guides pretty sure its there

You want tvtv.us siteconfig as index-only? Usually I am sharing siteconfigs as full and users may edit results by inverter option except siteconfigs which has only showicon etc. on detail page. On this kind of cases, I am sharing 2 siteconfigs as with and without detail page but for tvtv.us, I believe I can break that rule for it..

I disabled detail pages on below unofficial version and will not host in official repo but people may use from here;

[ENC][EX]tvtv.us_1.siteconfig.txt

Just change the extension as .php after download. Note that there is no description and some other elements since they are available in detail pages

Toei79 commented 1 year ago

oh sorry no i mean i was unaware of the difference of both type of siti files, so i though there some to make different when you see the siti files there? thats what i try to say, its been great so far idk what slow down my first xml one , i did other xml file and runs,

what happens if you add same channel from one siti file? like add x channel twice, idk if that can happen or after add its not longer on siti left panel list.

so the right path in my case its choose all-index ones if i wanted detailed info.

anyway im going to try that one you share thanks

question i want look for an online channel whats the one with more of it? from usa im going to check anyway one by one anyway.

edit searching in windows let you know if a name its inside archive

K-vanc commented 1 year ago

Hi, as per xmltv directive, xmltv_id of each channel shall be unique.Tempest is auto checking xmltv_ids and duplicated ones will be highlighted as red and you will not be allowed to save your config without changing duplicated xmltv_id. Rest of channel details as site_id or display_name change be same. To adding same channel from 2 different revision of same siteconfig(such as rev0 is detail-page and rev1 is index-page);

In the new release of Tempest(v1.5.1), there will be search bars for channel selectors so you will type your channel name and all matching ones will be filtered for easy finding

Toei79 commented 1 year ago

idk but tvguide its not working? usa and multi, usually its something that i did wrong, i check permissions but those are fine all

sudo chown -R www-data:www-data /var/www/html sudo chmod -R 777 /var/www/html

i did this again to be sure still

i check passport, zap2it, and others and are fine.

Initializing... Checking required modules...

No Channel Block Found

this file turns 1 kb

[ENC][EX]tvguide.com[Multi]_0.channel.xml

on siti folder its 847 kb i pass to the siti folder used by tempest and turns like 1 kb after you place user key, zip code or country code and generate.

image

i get this when i transfer. using winscp , using cp on command line dont update file. permissions are fine. despite this error file its update i try this to replace 1kb file, also copy paste works.

image also this i have some saved epgs from tvguide .

K-vanc commented 1 year ago

hi @Toei79

Sorry for the late reply. It is a day from hell at work...

As far as I see, website made some changes so I will fix and upload to repo. Till that time, please stop torturing yourself and be patient :)

Toei79 commented 1 year ago

Hi its fine :)

Usually its on my side Thanks!

K-vanc commented 1 year ago

Both tvguide.com[US] and tvguide.com[Multi] updated. You can redownload from repo or just use Siteconfig Updater for your remote device

Toei79 commented 1 year ago

great :) thanks its very handy tvguide. updated looks have more data the multi was 843 now is 955kb yea and now all its working fine both!

its possible add this one? just some channels are no available in other epgs i checked on all if not its okay thanks `

https://www.canela.tv/?section=canalesepg

` looks a little different type epg at least visually

when i try user key with this one epg_ripper_ALL_SOURCES1 for [ENC]epgshare01.online_0.siteconfig.php

get stop like this

image

the other user key for [ENC]epgshare01.online_0.siteconfig.php works fine just this one not work

K-vanc commented 1 year ago

I will have a detailed look if it can be done because my first look is there is nothing looking like a tv schedule.

For your other query, that file is massive size and parsing big xml files are very expensive operation in terms of RAM usage. Also file is gz compressed and it is being uncompressed by Tempest during opeartion which is another RAM hungry thing. I am trying to minimize RAM usage with my regex operations but when file size reach a certain amount, it will reach PHP's memory limit and operation will be terminated by PHP. I can but I didnt set any special memory limit and Tempest using default system value as 128MB.

To make it work on epg_ripper_ALL_SOURCES1, you need to increase your allowed memory limit(something like 256MB) from php.ini and restart your server. Then it will work also but as I said I will not make any change from script side as Tempest's default memory usage is more than enough for %99.99 of cases.

Toei79 commented 1 year ago

understand this case can be a rare exception then, thanks , i guess leave like that maybe tvguide cover it for now.

K-vanc commented 1 year ago

Yes, it is extreamly rare case. Source file is 120Mb xml.gz file. When it is uncompressed, file size will go above 200mb. With default memory limit of PHP, parsing such a file in the way that Tempest uses, is impossible. Better users set their best values on server side if they need more than default values

K-vanc commented 1 year ago

Hi @Toei79 , canela.tv added to USA folder.

Please create new topic for your next requests/issues(if any) because this topic became a mess with multiple subjects and sometimes I cannot find where to reply.

Toei79 commented 1 year ago

Woah amazing! yes i will thank you so much again