K-vanc / Tempest-EPG-Generator

The fastest XMLTV formatted EPG Generator with GUI and the lowest memory usage
https://hub.docker.com/r/kvanc/tempest_epg
Other
125 stars 23 forks source link

Tempest Roadmap #17

Closed K-vanc closed 3 months ago

K-vanc commented 1 year ago

Hi everyone. In last couple of months, I received some really good feature requests/advises from you which some of them was already in my to-do list and others were worth to do it. So I am really glad to see as we are thinking same and expecting same things to make Tempest the best grabber ever which it already is for me :) So, I want to mention about my roadmap for Tempest and you can also comment/add your requests if any;

Delitants commented 1 year ago

Looks promising. But how to automate generation (refresh every n days) via cron after config is created?

K-vanc commented 1 year ago

For scheduled operations, I recommend enabling Fusion mode and setting level to 0 for all from Tempest Configurator. This will review grabbed shows and update changes on repeated operations. Also will restore previously grabbed show data in case on failure on new run. After that cli string and parameters are explained in page 49 of TempestWIKI.pdf. You can decide with options you need as parameter then add command line string directly into cron or prepare a .sh file, insert all commands to sh file and set .sh file to run by cron with your defined time interval. An example from my own sh file;

!/bin/bash

sudo php /var/www/html/tempest.php engine=Generate createxmlgz=on createinvxmlgz=on

ghost commented 1 year ago

Is there any way to automate epg generation and upload that's file in dropbox (any other platform) in Windows? I'm currently running this in Xampp with CLI Mode cd c:/xampp/htdocs php tempest.php engine=Generate tempconfig=dbmtech.config.xml createinvxmlgz=on

how to automate this command everyday in Windows 11?

K-vanc commented 1 year ago

Create a .bat file such as epg.bat on your desktop, add your 2 lines into file and save. Now, press windows key and search for "Task Scheduler" in your pc. set the date/time options to run epg.bat based on your needs. For more info, you can google windows scheduler. Below example content not belongs to me;

https://www.thewindowsclub.com/how-to-schedule-batch-file-run-automatically-windows-7

For how to upload any drive etc. You may follow similar way since it is not my business. I am using rclone on my rpi to upload google drive but this part is out of my and Tempest's scope.

ghost commented 1 year ago

Ok! Thanks, I will try it.

Delitants commented 1 year ago

And how do I suppose to know which XML channel is in? Why there is no basic search by channel name?

K-vanc commented 1 year ago

And how do I suppose to know which XML channel is in? Why there is no basic search by channel name?

What this means by which xml channel in? What this means by basic search by channel name?

Delitants commented 1 year ago

And how do I suppose to know which XML channel is in? Why there is no basic search by channel name?

What this means by which xml channel in? What this means by basic search by channel name?

Like, I open configurator, I can select xml source, and it lists all the channels in it. I don’t initially know which source contains my channel. I have to either do an ugly recursive grep across all site configs or click each one one by one in GUI and perform a search on the page by browser. It’s just time consuming. Here is an example https://iptv-org.github.io/ searching channels and seeing the source for it. In your case I need to select source first and guess where the channel possibly can be.

K-vanc commented 1 year ago

Do you have any idea about how your example working? it is website hosted on a remote server which reading results from database files(there are ready several json files) with several javascript, jquery and maybe SQL database requests. Tempest is a single php script running on your local server which using basic html and very little javascript features. Configurator developed to minimize copy/paste and typing mistakes which causing syntax errors when directly working on configuration file and it may help people to save multiple selections safely. It supports basic alphanumeric search by pressing keyboard as limited and allowed by html. Same basic alphanumeric search also applicable for siteconfig names. For much more better search, you can use traditional NON-UGLY direct copy-paste method from xml to xml via any text editor or you can use some other NON-UGLY cli solutions as you got used to use(I dont know how you were handling this on them) but I cannot guess which channel you need and which siteconfig has it. Tempest is an epg generator, not a ready-epg hosting service

Anyway, i will have a look if i can add a quick search bar inside select box without breaking or recoding all structure

Toei79 commented 1 year ago

I use rcclone too, honestly i just share with my family not about use for bussiness. my sister , my brother and my mother.

so far no good you need use dropbox premium.

im been using google drive but its not working. even for a playlist its get issues.

sometimes depends the site it gets the epg automatically without need to do manually, its just one time thing and its good to go. the plus all in one place.

K-vanc commented 1 year ago

Hi @Toei79 . I cannot help on other software issues but probably it is related with your setup. I am using rclone as part of my Tempest run .sh file in crontab for years and it is synchronizing generated xml file with my google drive folder for more than 2 years without any touch. It shall be uploaded with "sync" command otherwise download link will change on each update. After uploading, the file shall be enabled for sharing in google drive settings so my friends are just using static google drive download link that I gave them, by pasting it on their iptv player's epg location path. Just keep in your mind that there is a daily limit of sharing file then your link will be temporarily blocked for downloading. I didnt use any other cloud setup rclone so cannot say anything about dropbox.

Toei79 commented 1 year ago

I was using dropbox it was good

i have my playlist there and gets blocked that can be issue.

I use tempest but i mix with other epgs that i did for radios kinda dummy thats fine for some reason no matter i change it don't detected some on the playlist and on the epg , im sure because locally its fine, but maybe can be google limit daily i never know about that , i only use myself at moment

K-vanc commented 1 year ago

Dear People of Tempest,

I need some suggestion regarding a possible feature. I saw some people using Tempest with port-forwarding or from hosting services so it is accessible from beyond firewall but that caused some security concerns (and also curiosity about how to do) on me then I wrote some block of code for additional login layer. It will shown as below;

image

Idea is, the users who needs login, will search in tempest.php file and for;

$username = ''; $password = '';

strings and hard code their credentials it into file such as;

$username = 'test'; $password = 'login123';

After saving tempest.php and reloading page, login layer will be activated. After clicking Logout, you will be back to login layer.

image

If you change hard coded username/password strings back into empty state, Tempest will deactivate login layer. Of course it is not a bullet-proof solution like a real website login but still will be an extra precaution with file renaming.

Anyway, I am not so sure about including that feature on my new release so wanted to get your opinions

ghost commented 1 year ago

will it effect automatic scheduled epg generation?

Toei79 commented 1 year ago

nice thing, in my case its no necessary , i mean i share with my family barely knows about handle this stuff , myself take me like 6 months to pull the trigger, yea its not hard just im been like a year in linux stuff.

im been doing google drive but i dont get fully right at all honestly.

vpn tunnel? windergard i guess its called, thats safest? im been using asus feature instant guard on their routers, just start to check if i can install in other devices so it be useful. that can be safe? vpn tunnel?

K-vanc commented 1 year ago

will it effect automatic scheduled epg generation?

It will not effect CLI mode regardless of it is enabled or not. It will appear only on GUI mode(browser) requests. For users on localhost, it is completely useless. It may be useful only for port-forwarded or webhost users. These users may run Tempest on CLI mode via SSH connection and if you have SSH credentials, you can do anything you want so login layer will not have any purpose on CLI mode.

K-vanc commented 1 year ago

nice thing, in my case its no necessary , i mean i share with my family barely knows about handle this stuff , myself take me like 6 months to pull the trigger, yea its not hard just im been like a year in linux stuff.

im been doing google drive but i dont get fully right at all honestly.

vpn tunnel? windergard i guess its called, thats safest? im been using asus feature instant guard on their routers, just start to check if i can install in other devices so it be useful. that can be safe? vpn tunnel?

I think you mean Wireguard? It is similar like openvpn but using a new encryption method. As far as I know, it can be installed on many operating systems but I have no idea about security etc.

Toei79 commented 1 year ago

yea that one, looks easier like other thing that you can install on your server or linux distro.

im think im going to take that path , its easier just install on remote device and mine, then they can use epg that i have here because its local for them.

asus have one but only supports cell phones and tablets. no android tv. the one from asus its easier one. no complex setup.

i did openvpn one time ago but barely understanding now i get a better logic of this , its a text file with some info there and here. wireguard its better for what i read.

K-vanc commented 1 year ago

Hi, actually it is not since all you need to do to set up a php server. I need some details such as;

What is your OS (Windows/Linux)? Do you have a php server? Do you know how to run a php file?

Please send your replies to my mail ( tempestepg@gmail.com ). I will reply you from there in Turkish since I am keeping this thread only for informative purpose about features/suggestions

K-vanc commented 1 year ago

New release coming soon, with self-update module..

Screenshot_20230927-150628_Chrome.jpg

Screenshot_20230927-150636_Chrome.jpg

ghost commented 1 year ago

Yeah it will be good. Appreciation for your efforts.. next waiting for docker release!

K-vanc commented 1 year ago

This actually will be a conceptual milestone for docker. I still not yet started to work on docker but reading when I have time. If my understanding is correct, they are changing the container for updates or you have to manually close the container and replace etc which is not so user-friendly. This will solve the dealing with containers problem. Otherwise, you can do the some by downloading and copy/pasting as you do now

Toei79 commented 1 year ago

nice easier to update , never figure out docker i have a couple on my nas , i feel it complicated.

K-vanc commented 1 year ago

Ok guys, as promised some lonnnng time ago, managed to build and run first Tempest docker container yesterday and grabbed epg/created channel list on GUI and CLI modes. It will be on alpine linux build for minimum container size and with nginx + php8.2-fpm. Current container size is 100mb but it may be a little more or less depends on my final checks. Currently everything is running but for better multitasking, I need to tweak couple of things more on nginx and php since I notice some slow performance on detail paged siteconfigs compared with bare php performance.

In order to complete docker integration, there will be an Tempest update soon. With this update, Tempest will behave more like an application rather than script and as result generated epg and log files will be moved into tempest_config folder with their own folders as "epg" and "log".

image

This change is required for a smooth docker integration and users will be able access everything needed by just mounting tempest_config folder to any folder on their local host. With support of previous release for self-update, there will be no need to copy/paste tempest.php or rebuilding container.

Since i just learned basics of docker, i want to test more and be sure about everything before releasing it but this is just a advance notice for people who are waiting for docker that there is light now at the end of tunnel and notice for normal users regarding folder structure change on next update.

ghost commented 1 year ago

Great News 👍

K-vanc commented 1 year ago

first photo of application;

image

Around 90mb container memory usage

naijaping commented 1 year ago

Great Job Bro

K-vanc commented 1 year ago

And finally, docker release is available on;

https://hub.docker.com/r/kvanc/tempest_epg

Toei79 commented 1 year ago

im little ignorant but those docker its for advanced users or for more limited ones? its better docker? many people prefer it.

i have docker on my synology nas it open options to apps that you cant usually add there.

K-vanc commented 1 year ago

Docker images comes with mostly everything required configured so it is easy for many people since you dont have to install and set several things. It is also good for some systems that has no native support for required languages or software. As example there are several pvr softwares that are not supporting php or webserver install but support docker so you can install and run Tempest image on them through docker only.

Docker usage seemed to me pretty easy through docker desktop software but if you are running it on cli, it is very complicated with tons of commands, flags, parameters etc and some of them already outdated and deprecated usage but still available on the internet as example/guide so as a new user, you are just wasting your time till you notice that command is not working anymore. This situation is valid even for docker's official page examples..

For performance, bare php is almost always better from docker port as it is understandable that containerizing will cause slight performance drop but it is still usuable for the reasons i mentioned or just for quick testing to see how it is working instead of installing/configuring several things

K-vanc commented 1 year ago

Hi, Dear Tempest Users;

For the last couple of days, I am checking some relatively new developed/started epg tools/websites as epg provider etc. on the internet and unfortunately, found/confirmed that some of these people are using Tempest and resharing or trying to selling generated epg after editing generator name, information, website links with details' of their fake apps so it looks like generated by them..

This is actually stealing something free and really hard for me to understand why people are doing this. Just for making a few dollars? Or for showing people that they are very good coders/developers? Combination of those or completely different reason? I always said I will keep Tempest free-of-use and have a very straight license that more or less you can do whatever you want with it as long as give proper reference/credit to Tempest. License says non-commercial but I also dont care about it as people can purchase whatever they want from wherever they want as long as they know where it is coming from. There are tons of people that dont want to touch anything and just having epg by only copy/pasting an url. So I would have no objection if they didnt steal my reputation since I am spending my time to help you guys by excepting nothing else than reputation..

In order to stop this abusement and robbery, I will take some actions in the following days. Actually I can cut it off anytime I want but I dont want to effect any of regular users so I am trying to manage my anger(yesterday evening, I ran for 7kms as a precaution) and thinking for a better solution including things that I hate such as user detection systems, Ip/mac recognations and/or more. Even I can move repo to some user based access system..

So, I am sorry from now on about what may come but I promise that I will look for the best and most harmless solution for you, guys..

naijaping commented 1 year ago

Please dont cut it off Bro, it will affect us regular users

Toei79 commented 1 year ago

imho probably you already know that maybe can happen? you worked for the other big epg tool if im no wrong (i guess i read on tempest files or here). eventually your tool make famous and people take advantage in the wrong way, thats expected , but yea i understand your upset . only can say decide whats best for you, i mean i see how hard its keep update those sites, maybe easy but need keep checking those, and its a very versatile epg tool . anyway the path you decide, thanks :).

K-vanc commented 10 months ago

Hi guys;

I am planing to make some cleaning/maintenance in repo since I have noticed some siteconfigs are not working or some websites are not running/active anymore etc. I hope I can finish it within a week(or less) then will release new Tempest version 1.6.1 with some new features and improvements.

maintenance

K-vanc commented 9 months ago

Well, Gents. Finally repo maintenance is over... It took much more time than I estimated since I found many not working or partially working siteconfigs but also gave me good opportunity to identify and patch several bugs in Tempest core modules. Within day, I will release new version 1.6.1.0

tamkov commented 9 months ago

Thank you very much for your work.

If you have a lot of time, would you try to make a site config for port.hu? I tried, but it's too big mission for me, right at the beginning I got stuck that the url needs a channel name and a channel number too :(

K-vanc commented 9 months ago

Hi @tamkov

I will have a look on that. I am out of pc now but as far as i see from my mobile, it can be done. Also for siteconfig requests, please open a new issue with request label. I am keeping this issue open for general announcements and feature discussion.

K-vanc commented 9 months ago

Dear Tempest Users;

Regarding to issued critical patch 1.6.1.1, it fixed the problem of missing xmltv elements in some cases as expected but my investigation shows that with a rare (but possible) combination of php+siteconfig+some php libraries(+correct temperature/humidty..), you may get some not working siteconfigs. Probably this will never happen for %95 of you but in order to stay safe side and fix the case permanently for the remaining unlucky %5 of users, I decided to patch all [ENC] and [ENC][EX] siteconfigs for the last time. Since it will be only a patch-reupload process, I am planning to finish all within 1-2 days. So please keep your siteconfigs updated, Here we go..

K-vanc commented 9 months ago

Patching completed

fankloano commented 9 months ago

Hey K-vanc,

hope you are allright? Despite everything is working as it should, i would have a suggestion. Additionaly of making a .gz file after grabbing the epg-data, would it be possibile to add the possibility to create a .xz file immediately after the grabbing? I don't know what's the effort of such an implementation, alternatively I can still do it manually.

thanks and a good day

K-vanc commented 9 months ago

Hi @fankloano

For xz compression or lzma algo, there is no built-in php function and/or extension. This is mean some unofficial and partially working third-party libraries have to be used for adding such feature. As principle, i am always using only officially php supported libraries and extensions in my codes and solutions because unofficial extensions have great potential for messing up things with unpredictable errors, not only during usage but even during installations. So in the future, if php releases an official solution similar to .gz compressions, i would like to add support also for xz but currently i have to skip it.

ghost commented 8 months ago

Hi K-vanc

I have tested it in GitHub Code space, running successfully, but don't know how to setup cronjob. I have to run code space manually every time to grab epg.

K-vanc commented 8 months ago

hi @simperpie

I dont have any idea about Github Code space and how it is working. There are several examples of cron operations in other closed Issues and TempestWIKI.pdf as command line parameters but it could be better to google out how you can apply those command line parameters in Github Code space

Edit;

Seems like you can run commands via "postCreateCommand" option in Github Codespace also in examples, they are mentioning a shell input but I dont know how/where you can call it. Both of those will support regular linux running command as;

sudo php /var/www/html/tempest.php engine=Generate tempconfig=yourconfigname.config.xml

But I didnt see anything about scheduled operations. Probably there should be an option in one of their config files

Edit2:

This shell part is supporting apt-get update and install commands(seems you have connect via SSH) so probably you can install crontab (or cron/dont know exact name there) or maybe it is already installed so you can just configure it based on above command example

ghost commented 8 months ago

hi @simperpie

I dont have any idea about Github Code space and how it is working. There are several examples of cron operations in other closed Issues and TempestWIKI.pdf as command line parameters but it could be better to google out how you can apply those command line parameters in Github Code space

Edit;

Seems like you can run commands via "postCreateCommand" option in Github Codespace also in examples, they are mentioning a shell input but I dont know how/where you can call it. Both of those will support regular linux running command as;

sudo php /var/www/html/tempest.php engine=Generate tempconfig=yourconfigname.config.xml

But I didnt see anything about scheduled operations. Probably there should be an option in one of their config files

Edit2:

This shell part is supporting apt-get update and install commands(seems you have connect via SSH) so probably you can install crontab (or cron/dont know exact name there) or maybe it is already installed so you can just configure it based on above command example

Thanks for response. I tried this, but not succeed.

K-vanc commented 8 months ago

well as i said, I dont know how it is working. As far as I know GH actions can be scheduled so probably it should be available also for GH Codespace. Maybe you can directly ask GH Support Team if it is possible or not

naijaping commented 8 months ago

HI, Has anybody used the inbuilt tempest updater successfully? I have tried several times always return :

Your Tempest version is rv1.6.2.0-2402260957 Checking for updates... An error occurred while connecting to update server... Please check your internet connection and try again later...

K-vanc commented 8 months ago

Hi @naijaping

I just tried a local release copy of rv1.6.2.0-2402260957 to update and successfully updated into rv1.6.2.1-2403010828

The error message you got appears when you cannot reach version file hosted on github. It may be a temporary issue or may be related with your network/dns etc.

Also for such questions, please open a new issue since this thread for announcements

K-vanc commented 8 months ago

Guys, don't want to promise since it is very difficult task to-do in cross-platform (which attracts me as a challenge..) but what do you think about having tempest-cron managed from Tempest and independent from system cron/task scheduler?

Do you need such thing? In all unix os, cron or cronjob can be installed while same thing handled by Task Scheduler in Windows. Docker also can use both of them based on OS that it is working on. For android, I dont think it may work without root privileges so probably will be a waste of time.. Am I missing some other OS/Platform?

Toei79 commented 8 months ago

i dont need it, but make program more independent and complete, imo

K-vanc commented 8 months ago

i dont need it, but make program more independent and complete, imo

Thanks but I gave it up for now. I managed to run test scripts both on Windows and Linux in defined intervals but it will never be like a real cron without OS integration. It is just php so if you restart your pc or server etc., it will stop working till you restart and rerun all so this is basicly against the nature of cron..

But some methods I found by testing it, gave me some ideas about new features. lets see what will happen on next releases