Tempest Roadmap - Githubissues

K-vanc commented 1 year ago

Hi everyone. In last couple of months, I received some really good feature requests/advises from you which some of them was already in my to-do list and others were worth to do it. So I am really glad to see as we are thinking same and expecting same things to make Tempest the best grabber ever which it already is for me :) So, I want to mention about my roadmap for Tempest and you can also comment/add your requests if any;

User Logo Option (Done as of rv1.4.3) I am working on a new command for some time to let users add their own logo urls for each channel. For unlocked siteconfigs, this can be already done but for encrypted siteconfigs it was not possible. With this option, you can add or change logo url for each channel on all siteconfigs
How to Use Wikipages With latest patch, all siteconfigs are working flawlessly even in android servers. To take advantage of this, I will prepare Wikipages with screenshots for setup and usage on Android(KSWeb) , Windows(Xammp) and Linux(probably Nginx). This guides will help much for unexperinced php users.
CLI Channel Creation (Done as of rv1.4.3) ~~Tempest currently can make folder structure and can grab in CLI mode without any issue but for channel creation GUI is needed. I will code it allow channel creation also in CLI mode.~~
Siteconfig Updater (Done as of rv1.5.0) I am planning to add a small script to download all siteconfigs(as updated) in zip. I started to use github with Tempest and was thinking when I download a release version of project as zip, i will get everything as latest, but I found out(in a hard way) that i am downloading project as the time of release.
Docker Release (Done as of rv1.6.0.0) ~~I believe this is the correct time for creating an official server + Tempest container after completing above steps~~
Some improvements on Debug into File (Done as of rv1.5.3) ~~This is just for torturing myself~~
Append/Overwrite option for page grabbers (Cancelled since I am feeling too lazy for it) ~~This is also for making my life easier during siteconfig making.~~

Toei79 commented 5 months ago

idk if this its possible. on tempest configurator , after choose cyclone_site , and you need choose cyclone id, can be complex to navigate and choose the right channel, if you choose a long long cyclone site, that can be easier like a box like the one used on channel configuration

K-vanc commented 5 months ago

it is possible but I prefer not to do as I did. Channel indexer tab is the highest memory using part of Tempest. All that boxes you see for each channel is actually a table drawed by your browser when you click tab and there is no way to do it with low memory. Each configuration file supports upto 5000 channels. Actually there is no limit on that but more channel causing all browsers to go out of memory issue(I am speaking about >1GB ram usage only for browser). More input on that page causing more memory usage and result slow response times or termination. I will have a look but for sure drawing another box for several hundred(or thousand) channels will not go good

Toei79 commented 5 months ago

understand thanks :)

memory thing its one of my biggest headaches with browsers, right now i have 9 tabs open and getting slow.

ghost commented 5 months ago

well as i said, I dont know how it is working. As far as I know GH actions can be scheduled so probably it should be available also for GH Codespace. Maybe you can directly ask GH Support Team if it is possible or not

Thanks! Finally I have successfully apply cronjob through GitHub action to run tempest in GitHub code space. Now no need to install php server. GitHub Code space does it automatically.

K-vanc commented 5 months ago

good to hear,

Note: I checked your repo and you are using wrong siteconfig for jio

ghost commented 5 months ago

I know I'm using old one, Its working for me.. you updated jio config which have sadly one day epg only

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

From: Kivanc @.> Sent: Tuesday, March 19, 2024 2:20:06 PM To: K-vanc/Tempest-EPG-Generator @.> Cc: Simper Pie @.>; Mention @.> Subject: Re: [K-vanc/Tempest-EPG-Generator] Tempest Roadmap (Issue #17)

good to hear,

Note: I checked your repo and you are using wrong siteconfig for jio

— Reply to this email directly, view it on GitHubhttps://github.com/K-vanc/Tempest-EPG-Generator/issues/17#issuecomment-2006355191, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVXCSYYWTRYDCE4IBHJQ4ALYY7335AVCNFSM6AAAAAAUVRUM2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBWGM2TKMJZGE. You are receiving this because you were mentioned.Message ID: @.***>

fankloano commented 5 months ago

Hello, I have a question (or suggestion) regarding the use of own logos. When Tempest makes use of the cyclone mode how can I set it up the way that also the cyclone channel uses the logo that I added for the main channel? For example I have Channel A, with site_id: 0000||mylogourl when i choose a cyclone channel, the cyclone_id uses the logo added in the siteconfig, I can't add ||mylogourl there. Is there something I am missing or isn't this possible yet?

K-vanc commented 5 months ago

Hi Alex,

Since cyclone id is a select box in order to eleminate mistakes/typos, it is not possible to edit from GUI and I have no plan to add another option there to manual editing but you can still set it by old school methods. First, I will explain how Tempest works for logo.

If both main channel and cyclone channel have logo and main channel fails, Tempest uses cyclone logo
If only main channel has logo and fails, Tempest grab data of cyclone channel but preserve logo of main channel
Cyclone is a normal channel grabbing but maintaining original channel xmltv_id so all commands will continue to work with it.

So in order to achieve what you want;

You may manually edit site_id of channel that you will pick as cyclone, in channel.xml file with example ||#logo#your_logo_url.png command
You may manually do the same above in related cyclone_id of your configuration file
You may manually delete ||logo_part of channel to be picked as cyclone in related channel.xml file so Tempest will use main channel's logo in case of failure
and all similar methods will work

K-vanc commented 5 months ago

@fankloano

I am forgetting all the options that i made.. i've just checked and if you set your logo by ||#logo# command, it will preserve only the url set by command so no need to edit anything

fankloano commented 5 months ago

@K-vanc Yes, there are quite a few.... :-)

So in my config, instead of:

ChannelName

I set:

ChannelName

Is this correct?

K-vanc commented 5 months ago

@K-vanc Yes, there are quite a few.... :-)

So in my config, instead of:
ChannelName
I set:
ChannelName
Is this correct?

If this is your url for logo, yes, it will be set to the end of site_id. I am specificly telling url because logo command will not use anything related with logos from siteconfigs so it must be full url of logo. You can make it inside Configurator easily

fankloano commented 5 months ago

@K-vanc Just tested and works :-) thank you

K-vanc commented 4 months ago

idk if this its possible. on tempest configurator , after choose cyclone_site , and you need choose cyclone id, can be complex to navigate and choose the right channel, if you choose a long long cyclone site, that can be easier like a box like the one used on channel configuration

this will be available as of version 1.7

K-vanc commented 4 months ago

Dear Tempest Users;

here is a spoiler alert for next version update

Toei79 commented 4 months ago

thats pretty cool feature :)

K-vanc commented 3 months ago

Dear Tempest Users;

I am planning to release new version 1.7.0.0 within next week so this is an important notice for all of your awareness in advance about some critical changes that may effect your setup. I decided to change all CLI command parameters because they are so ugly and unnecessarily long.. To be honest, that commands were from the days that I run Tempest for myself only via curl request(with -F flags) without any setup for cli then I published repo suddenly and couldn't find a chance to modernize them.. Anyway, I think most of you will like new commands so examples for EPG grabbing;

Old one: php /your/tempest.php engine=Generate tempconfig=test.config.xml createxmlgz=on createinvxmlgz=on

WTF!, isn't it so ugly and long?

New one: php /your/tempest.php --epg config=test gz invgz

and for channel creation;

Old one: php /your/tempest.php chengine=Generate siteconfig=test_0 userkey=asdf rm_dupe_id=on rm_dupe_name=on name_modifier=||#addstart#blabla

New one: php /your/tempest.php --channel config=test_0 ukey=keyvalue dpid dpnm "nmod=||#addstart#blabla"

This new command style will match with existing Tempest update command;

php /your/tempest.php --update

and also will be match with newly introduced xml merger;

php /your/tempest.php --xmlmerge xml=file1 xml=file2 xml=fileN out=merged gz

So I hope you will like new version

K-vanc commented 3 months ago

Yes, Gents;

Finally rv1.7.0.0 has been released. I am waiting for your feedbacks especially for the new XML Merger module

ghost commented 3 months ago

Just tested XML Merger module, working good and fast. Just for curiosity I tried merging file like php tempest.php --xmlmerge xml=tempest xml=http://www.test/epg.xml out=merged gz. Here my second file is web address, can we add downloading feature in it. Like first xml merger will download second file from web or any other sources (not from tempest) and then merger will be worked.

K-vanc commented 3 months ago

hi, xml parameter searches given filename inside /epg/ folder. It is not designed for remote files. Xml merging is actually slow and memory hunger job especially when you have big files. The method I found is fast and low memory solution with local files only. I can merge two >50mb xml files less then half minute with default memory setting. This job takes 13minutes with traditional xml processing methods when you add duplicate xmltv_id into account. The trick here is not reading file so no matter how big it is. Also there is another issue with security. Some php solutions such as url_fopen is disabled by default for %99 of the servers/systems due to security. This cannot be enabled from script side. Normal download methods will read file into memory and allowing such thing will create a security hole for downloading malicious data into your system. My personal opinion this is not a good idea for many reasons but I will add it to my "MAYBE-TO-DO" list. I think you can do the same from commandline by adding 1 wget/curl line to download file into your /epg/ folder before --xmlmerge line so the risk of installing virus into your system will be your responsibility, not mine 😄

ghost commented 3 months ago

hi, xml parameter searches given filename inside /epg/ folder. It is not designed for remote files. Xml merging is actually slow and memory hunger job especially when you have big files. The method I found is fast and low memory solution with local files only. I can merge two >50mb xml files less then half minute with default memory setting. This job takes 13minutes with traditional xml processing methods when you add duplicate xmltv_id into account. The trick here is not reading file so no matter how big it is. Also there is another issue with security. Some php solutions such as url_fopen is disabled by default for %99 of the servers/systems due to security. This cannot be enabled from script side. Normal download methods will read file into memory and allowing such thing will create a security hole for downloading malicious data into your system. My personal opinion this is not a good idea for many reasons but I will add it to my "MAYBE-TO-DO" list. I think you can do the same from commandline by adding 1 wget/curl line to download file into your /epg/ folder before --xmlmerge line so the risk of installing virus into your system will be your responsibility, not mine 😄

Yeah! It is really fast and consume low memory rather than traditional one.

Toei79 commented 3 months ago

no yet installed its been running smooth so far , gonna try thanks

Toei79 commented 3 months ago

installed see how goes, old line no work , new one yes replacing on crontab.

SpitFireRSA commented 3 months ago

Dear Tempest Users;

I am planning to release new version 1.7.0.0 within next week so this is an important notice for all of your awareness in advance about some critical changes that may effect your setup. I decided to change all CLI command parameters because they are so ugly and unnecessarily long.. To be honest, that commands were from the days that I run Tempest for myself only via curl request(with -F flags) without any setup for cli then I published repo suddenly and couldn't find a chance to modernize them.. Anyway, I think most of you will like new commands so examples for EPG grabbing;

Old one: php /your/tempest.php engine=Generate tempconfig=test.config.xml createxmlgz=on createinvxmlgz=on

WTF!, isn't it so ugly and long?

New one: php /your/tempest.php --epg config=test gz invgz

and for channel creation;

Old one: php /your/tempest.php chengine=Generate siteconfig=test_0 userkey=asdf rm_dupe_id=on rm_dupe_name=on name_modifier=||#addstart#blabla

New one: php /your/tempest.php --channel config=test_0 ukey=keyvalue dpid dpnm "nmod=||#addstart#blabla"

This new command style will match with existing Tempest update command;

php /your/tempest.php --update

and also will be match with newly introduced xml merger;

php /your/tempest.php --xmlmerge xml=file1 xml=file2 xml=fileN out=merged gz

So I hope you will like new version

Thank you for this reference! Managed to get the EPG updated on a schedule.

Will you be updating the docker to image rv1.7 soon? This is what is currently running: <div id="versx">rv1.6.0.0-2310141014</div>

K-vanc commented 3 months ago

Hi, you can update Tempest without container rebuilt. For cli, use below line;

docker exec -u root yourcontainername php /var/www/html/tempest.php --update

And for Gui, just use below option from your browser;

SpitFireRSA commented 3 months ago

Thanks! I obviously missed this (first coffee of the morning). Thank you for a great tool 👍

K-vanc commented 1 month ago

Dear Tempest Users;

After some passed time without receiving any update, I would like to mention that work is on-progress for the final(planned) evolvement of Tempest as V2.0.0.0 with the last major module add-on.. I will not tell the details but this last module will turn Tempest into an unrivalled grabbing monster(Imagine what is coming since Tempest is already unrivalled as its current form😄). Maybe some of you may know meaning of below diagram;

But, the first and more important part is [ENC][EX] siteconfigs will expire on 03.08.2024 so I will start updating repo for all [ENC][EX] files in the following days since expiry date is less than a month..

Anyway guys, just stay tuned for the coming updates 😄

Toei79 commented 1 month ago

looks very intriguing and crazy for sure! plus insane.

K-vanc commented 1 month ago

Expiry dates of all [ENC][EX] siteconfigs are extended for 1 year. Please do not forget to update your local copies before 03.08.2024 which is the last day of old copies'

K-vanc commented 1 month ago

Gents;

It is time to try Tempest's new particle accelerator module "Synchrotron" on rv2.0.0.0. It reduced my daily 315 channels/57 siteconfigs grabbing time from 20 mins to 8-9 mins with its default settings.

K-vanc commented 1 week ago

Hi gents;

This thread is probably reached the end of its life. There are currently too many topics inside and it started to be difficult to find what you are looking so I am closing this issue. New announcements/discussions will be published under "Discussions" section as it should be since the beginning, sorry, this was my first project in Github..

Anyway guys, thanks all of you for your participation here. Your thoughts and suggestions were great and very useful for improving Tempest if you consider several features added after your suggestions here..

K-vanc / Tempest-EPG-Generator

Tempest Roadmap #17