tvgrabbers / tvgrabpyAPI

An xmltv-API for extracting and merging tv programme information from several sources
https://github.com/tvgrabbers/tvgrabpyAPI/releases/latest
GNU General Public License v3.0
27 stars 8 forks source link

Never stops running and veronica/disney-xd broken. #17

Closed lapion closed 7 years ago

lapion commented 7 years ago

Sometimes never stops running when run from either mythfilldatabase or update_epg.sh. And if it ever finishes it only fills current day for Veronica/Disney-XD. For a full-run with mythfilldatabase and sometimes update_epg.sh I have had it running for several days without stopping.

hikavdh commented 7 years ago

Can you give more info, like your configuration and a log

lapion commented 7 years ago

http://filebin.ca/3ITjNH8hx6a3/tvgids.xmltv http://filebin.ca/3ITjmsSKjmih/tvgids.xmltv.log.old

hikavdh commented 7 years ago

You should start by using a correct configuration. Or better said using the correct configuration lines. When your version 2 configuration was converted (I guess you did it with an earlier beta version) it had trouble doing so and added the old not converted lines remarked out at the bottom. THEY ARE NOT CONVERTED!!! so do not use those. They are only added in case it contains personal adapted lines, if not delete them and use the correct lines further up. For one thing you will not get data from tvgids.nl through those lines. For Veronica try using "Veronica / Disney XD (experimenteel)" which on a fresh install would be named "Veronica / Disney XD (virtual)" If you do not want to change the xmltvid in MythTV add the following:

[Channel 11-veronicadxd]
xmltvid_alias = 1-veronica

DO NOT TOGETHER WITH THE ALIAS ALSO ENABLE THE "1-veronica" CHANNEL. Or you can expect unexpected results. If you still experience problems, post again a log and configuration, but please trim down the log_level to 7 as the other more detailed logging will very probably have no bearing on any issues at hand. It is mostly only useful for fine-tuning.

lapion commented 7 years ago

Hmm I thought I did a fresh install and not a conversion, apparently something went wrong, I might have had an old configuration file somewhere. Thank you will try to make a fresh config.

lapion commented 7 years ago

Hi, how long after there are no more programs left to process in the queue should the integration by the grabber into mythtv be finished?

hikavdh commented 7 years ago

Not more then minutes after all three detailqueues and the ttvdb queue are empty. It is however possible some request got lost. After 15 or 30 minutes you then will see a message to that effect and the run is finished. So let it run. You can check which detail reports for the channels are still missing in the log. For 11-veronicadxd there should be 3, for veronica, for disney xd and for the combination.

lapion commented 7 years ago

After 2 hours I decided to kill the integration processes and do a manual update_epg.run which has finished most stuff from cache and is now repeating the following block:

"0 left in tvgids.nl queue to process 0 left in primo.eu queue to process 0 left in tvgids.tv queue to process 0 left in theTVdb.com queue to process"

hikavdh commented 7 years ago

Try disabling ttvdb: by either adding --disable-ttvdb on the command-line or adding disable_ttvdb in the general part of your configuration. It has been playing up in the past and I saw yesterday a lot of failures on ttvdb.

hikavdh commented 7 years ago

My guess at present is that soaps in ttvdb are playing up. Those larger ttvdb fetches are pushed to the background as the processing can take more then half an hour, although not two hours, unless you have a very old machine. If disabling ttvdb works... The positive side is that any series is only rechecked ones a month, providing of cause it completed. And only if ttvdb indicates updates, an update is done. If I have time to finish implementing ttvdbv2, the fetches can be made smarter.

lapion commented 7 years ago

For now I think I have to take a break I am getting timeouts >10s I think I have been doing this too often and the server is starting to notice. Even worse I inadvertently deleted my cache files so now I have no fallback for testing. I hope that by tomorrow morning 4:00 I will have cache data again when/if the task completes.

hikavdh commented 7 years ago

Building up the cache takes more then a week as a maximum of three days is actually fetched (today + two days past what is present in the database up to the requested days). Also you should on a new cache file at least once run --configure as extra source data is then stored in the database needed by some sources. The higher time-outs only have to do with it now being prime-time on internet and on the source servers. But you can always increase the time-out value in your configuration-file.

lapion commented 7 years ago

It would seem the veronica problem is solved.

hikavdh commented 7 years ago

I think I now know what happened with ttvdb. Around the time your issue started to play, my colleague added several ttvdb aliasses to for instance catch the Dutch namings for child series. So more then usual first time hits occurred. And since some of those series do have a lot of episodes the combined ttvdb runs took suddenly a lot more time. Ones those are read into the database that extra load diminishes. Only once a month a check on updates will occur. And only if any new episodes have been added a renewed fetch will be done.