search

thetvdb / metadata.tvshows.thetvdb.com.v4.python

TheTVDB Official Kodi TV plugin
9 stars 3 forks source link

The scanning for TV shows never stops #7

Closed Hawkedon closed 2 years ago

Hawkedon commented 2 years ago

When I choosing to scan the new content for TV shows of my video library, even though there is no change to my library, the TVDB scanner never stops, it always shows "scanning for new content XXX%", when the XXX get to 100 (it takes several minutes for this), it resets to 33 and starts again. I waited for half an hour, and it still shows the progress there. I never encountered this issue from other scrapers. I switched to the old TVDB scraper, and the scraping process stopped very quickly.

Edit: I just checked the kodi log, and found the scraper did not only scan the new items, but also scan all existing items again. This may explain why it took so long, because my TV shows library is very big.

KarellenX commented 2 years ago

I just checked the kodi log, and found the scraper did not only scan the new items, but also scan all existing items again

Yes, that is meant to happen. When you run an Update Library, Kodi has to scan your sources to locate the new content. That means it is scanning the folders already in your library looking for new episodes as well as looking for entirely new folders/tv shows.

How many sources do you have and how big is your library? There are some little tricks you can use to reduce the repetitive scanning,

Hawkedon commented 2 years ago

Hi @KarellenX, I'm aware of what you're saying (Kodi will scan all my video sources), but that's not what's supposed to happen (scrape all existing items when they are already in Kodi database).

For example, I just added a new episode to a TV show, and I click the "scan for new content" context menu for the video source from Kodi Settings/Media/Videos. I did this tons of times before. With other scrapers, I can see Kodi will scan all sources, but it only scrapes the newly added episode, not all existing ones, the whole process is very fast (as most work is done locally to query local Kodi database, and only the newly added episode will be queried on tvdb.com site).

But for this TVDB scraper, from the log I can see it sends information about existing episodes (which are already in Kodi database) to the TVDB server (scraping), as the existing TV shows are a lot, so the whole process is very slow.

I'm not sure why this happening or is this related with Kodi or the scraper. But maybe it's related with the situation that I switched scraper recently (I used TMDB TV Shows scraper as it's default for Matrix, but it doesn't work for TV shows specials, so I switched to the new TVDB V4 scraper). I just removed the TV shows video sources and added them again and scraping them again. I'll update later when the scraping finished to see if it works in the expected way.

KarellenX commented 2 years ago

from the log I can see it sends information about existing episodes (which are already in Kodi database) to the TVDB server (scraping),

I need a debug log that shows that. Also give the line numbers of where you see that. In my extensive testing of this scraper, I did not notice this behaviour, and am wondering if you are misinterpreting the log?

Without the debug log we can provide no further help.

Hawkedon commented 2 years ago

Thanks @KarellenX. I figured out the problem after reading your "misinterpreting the log". :) It's not the fault of TVDB V4 scraper, but the TMDB TV Shows scraper. Many episodes and some whole TV shows (Altered Carbon (2018) for example) are totally missed from TMDB TV Shows scraper, they were never scraped as they are not in TMDB site and I never noticed that. Thus there are so many things (TV sows and episodes) are new to TVDB V4 scraper and thus a very long scraping time.

I checked the following log snippet and realized the problem:

2022-05-22 18:39:41.222 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: logging in 2022-05-22 18:39:41.313 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: THE TVDB TV SHOWS SCRAPER V.4 2022-05-22 18:39:41.313 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: {'action': 'getartwork', 'id': '73141', 'pathSettings': '{"gender":"Other","language":"English","pin":"","season_type":"1","year":1900}'} 2022-05-22 18:39:41.316 T:9076 DEBUG : requested setting (uuid) was not found. 2022-05-22 18:39:41.319 T:9076 DEBUG : Skipped 1 duplicate messages.. 2022-05-22 18:39:41.319 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: settings: 2022-05-22 18:39:41.319 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: {'gender': 'Other', 'language': 'English', 'pin': '', 'season_type': '1', 'year': 1900, 'uuid': 'e43a90a5-d001-4000-8000-000000000000'} 2022-05-22 18:39:41.319 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: about to call get artworks 2022-05-22 18:39:41.319 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: body in auth call 2022-05-22 18:39:41.319 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: {'apikey': '', 'gender': 'Other', 'birthYear': 1900, 'uuid': 'e43a90a5-d001-4000-8000-000000000000'} 2022-05-22 18:39:41.436 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: about to make request to url https://api4.thetvdb.com/v4/series/73141/extended 2022-05-22 18:39:41.436 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: https://api4.thetvdb.com/v4/series/73141/extended 2022-05-22 18:39:41.592 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: about to make request to url https://api4.thetvdb.com/v4/series/73141/translations/eng 2022-05-22 18:39:41.592 T:9076 DEBUG : [metadata.tvshows.thetvdb.com.v4.python (1.1.0)]: https://api4.thetvdb.com/v4/series/73141/translations/eng 2022-05-22 18:39:41.680 T:9076 WARNING : Attempt to use invalid handle 379 2022-05-22 18:39:41.680 T:5128 ERROR : XFILE::CDirectory::GetDirectory - Error getting 2022-05-22 18:39:41.680 T:9076 INFO : CPythonInvoker(411, Q:\Users\UserMgr0\AppData\Local\Packages\XBMCFoundation.Kodi_4n2hpmxwrvr6p\LocalState\addons\metadata.tvshows.thetvdb.com.v4.python\thetvdb.py): script successfully run 2022-05-22 18:39:41.681 T:9076 DEBUG : CPythonInvoker::onExecutionDone(411, Q:\Users\UserMgr0\AppData\Local\Packages\XBMCFoundation.Kodi_4n2hpmxwrvr6p\LocalState\addons\metadata.tvshows.thetvdb.com.v4.python\thetvdb.py) 2022-05-22 18:39:41.728 T:5128 DEBUG : VideoInfoScanner: Skipping dir 'smb://Media/TVShows/American Crime Story (2016)/' due to no change 2022-05-22 18:39:41.796 T:5128 DEBUG : VideoInfoScanner: Scanning dir 'smb://Media/TVShows/Altered Carbon (2018)/' as not in the database

KarellenX commented 2 years ago

Many episodes and some whole TV shows (Altered Carbon (2018) for example) are totally missed from TMDB TV Shows scraper, they were never scraped as they are not in TMDB site and I never noticed that

Just to clarify, Altered Carbon is at TMDB... https://www.themoviedb.org/tv/68421-altered-carbon I guess it did not scrape for some other reason. What that reason is, I don't know.

But glad you figured it out and it is working correctly now.