Flexget / Flexget

The official FlexGet repository
http://www.flexget.com
MIT License
1.72k stars 471 forks source link

Unhandled exception during task queue run loop in letterboxd plugin #2790

Closed effemmeffe closed 2 years ago

effemmeffe commented 3 years ago

Expected behaviour:

Get movies from letterboxd list and add them to trakt list

Actual behaviour:

Get a crash report

Steps to reproduce:

Run: flexget execute --tasks *comedy*

Config:

https://github.com/effemmeffe/flexget/blob/master/config.yml

Log:

(click to expand) ``` 2020-12-24 11:29:11 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:11 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/687136 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:12 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/david-batra-elephant-in-the-room/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:12 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:12 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:12 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/688527 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:13 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/tom-segura-ball-hog/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:14 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:14 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:14 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/676601 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:14 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/bert-kreischer-hey-big-boy/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:15 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:15 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:15 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/675267 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:15 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/marc-maron-end-times-fun/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:17 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:17 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:17 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/675595 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:17 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/amit-tandon-family-tandoncies/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:18 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:18 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:18 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/656566 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:18 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/pete-davidson-alive-from-new-york/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:21 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:21 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:21 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/673768 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:21 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/bo-burnham-make-happy/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:22 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:22 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:22 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/400608 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:22 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/mike-birbiglia-my-girlfriends-boyfriend/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:24 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:24 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:24 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/216652 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:24 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/oh-hello-on-broadway/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:26 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:26 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:26 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/460822 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:26 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/mike-birbiglia-the-new-one/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:27 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:27 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:27 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/642739 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:27 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/neal-brennan-3-mics/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:29 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:29 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:29 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/435351 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:29 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/bill-burr-im-sorry-you-feel-that-way/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:31 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:31 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:31 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/308571 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:31 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/bill-burr-you-people-are-all-the-same/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:32 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:32 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:32 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/191489 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:33 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/mike-birbiglia-thank-god-for-jokes/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:34 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:34 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:34 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/442087 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:34 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/nate-bargatze-the-tennessee-kid/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:36 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:36 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:36 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/590872 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:36 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/patton-oswalt-annihilation/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:37 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:37 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:38 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/479263 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:38 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/trevor-noah-son-of-patricia/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:39 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:39 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:39 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/558341 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:40 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/jim-jefferies-bare/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:41 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:41 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:41 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/289333 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:41 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/tom-segura-disgraceful/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:42 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:42 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:42 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/497520 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:43 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/adam-sandler-100-fresh/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:44 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:44 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:44 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/517596 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:44 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/ricky-gervais-humanity/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:45 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:45 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:45 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/508933 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:46 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/tig-notaro-happy-to-be-here/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:47 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:47 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:47 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/521512 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:47 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/sarah-silverman-a-speck-of-dust/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:48 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:48 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:48 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/459295 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:49 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/bert-kreischer-secret-time/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:50 DEBUG api_tmdb letterboxd-comedy Looking up TMDb information for 2020-12-24 11:29:50 VERBOSE api_tmdb letterboxd-comedy Searching from TMDb 2020-12-24 11:29:50 DEBUG utils.requests letterboxd-comedy GETing URL https://api.themoviedb.org/3/movie/542694 with args () and kwargs {'params': {'append_to_response': 'alternative_titles', 'language': 'en', 'api_key': 'bdfc018dbdb7c243dc7cb1454ff74b95'}, 'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:50 DEBUG utils.requests letterboxd-comedy GETing URL http://letterboxd.com/film/daniel-sloss-live-shows/ with args () and kwargs {'allow_redirects': True, 'timeout': 30} 2020-12-24 11:29:52 CRITICAL task_queue BUG: Unhandled exception during task queue run loop. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/flexget/task_queue.py", line 46, in run self.current_task.execute() File "/usr/local/lib/python3.6/dist-packages/flexget/task.py", line 87, in wrapper return func(self, *args, **kw) File "/usr/local/lib/python3.6/dist-packages/flexget/task.py", line 722, in execute self._execute() File "/usr/local/lib/python3.6/dist-packages/flexget/task.py", line 688, in _execute self.__run_task_phase(phase) File "/usr/local/lib/python3.6/dist-packages/flexget/task.py", line 517, in __run_task_phase for e in response: File "/usr/local/lib/python3.6/dist-packages/flexget/utils/cached_input.py", line 228, in __iter__ for item in self.iterable: File "/usr/local/lib/python3.6/dist-packages/flexget/plugins/input/letterboxd.py", line 121, in on_task_input yield self.parse_film(film, config) File "/usr/local/lib/python3.6/dist-packages/flexget/plugins/input/letterboxd.py", line 84, in parse_film result = self.tmdb_lookup(soup.find(attrs={'data-tmdb-id': True}).get('data-tmdb-id')) File "/usr/local/lib/python3.6/dist-packages/flexget/plugins/input/letterboxd.py", line 70, in tmdb_lookup tmdb = plugin.get('api_tmdb', self).lookup(tmdb_id=search) File "/usr/local/lib/python3.6/dist-packages/flexget/utils/database.py", line 30, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/flexget/components/tmdb/api_tmdb.py", line 322, in lookup raise LookupError('No criteria specified for TMDb lookup') LookupError: No criteria specified for TMDb lookup ```

Additional information:

kepi commented 3 years ago

I'm not using the letterbox plugin but my 2 cents, guessing from code and crash log: The issue is that https://letterboxd.com/film/daniel-sloss-live-shows/ doesn't have data-tmdb-id attribute value filled in <body> tag and letterboxd plugin doesn't count with such case.

Fast workaround to get at least rest of it running is to change plugins/input/letterboxd.py arround line 81 from:

    def parse_film(self, film, config):
        url = base_url + film.get(config['f_slug'])
        soup = get_soup(requests.get(url).content)
        result = self.tmdb_lookup(soup.find(attrs={'data-tmdb-id': True}).get('data-tmdb-id'))

to something like:

    def parse_film(self, film, config):
        url = base_url + film.get(config['f_slug'])
        soup = get_soup(requests.get(url).content)
        tmdb_id = soup.find(attrs={'data-tmdb-id': True}).get('data-tmdb-id')
        if tmdb_id is None:
          return []
        result = self.tmdb_lookup(tmdb_id)

This workaround should simply skip the entries with data-tmdb-id attribute missing. It would be better to fix the plugin to try to search for tmdb_id later in text, as it is present on the page. I'm not fluent with Soap but it shouldn't be hard to fix.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open 150 days with no activity. Remove stale label or comment or this will be closed in 30 days.

effemmeffe commented 3 years ago

Someone checked this?

gazpachoking commented 2 years ago

Can you paste the log in the actual ticket rather than pastebin? It needs an account to open up, and if the paste ever expires this ticket will lose all context.

effemmeffe commented 2 years ago

Can you paste the log in the actual ticket rather than pastebin? It needs an account to open up, and if the paste ever expires this ticket will lose all context.

Done

gazpachoking commented 2 years ago

@effemmeffe Thanks! Looks like this crash was fixed in https://github.com/Flexget/Flexget/commit/9da2b1ed1dcff7b4f22023067f51b7c7cb40b1fe

Not sure why that show doesn't have a tmdb id on letterboxd though.