FlexGet stopped updating my imdb_watchlist

jmaliska commented 3 months ago

Expected behaviour:

FlexGet downloads all items from my public IMDb watchlist and puts them into movie_list called imdb

Actual behaviour:

Task doesn't fail but there is an error message in the log saying: There was an error during imdb_watchlist input (Unable to get imdb list from imdb react widget. Either the list is empty or the imdb parser of the imdb_watchlist plugin is broken. Original error: 'NoneType' object has no attribute 'group'.), using cache instead.

It started happening 2024-05-20 19:30 CEST

Steps to reproduce:

Step 1: Execute task which updates imdb_watchlist /opt/flexget/bin/flexget -L debug execute --tasks IMDb_queue

Config:

tasks:
  IMDb_queue:
    imdb_watchlist:
      user_id: ur20362530
      list: watchlist
    accept_all: yes
    list_add:
      - movie_list: imdb
    disable:
      - seen
      - seen_movies
      - seen_info_hash
    priority: 1

Log:

(click to expand)

``` 2024-05-22 14:18:09 VERBOSE imdb_watchlist IMDb_queue Retrieving imdb list: watchlist 2024-05-22 14:18:09 DEBUG imdb_watchlist IMDb_queue Requesting: http://www.imdb.com/user/ur20362530/watchlist {'Accept-Language': 'en-us'} 2024-05-22 14:18:09 DEBUG utils.requests IMDb_queue GETing URL http://www.imdb.com/user/ur20362530/watchlist with args () and kwargs {'params': {'view': 'detail'}, 'headers': {'Accept-Language': 'en-us'}, 'allow_redirects': True, 'timeout': 30} 2024-05-22 14:18:12 VERBOSE input_cache IMDb_queue Restored 63 entries from db cache for imdb_watchlist 2024-05-22 14:18:12 ERROR input_cache IMDb_queue There was an error during imdb_watchlist input (Unable to get imdb list from imdb react widget. Either the list is empty or the imdb parser of the imdb_watchlist plugin is broken. Original error: 'NoneType' object has no attribute 'group'.), using cache instead. ```

Additional information:

FlexGet version: 3.11.33
Python version: 3.12.3
Installation method: pip install flexget in virtualenv
Using daemon (yes/no): no
OS and version: Fedora 39
Link to crash log:

miroppb commented 3 months ago

I'm running Flexget on Windows 11, and I'm getting the same error. According to logs, it started around 5/6/24 for me.

Windows 11 Build 22631.3593 Flexget: 3.11.33 Python: 3.12.0 Daemon: yes


2024-05-06 17:36:40 INFO     db_analyze    movies          Running ANALYZE on database to improve performance.
2024-05-06 22:31:42 ERROR    input_cache   movies          There was an error during imdb_watchlist input (Unable to get imdb list from imdb react widget. Either the list is empty or the imdb parser of the imdb_watchlist plugin is broken. Original error: 'NoneType' object has no attribute 'group'.), using cache instead.```

ianbarker commented 3 months ago

IMDB seem to have changed their watchlist page. It no longer uses IMDbReactInitialState.push to set all the IMDB codes

rufftruffles commented 3 months ago

Same issue with flexget 3.11.34

$ flexget execute --task movie-watchlist --no-cache --discover-now
2024-05-28 19:38:42 VERBOSE  task_queue                    There are 1 tasks to execute. Shutdown will commence when they have completed.
2024-05-28 19:38:42 VERBOSE  imdb_watchlist movie-watchlist Retrieving imdb list: ls090066867
2024-05-28 19:38:46 VERBOSE  imdb_watchlist movie-watchlist No movies were found in imdb list: ls090066867
2024-05-28 19:38:46 CRITICAL task          movie-watchlist BUG: Unhandled error in plugin imdb_watchlist: 'NoneType' object is not iterable
Traceback (most recent call last):

  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f7db16c9260>
    └ <Thread(task_queue, started daemon 140177710909120)>
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f7db16c8f40>
    └ <Thread(task_queue, started daemon 140177710909120)>
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(task_queue, started daemon 140177710909120)>
    │    │        │    └ ()
    │    │        └ <Thread(task_queue, started daemon 140177710909120)>    │    └ <bound method TaskQueue.run of <flexget.task_queue.TaskQueue object at 0x7f7daad31b90>>
    └ <Thread(task_queue, started daemon 140177710909120)>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task_queue.py", line 46, in run
    self.current_task.execute()
    │    │            └ <function Task.execute at 0x7f7daedba2a0>
    │    └ <flexget.task.Task object at 0x7f7daadb0750>
    └ <flexget.task_queue.TaskQueue object at 0x7f7daad31b90>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task.py", line 87, in wrapper
    return func(self, *args, **kw)
           │    │      │       └ {}
           │    │      └ ()
           │    └ <flexget.task.Task object at 0x7f7daadb0750>
           └ <function Task.execute at 0x7f7daedba200>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task.py", line 716, in execute
    self._execute()
    │    └ <function Task._execute at 0x7f7daedba160>
    └ <flexget.task.Task object at 0x7f7daadb0750>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task.py", line 685, in _execute
    self.__run_task_phase(phase)
    │                     └ 'input'
    └ <flexget.task.Task object at 0x7f7daadb0750>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task.py", line 514, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
               │                 │       │      └ (<flexget.task.Task object at 0x7f7daadb0750>, {'user_id': 'ur77350165', 'list': 'ls090066867', 'force_language': 'en-us', 't...
               │                 │       └ 'input'
               │                 └ <PluginInfo(name=imdb_watchlist)>
               └ <flexget.task.Task object at 0x7f7daadb0750>
> File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/task.py", line 547, in __run_plugin
    result = method(*args, **kwargs)
             │       │       └ {}
             │       └ (<flexget.task.Task object at 0x7f7daadb0750>, {'user_id': 'ur77350165', 'list': 'ls090066867', 'force_language': 'en-us', 't...
             └ <Event(name=plugin.imdb_watchlist.input,func=wrapped_func,priority=128)>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/event.py", line 21, in __call__
    return self.func(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ (<flexget.task.Task object at 0x7f7daadb0750>, {'user_id': 'ur77350165', 'list': 'ls090066867', 'force_language': 'en-us', 't...
           │    └ <bound method cached.__call__.<locals>.wrapped_func of <flexget.components.imdb.imdb_watchlist.ImdbWatchlist object at 0x7f7d...
           └ <Event(name=plugin.imdb_watchlist.input,func=wrapped_func,priority=128)>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/utils/cached_input.py", line 177, in wrapped_func
    cache = IterableCache(response, self.store_to_db if self.persist else None)
            │             │         │    │              │    └ datetime.timedelta(seconds=7200)
            │             │         │    │              └ <flexget.utils.cached_input.cached object at 0x7f7dab377bd0>
            │             │         │    └ <function cached.store_to_db at 0x7f7dacbb77e0>
            │             │         └ <flexget.utils.cached_input.cached object at 0x7f7dab377bd0>
            │             └ None
            └ <class 'flexget.utils.cached_input.IterableCache'>
  File "/home/torrent/flexget/lib/python3.11/site-packages/flexget/utils/cached_input.py", line 225, in __init__
    self.iterable = iter(iterable)
    │                    └ None
    └ <flexget.utils.cached_input.IterableCache object at 0x7f7daadd0090>

TypeError: 'NoneType' object is not iterable
2024-05-28 19:38:46 CRITICAL manager       movie-watchlist An unexpected crash has occurred. Writing crash report to /home/torrent/.flexget/crash_report.2024.05.28.193846767754.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at https://flexget.com/Plugins/version_checker. You are currently using version 3.11.34
2024-05-28 19:38:46 WARNING  task          movie-watchlist Aborting task (plugin: imdb_watchlist)

rufftruffles commented 3 months ago

@BrutuZ @gazpachoking can we please get some help here?

Thanks

BrutuZ commented 3 months ago

Wut, why me? I've never even used IMDB Watchlists o_O But if IMDB changed their page (again) then the plugin needs updating (again).

KronK0321 commented 2 months ago

Chiming in to say I have also experiencing this problemfor the past month or so. Can provide logs if needed.

FlexGet version: 3.11.34 Python version: 3.9.2 Installation method: pip install flexget in virtualenv Using daemon (yes/no): yes OS and version: Debian Bullseye as an LXC container

ranirahn commented 1 month ago

Like I understand the export csv link does not work anymore, at least not automatically. How can I use that csv file to update my movie list? Problem for me is i purge movie list before I download new list but now that this plugin is broken i have empty list. And i definataly will not add 542 movies by hand into the list. If I know a way to update the list with the exported csv file it will work as workaround for me until plugin gets fixed.

Never mind. I already found temporary solution. Added another task that uses this downloaded csv file and makes another movie list and can temporary use that one instead. Had little trouble because it added also headers to list and then it had trouble using the list because year field had "Year" in it with movie named "Title" and so on. but after I rejected that entry everything started working again.

CSV:
    disable: seen
    list_clear:
      what:
        - movie_list: imdb_movies2
    csv:
      url: file:///home/user/watchlist.csv
      values:
        title: 6
        year: 12
        url: 8
        imdb_name: 6
        movie_name: 6
        imdb_id: 2
        imdb_year: 12
        movie_year: 12
    regexp:
      reject:
        - Const
      from: imdb_id
    accept_all: yes
    list_add:
      - movie_list: imdb_movies2

KronK0321 commented 1 month ago

Never mind. I already found temporary solution. Added another task that uses this downloaded csv file and makes another movie list and can temporary use that one instead.

Can you share how you download the CSV file for your workaround?

ranirahn commented 1 month ago

Can you share how you download the CSV file for your workaround?

Sure. When you go to your Watchlist page there is button up right corner called "Export". After that it gives you link to your exports https://www.imdb.com/exports/?ref_=wl and there are all your requested export lists. And from there you can download these. I renamed my file so that I dont need to change my code when I update the file.

Flexget / Flexget