Mobygames Scraper broken

[ISharkI]

Hello,

I just tried to scrape covers online from mobygames and ran into an error. It seems the html code of the result page changed somehow and the handed url to urllib2 is empty. I do not know how the mobygames page worked before so I have no Idea if something changed.

Scraping with e.g. thegamedb works fine.

Tested in 0.97 and 0.98a, master branch gives some other error but I assumed that was work in progress for something else.

See the Log attached below.

ERROR: AEL INFO : _roms_import_roms() Starting ROM scanner ...
ERROR: AEL INFO : Launcher name "dolphin (Wii)"
ERROR: AEL INFO : launcher ID   "6659c57952b6a431b6cab4ed9bacd353"
ERROR: AEL INFO : ROM path  "/share/games/Console Games/Nintendo - Wii/"
ERROR: AEL INFO : ROM exts  "wbfs|iso"
ERROR: AEL INFO : Multidisc True
ERROR: AEL INFO : Platform  "Nintendo Wii"
ERROR: AEL INFO : Report file OP "special://profile/addon_data/plugin.program.advanced.emulator.launcher/reports/roms_root_category_dolphin_(Wii)_6659c5_report.txt"
ERROR: AEL INFO : Report file  P "/home/user/.kodi/userdata/addon_data/plugin.program.advanced.emulator.launcher/reports/roms_root_category_dolphin_(Wii)_6659c5_report.txt"
ERROR: AEL INFO : Launcher ROM database contain 0 items
ERROR: AEL INFO : Checking for unset artwork directories ...
ERROR: AEL INFO : Checking for duplicated artwork directories ...
ERROR: AEL INFO : No duplicated asset dirs found
ERROR: AEL INFO : Scanning and caching files in asset directories ...
DEBUG: DialogProgress::Open called
DEBUG: ------ Window Init (DialogConfirm.xml) ------
ERROR: AEL INFO : Removing dead ROMs ...
ERROR: AEL INFO : Launcher is empty. No dead ROM check.
DEBUG: DialogProgress::Open called (already running)!
DEBUG: ------ Window Init (DialogConfirm.xml) ------
ERROR: AEL INFO : Scanning files in /share/games/Console Games/Nintendo - Wii/
ERROR: AEL INFO : Recursive scan activated
ERROR: AEL INFO : File scanner found 63 files
DEBUG: DialogProgress::Open called (already running)!
DEBUG: ------ Window Init (DialogConfirm.xml) ------
ERROR: AEL INFO : ROM does not belong to a multidisc set.
ERROR: AEL ERROR: _roms_scrap_asset() Caching selected game "Dawn of Discovery / Nintendo Wii"
ERROR: AEL ERROR: _roms_scrap_asset() Scraper object ID 139837827862968 (name MobyGames)
ERROR: AEL ERROR: SingleInstance::__exit__() Unhandled excepcion in protected code
ERROR: EXCEPTION Thrown (PythonToCppException) : 

-->Python callback/script returned the following error<--- NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
   Error Type: <type 'exceptions.ValueError'>
   Error Contents: unknown url type: 
   Traceback (most recent call last):
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/addon.py", line 39, in <module> main.Main().run_plugin()
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 270, in run_plugin self.run_protected(command, args)
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 363, in run_protected self._command_add_roms(args['launID'][0])
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 2401, in _command_add_roms self._roms_import_roms(launcher)
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 8809, in _roms_import_roms romdata = self._roms_process_scanned_ROM(launcherID, ROM)
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 9071, in _roms_process_scanned_ROM local_asset_list[i], ROM, launcher)
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/main.py", line 9252, in _roms_scrap_asset net_download_img(image_url, image_path)
     File "/home/user/.kodi/addons/plugin.program.advanced.emulator.launcher/resources/net_IO.py", line 82, in net_download_img f.write(urllib2.urlopen(req).read())
     File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
     File "/usr/lib/python2.7/urllib2.py", line 421, in open
protocol = req.get_type()
     File "/usr/lib/python2.7/urllib2.py", line 283, in get_type
   raise ValueError, "unknown url type: %s" % self.__original
   ValueError: unknown url type: 
-->End of Python script error report<--

19:29:50.638 T:139840363166080   DEBUG: ------ Window Init (DialogNotification.xml) ------
19:29:50.639 INFO: Python script stopped
19:29:50.639DEBUG: Thread LanguageInvoker 139838384019200 terminating
19:29:50.640 T:139840363166080   DEBUG: ------ Window Deinit (DialogConfirm.xml) ------
19:29:56.194 T:139840363166080   DEBUG: ------ Window Deinit (DialogNotification.xml) ------

Edit: TheGamesDB just seems to work for 0.98a as I stumbled accross Issue #52 ...

[Wintermute0110]

Thanks a lot for reporting. I am unable to reproduce the bug. However, when doing so I found another Unicode related bug an fixed it :)

Can you provide another log reproducing the bug? This time please enable AEL DEBUG log level (open AEL settings window, go to Advanced tab, set DEBUG in Log level box) and disable Kodi debug.

[ISharkI]

I will do that later when I'm back at the device.

But due to trial and error (what games got pictures and what games not) I can already tell you that the cause was a ROM with "Pokémon" in it.

[Wintermute0110]

Yes, this issue affects ROMs with Unicode characters in the filename when scraping pictures you already have and are in the Kodi cache. issue #52 affected scraping all images for ROMs with Unicode characters whether you have the pictures or not. Both issues should be fixed now in release-0.9.8 branch. Please install it and close this issue if everything OK. If not, please provide an AEL debug log as I told you a couple of post above.

[ISharkI]

Mobygames still does not seem to work. This time it crashed for the rom "ANNO - Create a New World [RN4P41]".

See the attached file for the debug log. ael-mobygames.log

[Wintermute0110]

OK, thanks again for the report. Will have a look ASAP.

[Wintermute0110]

You were right, MobyGames has made some changes in the screenshot page for some games and the scraper needed an update. I was able to reproduce the bug and I believe it is fixed in the release-0.9.8 branch. I will close the issue now. Reopen or create a new one if you find something else.