search

Fr75s / bigscraperqt

Easy to use GUI Scraping tool; supports LaunchBox and more
https://fr75s.github.io/bigscraperqt
GNU General Public License v3.0
7 stars 1 forks source link

Issue: screenscraper error after 5 games #10

Closed Sebio1980 closed 2 years ago

Sebio1980 commented 2 years ago

Hello, i have this issue when i scrap sega32x

# /usr/bin/bigscraper-qt-1.4.1-x86_64.AppImage -d
[I] Program Init... 2022-10-25 14:36:02
[I] Starting application
[I] bigscraper-qt v1.4.1. Made by Fr75s. Licensed under GPLv3
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
[I] Reading options file
[D] Saved Options to File
[I] QML Loaded
[U] Selected Folder (file:///recalbox/share/roms/sega32x)
[U] Selected System (Sega 32X)
-----
[I] Begin Task: Scrape Folder
-----
[I] Data is ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[I] Will write to /recalbox/share/system/.local/share/bigscraper-qt/
[I] Scraping from ScreenScraper
[O] Starting...
[I] Data formatted to [['AFTER_BURNER_COMPLETE', 'AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE', 'BC_RACERS', 'BLACKTHORNE', 'BRUTAL_ABOVE_THE_CLAW', 'COSMIC_CARNAGE', 'DARXIDE', 'DOOM', 'FIFA_SOCCER_96', 'GOLF_MAGAZINE_PRESENTS_36_GREAT_HOLES_STARRING_FRED_COUPLES', 'KNUCKLES_CHAOTIX', 'KOLIBRI', 'METAL_HEAD', 'MORTAL_KOMBAT_II', 'MOTHERBASE', 'MOTOCROSS_CHAMPIONSHIP', 'NBA_JAM_TOURNAMENT_EDITION', 'NFL_QUARTERBACK_CLUB', 'PITFALL_THE_MAYAN_ADVENTURE', 'PRIMAL_RAGE', 'RBI_BASEBALL_95', 'SANGOKUSHI_IV', 'SPACE_HARRIER', 'STAR_TREK_STARFLEET_ACADEMY_STARSHIP_BRIDGE_SIMULATOR', 'STAR_WARS_ARCADE', 'STELLAR_ASSAULT', 'T_MEK', 'TEMPO', 'TIME_WARNER_32X_CMD_DOWNLOAD_CARTRIDGE', 'TOUGHMAN_CONTEST', 'VIRTUA_FIGHTER', 'VIRTUA_RACING_DELUXE', 'WORLD_SERIES_BASEBALL_STARRING_DEION_SANDERS', 'WWF_RAW', 'WWF_WRESTLEMANIA_THE_ARCADE_GAME'], ['/recalbox/share/roms/sega32x/After Burner Complete (Europe).zip', '/recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip', '/recalbox/share/roms/sega32x/BC Racers (USA).zip', '/recalbox/share/roms/sega32x/Blackthorne (USA).zip', '/recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip', '/recalbox/share/roms/sega32x/Cosmic Carnage (Europe).zip', '/recalbox/share/roms/sega32x/Darxide (Europe) (En,Fr,De,Es).zip', '/recalbox/share/roms/sega32x/Doom (Europe).zip', '/recalbox/share/roms/sega32x/FIFA Soccer 96 (Europe) (En,Fr,De,Es,It,Sv).zip', '/recalbox/share/roms/sega32x/Golf Magazine Presents - 36 Great Holes Starring Fred Couples (Europe).zip', "/recalbox/share/roms/sega32x/Knuckles' Chaotix (Europe).zip", '/recalbox/share/roms/sega32x/Kolibri (USA, Europe).zip', '/recalbox/share/roms/sega32x/Metal Head (Europe) (En,Ja).zip', '/recalbox/share/roms/sega32x/Mortal Kombat II (Europe).zip', '/recalbox/share/roms/sega32x/Motherbase (Europe).zip', '/recalbox/share/roms/sega32x/Motocross Championship (Europe).zip', '/recalbox/share/roms/sega32x/NBA Jam - Tournament Edition (World).zip', '/recalbox/share/roms/sega32x/NFL Quarterback Club (World).zip', '/recalbox/share/roms/sega32x/Pitfall - The Mayan Adventure (USA).zip', '/recalbox/share/roms/sega32x/Primal Rage (USA, Europe).zip', "/recalbox/share/roms/sega32x/RBI Baseball '95 (USA).zip", '/recalbox/share/roms/sega32x/Sangokushi IV (Japan).zip', '/recalbox/share/roms/sega32x/Space Harrier (Europe).zip', '/recalbox/share/roms/sega32x/Star Trek - Starfleet Academy - Starship Bridge Simulator (USA).zip', '/recalbox/share/roms/sega32x/Star Wars Arcade (Europe).zip', '/recalbox/share/roms/sega32x/Stellar Assault (USA, Europe).zip', '/recalbox/share/roms/sega32x/T-MEK (USA, Europe).zip', '/recalbox/share/roms/sega32x/Tempo (Japan, USA).zip', '/recalbox/share/roms/sega32x/Time Warner 32X CMD Download Cartridge (USA).zip', '/recalbox/share/roms/sega32x/Toughman Contest (USA, Europe).zip', '/recalbox/share/roms/sega32x/Virtua Fighter (Europe).zip', '/recalbox/share/roms/sega32x/Virtua Racing Deluxe (Europe).zip', '/recalbox/share/roms/sega32x/World Series Baseball Starring Deion Sanders (USA).zip', '/recalbox/share/roms/sega32x/WWF Raw (World).zip', '/recalbox/share/roms/sega32x/WWF WrestleMania - The Arcade Game (USA).zip'], 'sega32x', 19]
[I] Getting User Page
[O] Showing Main Bar
[O] Hiding Game Bar
[O] Getting User Page
[D] Page Request Successful
[D] ScreenScraper Requests As of this line: 19
[D] Maximum alotted requests: 300000
[D] Your Level: 11 (will be helpful for me to mimic your thread count)
[D] Max number of threads: 7
[D] Threadpool Maximum Capacity: 7
[I] Initializing Thread for AFTER_BURNER_COMPLETE
[D] GAME FILE: /recalbox/share/roms/sega32x/After Burner Complete (Europe).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['AFTER_BURNER_COMPLETE', '/recalbox/share/roms/sega32x/After Burner Complete (Europe).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: 0a8cbd7835e53bfc0ac2363b2d9abb7b
[D] SHA1 Hash: 06d3034ef60b032e755ec0e3e715a50906d07e7e
[D] CRC32 Hash: 708147714, CRC32 Hex-Formatted Hash: 2a357a02
[I] Attempting to get page for AFTER_BURNER_COMPLETE
[D] (AFTER_BURNER_COMPLETE) Page Request Successful
[I] (AFTER_BURNER_COMPLETE) Collecting Game Metadata
[D] Checking for Game Name
[D] Adding Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Game Name: ['After Burner Complete']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for After Burner Complete
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Initializing Thread for AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE
[D] GAME FILE: /recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE', '/recalbox/share/roms/sega32x/Amazing Spider-Man, The - Web of Fire (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: e03077aef49dd67445d96d76ad6e457c
[D] SHA1 Hash: fc63b1cb18af47959541bd3a6157467790ca00bb
[D] CRC32 Hash: 471177555, CRC32 Hex-Formatted Hash: 1c159953
[I] Attempting to get page for AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] (AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE) Page Request Successful
[I] (AMAZING_SPIDER_MAN,_THE_WEB_OF_FIRE) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Adding Game Name
[D] Checking for Game Release
[D] Game Name: ['Spider-man - Web Of Fire']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (2)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Spider-man - Web Of Fire
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Initializing Thread for BC_RACERS
[D] GAME FILE: /recalbox/share/roms/sega32x/BC Racers (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BC_RACERS', '/recalbox/share/roms/sega32x/BC Racers (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: cef7e91bedaba3e87e02da09cd8246d3
[D] SHA1 Hash: ad6089e370c887e11428259b57447529f50c95b7
[D] CRC32 Hash: 378450490, CRC32 Hex-Formatted Hash: 168eb23a
[I] Attempting to get page for BC_RACERS
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] (BC_RACERS) Page Request Successful
[I] (BC_RACERS) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Checking for Game Name
[D] Checking for Game Name
[D] Adding Game Name
[D] Game Name: ['Bc Racers']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Bc Racers
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Downloading Media (2 / 40) (for Bc Racers)
[I] Initializing Thread for BLACKTHORNE
[D] GAME FILE: /recalbox/share/roms/sega32x/Blackthorne (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BLACKTHORNE', '/recalbox/share/roms/sega32x/Blackthorne (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: 16511f98621ff43adbca331985a99b43
[D] SHA1 Hash: 4f25dda03075f544ef4ad6034225decd619696df
[D] CRC32 Hash: 1531738562, CRC32 Hex-Formatted Hash: 5b4c79c2
[I] Attempting to get page for BLACKTHORNE
[D] Download Successful, Writing to file
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (2 / 40) (for Bc Racers)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[I] Initializing Thread for BRUTAL_ABOVE_THE_CLAW
[D] GAME FILE: /recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip
[D] SYSTEM INFO: 19, sega32x, Sega 32X
-----
[D] Thread Start
[D] ['file:///recalbox/share/roms/sega32x', 'Sega 32X']
[D] {'video': True, 'videoOverLimit': False, 'glassyTitle': True, 'recache': False, 'localPaths': False, 'region': 'Europe', 'module': 'ScreenScraper', 'languageOverride': 'None', 'maxLogFiles': 50, 'screenScraperUser': 'Sebio', 'screenScraperPass': '19561956'}
[D] ['BRUTAL_ABOVE_THE_CLAW', '/recalbox/share/roms/sega32x/Brutal - Above the Claw (USA).zip', [19, 'sega32x', 'Sega 32X']]
[D] MD5 Hash: b9cfb502d47caf92f49db67cf5330162
[D] SHA1 Hash: 088dee9cd777b2951431810f975a6c69c46dec53
[D] CRC32 Hash: 2859814636, CRC32 Hex-Formatted Hash: aa754eec
[I] Attempting to get page for BRUTAL_ABOVE_THE_CLAW
[D] Download Successful, Writing to file
[D] (BLACKTHORNE) Page Request Successful
[I] (BLACKTHORNE) Collecting Game Metadata
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Checking for Game Name
[D] Checking for Game Release
[D] Adding Game Release
[D] Only Year Available
[D] Checking for Game Name
[D] Checking for Game Name
[D] Adding Game Name
[D] Game Name: ['Blackthorne']
[D] Checking for Game Overview
[D] Adding Game Overview
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (0)
[D] Checking for Game Genre
[D] This Genre name not found yet
[D] Adding this genre (1)
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Checking for Game Genre
[D] Getting Platform
[D] Getting Developers
[D] Getting Publishers
[D] Getting Max Players
[D] Getting Rating
[I] Getting Media for Blackthorne
[I] Downloading Media (3 / 48) (for Blackthorne)
[I] Downloading Media (3 / 48) (for Blackthorne)
[I] Downloading Media (3 / 67) (for After Burner Complete)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[I] Downloading Media (4 / 40) (for Spider-man - Web Of Fire)
[D] Download Successful, Writing to file
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[D] This image is not unique or is already downloaded.
[I] Downloading Media (2 / 40) (for Bc Racers)
[D] (BRUTAL_ABOVE_THE_CLAW) Page Request Successful
Traceback (most recent call last):
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/bsqt/scrapemany.py", line 804, in run
    page_content = page.json()
  File "/tmp/.mount_bigscrhiGuii/usr/lib/python3.10/site-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] Erreur : Rom/Iso/Dossier non trouv?e !  : 0
Aborted
FrenchGithubUser commented 2 years ago

Getting the same error here

Fr75s commented 2 years ago

This should hopefully be fixed in 4bcd69b, though be aware that this is an error on ScreenScraper's side, therefore whatever game you're scraping that results in this error would be skipped. If you're still having this error, feel free to reopen this issue.