search

muldjord / skyscraper

Powerful and versatile game scraper written in c++
GNU General Public License v3.0
480 stars 127 forks source link

ScreenScraper APIv2 returned invalid / empty Json #158

Closed vanativo closed 4 years ago

vanativo commented 5 years ago

Describe the bug Skyscraper is reporting that ScreenScraper is currently returning malformed JSON. Not sure if this is a problem with ScreenScraper or with Skyscraper's validation so starting here.

To Reproduce If I run the following command I will see the errors: /opt/retropie/supplementary/skyscraper/Skyscraper -p amiga -s screenscraper --videos (I also provide a username and password which I'm not including here).

Special circumstances NA

Terminal output ScreenScraper APIv2 returned invalid / empty Json First 256 characters of answer was: { "header" : { "APIversion" : "2.0", "dateTime" : "2019-08-13 01:39:09", "commandRequested" : "https://www.screenscraper.fr/api2/jeuInfos.php?devid=muldjord&devpassword=****&softname=skyscraper3.2.2&ssid=Vinny&sspassword=***&

Technical information

Additional context

muldjord commented 5 years ago

Can you please provide the exact filename of the game that causes this? I can't reproduce it with my own files and the returned error is truncated before the actual error.

bvassie commented 5 years ago

I am getting this with the majority of my Sega CD games.

All cue files have their matching bin files.

muldjord commented 5 years ago

Would it be possible for one of you to pull from master, compile that version and rerun it until you get this error? I still can't reproduce it, but I have an idea what is going on. But I need a full JSON document answer to figure out exactly how to fix it. And the version I currently have on master will output the entire answer to ~/.skyscraper/screenscraper_error.json. I need someone who gets this error to attach that file here.

The low-down: This probably happens because ScreenScraper's answer contains an error in the JSON it returns. Maybe they are using illegal characters or bad scoping. I don't currently know.

muldjord commented 5 years ago

IMPORTANT!!! I've just released Skyscraper 3.2.3 which, among other things, now exports screenscraper erroneous JSON to ~/.skyscraper/screenscraper_error.json. So if you guys could update and when you get this error again, please attach this file to this bug so I can analyze it. I'd really appreciate that. And don't worry, the passwords have been removed from the file prior to export.

bvassie commented 5 years ago

As requested...

Muldjord: JSON REMOVED AS I NOW HAVE IT LOCALLY, THANKS!
muldjord commented 5 years ago

Ok, so I've located the error. The returned JSON document has a comma at the end of an array that does not have any more variables in it. This makes the JSON parser invalidate the document. I've reported this to ScreenScraper and I'll update you when I know more. When they fix this, it will just start working without a need to update Skyscraper.

vanativo commented 5 years ago

Thank you @muldjord, much appreciate the quick reply!

muldjord commented 5 years ago

Hey guys, ScreenScraper replied, and they told me they fixed the issue yesterday. It can take up to 48 hours for their cache to be regenerated, so please wait a couple of days and try this again. If on Friday you still have trouble, let me know. :)

vanativo commented 5 years ago

@muldjord , sorry to report it looks like it's still happening.

bvassie commented 5 years ago

Not sure if updating to 3.2.3 is required but the issue is resolved for me.

vanativo commented 5 years ago

I am on 3.2.3 and still seeing the error. Attached is the error JSON. It still appears as if there is an extra comma at the end of the JSON array. Could it be a caching issue? I did purge the entire cache before trying again so either it's not cache related or the cache purge didn't work.

On Sat, 17 Aug 2019 at 16:01, Ben Vassie notifications@github.com wrote:

Not sure if updating to 3.2.3 is required but the issue is resolved for me.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/muldjord/skyscraper/issues/158?email_source=notifications&email_token=AMO4APN2ZFZNCEOBNUVUVE3QE6H4NA5CNFSM4ILF6BBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4QEJSQ#issuecomment-522208458, or mute the thread https://github.com/notifications/unsubscribe-auth/AMO4APKVMW3IPI4FHR6DQYLQE6H4NANCNFSM4ILF6BBA .

screenscraper_error.json.txt

muldjord commented 5 years ago

It's not a Skyscraper caching issue. The raw JSON is not saved locally, so if it complains about JSON it's because it came directly from ScreenScraper. I've re-reported this to the ScreenScraper guys. Thanks for letting me know! I'll keep you posted.

surtarso commented 5 years ago

hi, screenscraper getting APIv2 errors on Intellivision, wonderswan (black and white), saturn and amstradCPC, some games wont scrape at all on those platforms. Also, I couldnt get a platform name to scan channelf and pokemini. And for some reason, no NDS game of mine was found in any scraper...

screenscraper_error.json.txt

muldjord commented 5 years ago

@surtarso, thank you for the file, it contains the same error as the others are experiencing. I am currently still awaiting answer from ScreenScraper. I might end up fixing this myself by simply removing the error after receiving the data before parsing it.

muldjord commented 5 years ago

Hi guys, I haven't heard back from ScreenScraper, but I've decided to fix this on the fly instead. I've just released Skyscraper 3.2.4 where this should no longer occur. I simply replace any instances of the error with the same JSON but without the error. I haven't been able to test it since I haven't seen the error myself, so I would really appreciate it if some of you would try it out.

So please update and verify that you have 3.2.4 afterwards. Then try running it for some of the game you know didn't work before and report back here. :) Thanks!

surtarso commented 5 years ago

I see no more errors, and got lots of new games that previous scrapes missed : ) thx!

muldjord commented 5 years ago

Fantastic! Thanks for confirming. This should have alleviated all of the problems reported in this bug. If any of the rest of you are still having issues, please let me know. I'll leave this open for a week or so.

Taranchul commented 5 years ago

I would like to offer some more JSON errors for the Gameboy Advance, but Github rejects the .txt file as unsupported. But other commenters attached their screenscraper_error.json.txt. How do I do that? Thanks.

muldjord commented 5 years ago

I would like to offer some more JSON errors for the Gameboy Advance, but Github rejects the .txt file as unsupported. But other commenters attached their screenscraper_error.json.txt. How do I do that? Thanks.

Have you encountered these errors with the new 3.2.4 version of Skyscraper? That's what I am interested in knowing right now. :) Pre-3.2.4 did not contain my new on-the-fly fix to the faulty JSON returned from ScreenScraper.

If so, you can drag the file in which should attach it. Perhaps try to rename it to screenscraper_error.json.txt first.

vanativo commented 5 years ago

I was able to successfully complete a run with 3.2.4 without error. Thank you @muldjord !

Taranchul commented 5 years ago

Me too, thanks! With 3.2.4, the errors are gone. So, it was a problem of Skyscraper after all, and not of ScreenScraper, or both? (just curious)

muldjord commented 5 years ago

Me too, thanks! With 3.2.4, the errors are gone. So, it was a problem of Skyscraper after all, and not of ScreenScraper, or both? (just curious)

No, not a Skyscraper problem. I just fixed the faulty JSON returned by ScreenScraper on my end by removing the error before parsing it. :) Then it works.

Taranchul commented 5 years ago

So, if I understand you correctly, the error(s) at ScreenScraper are still there but just aren't displayed by Skyscraper anymore? So, nobody (of us) can notice them and report them to ScreenScraper for correction? I don't think I like this solution very much, since it robs us of the option to notice and report them, so they may be fixed on ScreenScraper's side. If so, an option to enable or disable the errors would be nice. <:)

muldjord commented 5 years ago

...since it robs us of the option to notice and report them

Incorrect. This is only a workaround for this single flaw. If a new flaw appears, it will be displayed to you as it should. In the meantime ScreenScraper might fix it and you won't notice the difference. At that point I can remove the fix again. I'm sure you can see the logic in that.

Taranchul commented 5 years ago

So if I (hopefully, finally) understand you correctly now, it's one single flaw in ScreenScraper that's producing JSON errors for some games, but not others. Then I can see the logic in suppressing the errors. I thought that there is a flaw in each dataset that produces this error and thus, makes it necessary to report every single case. Thank you for taking the time to make me understand. I really appreciate it. O:)

TheStarglider commented 5 years ago

I am on 3.3.0 of Skyscraper and I am getting the issues mentioned above. I have tried it on psx games where none can be found (previously all worked).

muldjord commented 5 years ago

Please attach the screenscraper_error.json file as described in the terminal output when you encounter the error. Otherwise I have nothing to debug from. Thanks.

TimberlAndRE commented 5 years ago

Hi muldjord, same for me, trying to scrape Game Boy files and other roms. Seems none is working anymore with screenscraper.fr (moby, igdb and gamesdb working fine) I'm also on 3.3.0 screenscraper_error.zip

muldjord commented 5 years ago

So yeah, there's something really wrong with their API at the moment. It's not on Skyscraper's end.

Wow, something is really going on at ScreenScraper right now, not sure what exactly. Take a look at their site screenscraper.fr. Seems like they are switching servers or something. It now has CPU1 and CPU2, which used to be just CPU. And the number of scrapers is very, very low. I wonder what's going on.

TimberlAndRE commented 5 years ago

Ok, thx for the fast reply (and not at least ur great work!!!) I'll wait a few days and try again

muldjord commented 5 years ago

Just had a look at their Discord and all scrapers using the API seem to have issues at the moment due to extreme server loads. So we'll have to wait it out. And while I'm at it, remember to support them if you can / want to (I'm not affiliated). Same goes for RetroPie.

TheStarglider commented 5 years ago

Please attach the screenscraper_error.json file as described in the terminal output when you encounter the error. Otherwise I have nothing to debug from. Thanks.

My apologies, totally forgot to attach it. However my error was the same as TimberlAndRE and you've already answered it incredibly quickly! Thank you!

donjengibre commented 5 years ago

I’m also having issues - signed up and contributed to get additional threads on screenscraper this weekend. Have tried a test scrape on my Atari Lynx files (only 80 or so) and on Friday the server kept timing out without scraping, and last night I was getting JSON errors on every file. The screenscraper website was down on and off over the weekend so as mentioned it sounds as though the issues could be with the server? I’ll test again this evening and let you know if it resolves.

donjengibre commented 5 years ago

Managed to run a number of successful scrapes this evening without changing anything on my end - so I suspect this issue may have been to do with the servers. Hopefully it’s OK now.

muldjord commented 5 years ago

It is a ScreenScraper server issue, I know that for sure as mentioned in a previous comment. Thanks for testing.

TimberlAndRE commented 5 years ago

Problem seems gone! although there is no new message from screenscraper at discord, it works (tested with skyscraper and ~1500 Game Boy Roms). Maybe they turned it off and on again ;-)