search

sselph / scraper

A scraper for EmulationStation written in Go using hashing
MIT License
449 stars 88 forks source link

ss: cannot parse response: "invalid character 'F' looking for beginning of value #251

Open lselinger opened 5 years ago

lselinger commented 5 years ago

Ok so I've seen a few posts regarding similar messages but can't seem to see how to rectify this. I was running a scrape with ss credentials and it was going fine then all of a sudden I started seeing that error:

ss: cannot parse response: "invalid character 'F' looking for beginning of value

I removed my credentials from the scraper.sh script and tried again and it processed fine ... for a little while then the errors started again. I dropped everything for a bit and tried again with credentials and same thing ... worked for a little while and then started seeing the same error. When there's a "hash not found" it processes really quick but these invalid character errors take forever to get through. I'm wondering if they've implemented something to restrict # of scrapes from even registered accounts and the code isn't understanding the response?

gnilebein commented 5 years ago

I have a similar problem:

2019/08/21 12:45:38 ERR: error processing 3 Choume no Tama - Tama and Friends - 3 Choume Obake Panic!! (Japan).gb: ss: cannot parse response: "invalid character '<' after top-level value"

sselph commented 5 years ago

The underlying issue is that ScreenScraper's errors are returned with a 200 OK code so I'm attempting to parse the response as json. I wrote several things to try and capture the errors at the time and handle them but as the error messages change or new ones are added, you get these responses. Another issue is the json is typically built using php and string concatenation so there have been several issues of things not being properly escaped or being invalid json.

invalid character 'F' looking for beginning of value would be an error of some sort if I had to guess. invalid character '<' after top-level value might be bad json but hard to tell.

I won't make any promises but maybe I'll add a flag or something so that these errors print the entire response so you can at least see why it is happening, then we can determine if we need to do something to better handle them.

lselinger commented 5 years ago

I think if there was any way to capture viable error output at least you could bake in an exclusion based on the error ... that being said you may have nailed the caveat ... they would have to either be pretty static or have something/things exactly the same in each return to properly identify it. I waited a day or two and everything completed with no issues so I either got screens of this error, or successful completion. @sselph very much appreciate you taking a look

sselph commented 5 years ago

For reference, in case someone gets to it before me, the relevant code is here https://github.com/sselph/scraper/blob/master/ss/ss.go#L366

gnilebein commented 5 years ago

Currently the api gives this error after the json string: <br /> <b>Warning</b>: sleep() expects exactly 1 parameter, 2 given in <b>/var/www/html/scraper/api/inc_bottom.inc.php</b> on line <b>10</b><br />

Do you think this is the cause scraper could not parse the json response?

sselph commented 5 years ago

Exactly. That would be the issue. It attempts to parse the json and sees '<' which is not valid. But in reality that should not be returning 200 OK and would've been caught earlier in my code. I may print the entire error (or a truncated version) on those unknowns so it is more clear to users what happened. In your case, that looks like an error in the php code that is used to generate the json so there is an error message included with the other json data.

gnilebein commented 5 years ago

Is there a simple posibility to cut of the response after the valid json part?

theknob commented 4 years ago

Same here. Still getting ss: cannot parse response: "ERR: error processing [game]: ss: cannot parse response: "invalid character '<' looking for beginning of value"

Wetzel402 commented 3 years ago

I am seeing this error today. I see the issue report is still open but wanted to confirm this is still the issue. I've noted that running the scrapper on four ROMS several times today it did complete successfully once. All of the other attempts failed with the reported error.

Edit: I've run several test scrapes with the same 4 ROMs today and have not had this error at all. I found some threads reporting this issue might be related to ss server load.

Dinth commented 3 years ago

Same here. It started parsing roms all right, parsed through all MAME roms, but when it started parsing GBA i started receiving this error and got the same with all GBA, NDS, PSX, SNES and PSP roms ive got.

2021/01/28 10:18:00 INFO: Starting: /home/pi/RetroPie/roms/fba/sengoku2.zip
2021/01/28 10:18:00 INFO: Starting: /home/pi/RetroPie/roms/fba/shocktr2.zip
2021/01/28 10:18:01 INFO: Starting: /home/pi/RetroPie/roms/fba/shocktro.zip
2021/01/28 10:18:01 INFO: Starting: /home/pi/RetroPie/roms/fba/sonicwi3.zip
2021/01/28 10:18:02 INFO: Starting: /home/pi/RetroPie/roms/fba/wakuwak7.zip
2021/01/28 10:18:05 arcade  srcs: 
2021/01/28 10:18:05 console srcs: ss
2021/01/28 10:18:05 INFO: Checking for new hash.csv.
2021/01/28 10:18:06 INFO: Upgrading hash.csv:  -> "d9eedf6e622a8ab6a1a14486bb90ffb8".
2021/01/28 10:18:07 INFO: Starting: /home/pi/RetroPie/roms/gba/Advance Wars 2 - Black Hole Rising (USA, Australia).gba
2021/01/28 10:18:07 INFO: Starting: /home/pi/RetroPie/roms/gba/Breath of Fire II (USA).gba
2021/01/28 10:18:07 INFO: Starting: /home/pi/RetroPie/roms/gba/Advanced Wars  # GBA.GBA
2021/01/28 10:18:07 INFO: Starting: /home/pi/RetroPie/roms/gba/Breath of Fire (E) (M3).gba
2021/01/28 10:18:08 ERR: error processing /home/pi/RetroPie/roms/gba/Breath of Fire II (USA).gba: ss: cannot parse response: "invalid character 'F' looking for beginning of value"
2021/01/28 10:18:08 INFO: Starting: /home/pi/RetroPie/roms/gba/Breath of Fire II (USA).gba
2021/01/28 10:18:08 ERR: error processing /home/pi/RetroPie/roms/gba/Advanced Wars  # GBA.GBA: ss: cannot parse response: "invalid character 'F' looking for beginning of value"
NeilRickards commented 3 years ago

www.screenscraper.fr/api/jeuInfos.php appears to returning "File not found" which scraper is trying to parse as json.

But in reality that should not be returning 200 OK and would've been caught earlier in my code.

I'm not sure you're actually checking the return code currently. I've submitted a PR to fix this: https://github.com/sselph/scraper/pull/261

The better question though is why ScreenScraper is returning a 404

NeilRickards commented 3 years ago

It looks like maybe api2 is up but api is down. Requests to api2 work, but the response format is different: name and synopsis are arrays for different languages.

neveride84 commented 3 years ago

I am also getting: ss: cannot parse response: "invalid character 'F' looking for beginning of value"

Is there anything we can do about it? I cannot scrape anything for such classics as Mario Bros or Sonic the Hedgehog...

BrainSlugs83 commented 3 years ago

@michalbilinski -- I've had this issue too last time I tried to use it -- I ended up moving on, because as far as I know this scraper has just been completely broken for like 3+ years. 🤷🏻‍♂️

tzachyb commented 3 years ago

Getting it for every parsing attempting using this scraper against ScreenScraper. Darn, works fine with SkyScraper. I prefer sselph but preference be ****ed if it hasn't worked for over a year.