search

muldjord / skyscraper

Powerful and versatile game scraper written in c++
GNU General Public License v3.0
479 stars 127 forks source link

World Of Spectrum not scraping #268

Open HoraceAndTheSpider opened 3 years ago

HoraceAndTheSpider commented 3 years ago

World of Spectrum appears to no longer be scraping. FIles are named as pretty much just the game name.

It looks like the style of the site has changed, so i guess this would need adaptation.

There is also a newer ZX database here: https://spectrumcomputing.co.uk/ that was intended by many to replace WoS ... might be worth considering?

muldjord commented 3 years ago

Do you know the people behind spectrumcomputing.co.uk? I need permission to scrape from them.

muldjord commented 3 years ago

WoS now has an API it seems. It requires a key. If you can provide me with one, I can update the implementation.

HoraceAndTheSpider commented 3 years ago

I don’t but I’ll get on the case :)

muldjord commented 3 years ago

Thanks! :)

leemm commented 2 years ago

Hate to bring this up again but I assume this was never fixed?

detain commented 2 years ago

This is how you use the new API, and you can just use 'test' as the API key. you need to follow redirects (301) as well (hence the -L option)

curl -s -L "http://worldofspectrum.org/infoseek/api/software?X-API-KEY=test&title=yumiko+in+the+haunted+mansion"

Returns a JSON like 

{"count":1,"totalRecords":"1","titles":[{"id":24171,"title":"Yumiko in the Haunted Mansion","slug":"yumiko-in-the-haunted-mansion-fun-forge","turn_types_text":null,"turn_type_id":null,"availability_id":"1","availability_text":"Available","no_players":"1","entry_type_id":"11","entry_type_text":"Arcade: Maze","entry_group":"games","is_x_rated":"0","is_crap":"0","comments":null,"roles":[{"role":"Music","role_slug":"music","person":"Yerzmyey","person_slug":"yerzmyey","person_id":"6163","role_id":"22241"},{"role":"(Unknown)","role_slug":"unknown","person":"Leszek Chmielewski Daniel","person_slug":"leszek-chmielewski-daniel","person_id":"1460","role_id":"22242"},{"role":"(Unknown)","role_slug":"unknown","person":"Yerzmyey","person_slug":"yerzmyey","person_id":"6163","role_id":"22243"}],"features":[{"id":"37","feature":"TR-DOS Support","slug":"tr-dos-support"}],"controls":[{"id":"2","control":"Interface 2 (right)","slug":"interface-2-right"},{"id":"4","control":"Kempston","slug":"kempston"},{"id":"5","control":"Redefinable Keys","slug":"redefineable-keys"}],"themes":[],"series":[],"publishers":[{"id":"4604","name":"Fun Forge","slug":"Fun-Forge"}]}]}
detain commented 2 years ago

Also There is a DB at zxdb/ZXDB which I believe is tied to https://api.zxinfo.dk/v3/ It seems to be a db combining all of these zx spectrum sites:

I plan on updating my fork of skyscraper to use the new WoS api and hopefully incorporate this as well.

muldjord commented 2 years ago

I started working on implementing the new API when you wrote the last message, but if you end up doing so, a PR would be appreciated for that feature. Thanks. I doubt I will complete my implementation anytime soon.