sselph / scraper

A scraper for EmulationStation written in Go using hashing
MIT License
449 stars 88 forks source link

Update hashes for all systems #20

Open sselph opened 9 years ago

sselph commented 9 years ago

Need to update the dataset with all the new games in the GDB.

sselph commented 7 years ago

Thanks, those seem to match. I'll work on adding the checks and tests.

sselph commented 7 years ago

@dd900 Fixed it

ghost commented 7 years ago

@sselph - Hot Off the Press!

Here is the completed set of all the .ctg files I could find for the TI-99/4A. There are more games which will be added in the future, but they are not yet converted to .ctg file type.

Notes:

  1. For the naming, I used a similar format to No-Intro to keep the clean appearance. This is similar to what we did with Intellivision (which turned out so nicely btw, thank you). The name format is different from the TI-99/4A names which are used on Atari-Age, this is on purpose to keep the names consistent to No-Intro

  2. The file names are different on three titles. The titles QBert, MASH, and Simon Says! all have illegal characters in the names. To not introduce any file errors, I replaced with dashes on the first two files and removed the ! on the last. If possible for display names however, it would be nice to have the illegal characters entered instead of using the nerfed file names. So if it is possible, let's go for it!

  3. On TGDB, I have all of the artwork created and am about halfway done with the descriptions / details. I am working through that the metadata so don't be surprised if on some of these titles only the artwork is displayed for now.

Here is the download for the TI-99/4A Game-Hashes: https://drive.google.com/open?id=1Kn0cVSAcH-47xv5djfUklOyBZ5EXwAgFD7Cg4RNvG7M

Please let me know if there is anything I can do. I am sure there will be a few errors I will need to correct. I should have the metadata done on TGDB soon to match up with the existing artwork which I previously have created.

Thank you!

sselph commented 7 years ago

@stevetb Thanks I've added these.

ghost commented 7 years ago

@sselph

Could you please add these too please? https://drive.google.com/open?id=1jMyUI1Wwxlw592jJRSlzQ-9d4EuFWKKMQtAdRQCp314

sselph commented 7 years ago

@stevetb Thanks I've added these.

marcteale commented 7 years ago

Could you add these as well, please?

https://docs.google.com/spreadsheets/d/1sY7i_3wq8_zvbjUrwpJlKqzTb6HvzxFUUeo3pknLfqM

dvanthuyl commented 7 years ago

Bare Knuckle III (Japan) [En by Twilight v1.0].md,hash not found,d1dc85abe0889ff8e880edcaf908a6b195b75952

english translation patched version for this one http://thegamesdb.net/game/25471/

ghost commented 7 years ago

de475924bae6cbe7cabdc84ac4cb8aeec58f076e F-Zero (Europe).sfc http://thegamesdb.net/game/43448

4b82c70824922380a3193e282d7674a442bd6dd4 Firemen, The (Europe) (En,Fr,De).sfc http://thegamesdb.net/game/44784

cbgoodman commented 7 years ago

ef726908b18a5ead67b8024ef664cc708cdf402e is in the hash dataset; however, it is unlabeled. It's Final Fight CD (Sega CD) (U).bin.

kodeFant commented 7 years ago

Dear @sselph . Thanks for a great scraper.

Went through a bunch of games: https://drive.google.com/file/d/0B777-EeiCezlYUZzVWowdjBJdUE/view?usp=sharing

kaltinril commented 7 years ago

I created a separate discussion. Proposal to create an auto gathering option for the scraper.

https://github.com/sselph/scraper/issues/176

sselph commented 7 years ago

thanks everyone. I started adding all these but it is getting late and will continue tomorrow.

sselph commented 7 years ago

@marcteale Thanks! @dvanthuyl Thanks! ghost Thanks! Whoever you were @cbgoodman Thanks!

@larsparsfromage Thanks! There were a couple of games that didn't seem to line up

nes/Wizards & Warriors (U).zip NES 0305a626acdc6523ff168725c402c6f81ec5b106 26409 7
nes/Tecmo Cup - Soccer Game (U).zip NES cbdf18ccbce9edcf59b30d85281410b782fc8355 2133 7

The first the ID seems to be for a different game and the second the linked game seems to be slightly different than the one I have 2131. The hash is for the US version and I think the ID you gave is for an EU version. Also for neogeo it seemed like things were named in a non-standard way. I was lazy and just copied them in but you'll have to use -mame which you are probably aware of.

There were also a couple of Mega Drive vs Genesis. I made the choice for Multi-region games that include Genesis and Mega Drive to use the Genesis version if available since Genesis sold more consoles based on the very quick search I did. I really didn't care but had to make some choice so that is what it ended up being. If you use the ss source it can probably do better at handling getting MD images if you use -region="eu,wor,us,jp,fr,xx" or something like that to prefer the EU region over the US region.

ericsnis commented 6 years ago

Thanks for the scraper tool. The scraper caught most of the ROMs but here are a few from the Super Nintendo which didn't get picked up.

Game Error Hash ID
Megaman X.smc hash not found e8921e243394b03382c03a6a08054f490c8f3dc8 143
Super Metroid.SMC hash not found 676c6610617d75bf90cea41d1af9f1ac23a36223 299
X Men Arcades Revenge.smc hash not found 168483c973aad51c482ed7e5b052310164ebc7c2 5871
animedude07 commented 6 years ago

Sorry to be that guy, but I found a handful of GBA titles that the scraper wouldn't find. Also, at the bottom of the list, one title that doesn't have a GBD ID, I suspect because it's a ROM rip from a GameCube disc that's been made playable on N64 (The Legend of Zelda: Ocarina of Time Master Quest). Not entirely sure what to do about that, so suggestions are welcome.

https://drive.google.com/file/d/1g6ASmnkAMXmvHW1RN-veo4sizVpofpkL/view?usp=sharing

Paul-Colucci commented 5 years ago

How do you generate the hashes? Is it MD5 or something? i don't see docs anywhere

marcalrepoles commented 5 years ago

Need help with the scraper in my Retropie. I saw on another link that I should run the scraper folder by folder and then open the csv file and post here the hash and the ID of my problematic games. But I'm having too many roms with issues on scraper. May I still post the csv file with the hash here or there's something else to try?

The link I've found about the scraper problem is this https://github.com/sselph/scraper/wiki/Scraper-can%27t-find-a-game%3F

kfbest commented 4 years ago

Few missing hashes `

Game Error Hash GamesDB ID
atari2600/Pac-Man.a26 hash not found 9ee09268281dc9d31b761acb32ff5fa866f983aa 36243
atari2600/Adventure.a26 hash not found 4ffe36c5113305714e27c72d7f3abecc9b08a630 2570
nes/Final Fantasy III.nes hash not found b182b75e019afe751e872763f90d81de78f3090d 83
snes/Mega Man X 2.smc hash not found e3160744be80529152247379fb1bdbaa83569c38 1015

`

OfficePop commented 4 years ago

Some missing SNES hashes.

https://www.dropbox.com/s/2eykaiahp6cdekm/file.csv?dl=0

marcalrepoles commented 4 years ago

THANKS

Em quarta-feira, 22 de janeiro de 2020 06:57:14 GMT+1, Michael Benavidez <notifications@github.com> escreveu:  

Some missing SNES hashes.

https://www.dropbox.com/s/2eykaiahp6cdekm/file.csv?dl=0

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

ahmedmahm0ud commented 4 years ago

Hey @sselph , Thank u for all of your great work, i hope u can help me with my issue , i have some psx roms with exction .PBP and i can scrap these roms using emulationstation scraper but when i using your script it always tells hash not found , Not there are not .bin .cue , because i had use PSX2PSP to convert them .. i think if we have name search flag it will solve the problem

Sorry for my bad English, and thanks for your great work .

marcteale commented 4 years ago

With the exception of Micro Mages, all of the games in this list are ROM hacks from romhacking.net. Thanks!

https://www.dropbox.com/s/mhfqyjkeb6cfeeg/file.csv?dl=0

Wetzel402 commented 3 years ago

Hello @sselph,

I ran your scraper using the fastscraper.bat with screenscraper.fr as the source and had two missing hashes. Interestingly the file did not contain the hash until I downloaded and used your shasum.exe to get the hash and added it manually. The file can be found here.

Thank you for the best scraper on the internet!

Edit: Added some missing Atari Lynx ROMs here.

Edit: Missing GameBoy ROM hashes: https://github.com/Wetzel402/missing-scraper-hashes/blob/main/_gb_missing.csv

Edit: Missing GBA ROM hashes: https://github.com/Wetzel402/missing-scraper-hashes/blob/main/_gba_missing.csv

Edit: Missing GameBoy Color ROM hashes: https://github.com/Wetzel402/missing-scraper-hashes/blob/main/_gbc_missing.csv