search

Universal-Rom-Tools / Universal-XML-Scraper

Scraper de Rom
195 stars 46 forks source link

Games cannot be found (Amiga?) #43

Closed verybadsoldier closed 7 years ago

verybadsoldier commented 7 years ago

I just scraped my Amiga collection and lots of games could not be matched by UXS. One example is "Apidya (Team 17)".

All of my disks are from the Gamebase collection and I added both Apidya disks to one ZIP named "Apidya (Team 17).zip". It seems that this ZIP cannot be matched. I understand that the hash cannot match since I zipped it myself, but also the disks inside do not match.

Apidya (Team 17)_Disk1.adf MD5: 52A959E7FC99D483750EF3C1D67234B8

Apidya (Team 17)_Disk2.adf MD5: 804EC6893B4A14CF507E0954FDA3EEA6

This is the content of _Amiga_missing.txt (I only scraped this one game to find out whats the problem): Apidya (Team 17).zip 320BD17B 1 517 229 22:04.00 2016-11-11 90F0D3A5ADCBD7C343F443F9CEAF647D

I think the Gamebase romset is quite popular but is it possible that screenscraper does not cover it? Also matching by filename seemed to not work in this case. When I search for "Apidya (Team 17)" manually then I get no results also.

Any ideas? Thanks!

verybadsoldier commented 7 years ago

If GameBase amiga really is not covered I could offer to generate a list of all files with hashes for it. I did not find a way to add New roms on the website myself though. To be honest the website being mostly in french makes it a bit hard. I think you could reach far more users in the world with an english website and english Forum.

paradadf commented 7 years ago

You can change the website to be in english directly on the first page :S

About your scraping problem... could you unzip your self-created files and scrape it, please? The generated missing.txt can then be used to add your games.

verybadsoldier commented 7 years ago

You can change the website to be in english directly on the first page :S

Yes I know, but even if I change it to english then still most of the page stays french. Also I cannot participate on the forums cause it seems exclusively french :( I can live with it but I'm really afraid that you guys miss a lot of users from all over the world with this. While your page is really fantastic!

I'll provide the txt soon!

verybadsoldier commented 7 years ago

Ok, here is the set of ADF files from Gamebase 2.1. For some reason it only matched a single file.

Gamebase Amiga 2.1 ADFs.zip

verybadsoldier commented 7 years ago

...and this is Gamebase 2.0 SPS files: Gamebase Amiga 2.10 SPSs.zip

Universal-Rom-Tools commented 7 years ago

I'll pass that to neogeronimo ;) our Amiga/Atari Expert ;)

verybadsoldier commented 7 years ago

Thanks alot!

paradadf commented 7 years ago

I see a "problem" with some games there... Their names are cut. Did you scrape them with the latest version? @Universal-Rom-Tools didn't you fix that already?

verybadsoldier commented 7 years ago

My version might have been like 3 or 4 days old. I can do it again if that was too old. Currently UXS cannot be started though #48

paradadf commented 7 years ago

The best way IMO is to use http://ffulgore.free.fr/site/f-crc/us/ But I don't know if your roms were already added or not. Anyway, that is something Screech should look into.

verybadsoldier commented 7 years ago

Hm, that did not work either... That prog always crashes after 2500 files (from 5000)...

I tried "HasyMyFiles" and it works. But it produces output like this:

==================================================
Filename          : 'Allo 'Allo! Cartoon Fun!_Disk1.adf
MD5               : 83ea10af7853cf0bb6e64c31cec66ca3
SHA1              : bd95ec1a5c38ada6ce0b6c48aa9980920bc0f9b6
CRC32             : 7b253653
SHA-256           : 82636d6a3a85bf0e1c6752d323227801d0bd9b1ee3696b5c4234c0c1b2a6f3cd
SHA-512           : 6baa74f28188004f883fbc0fe9803f4f900164607b6122309be75b7c13d9e80862ce0c8390e5114b1720ada72fe7a973ee7afdd8e1ef254fa2f2e4be9961cb6c
SHA-384           : 6cf40086ec78eb8a5637d1ecc27f93c3ad078b07dc5aaa0552901de1df6e5e04ec68731bec99f174c86158c812bc5548
Full Path         : E:\amiga\ADF_Ex\'Allo 'Allo! Cartoon Fun!_Disk1.adf
Modified Time     : 
Created Time      : 24.11.2016 18:22:14
File Size         : 901.120
File Version      : 
Product Version   : 
Identical         : 
Extension         : adf
File Attributes   : A
==================================================

==================================================
Filename          : 'Allo 'Allo! Cartoon Fun!_Disk2.adf
MD5               : baee76443ecc6d34abb482e2a5218bf6
SHA1              : 776072153d3b3b68a04797c329548d039331b713
CRC32             : d91926fc
SHA-256           : cc78d0e2647cc5d24966ffb197e3a269f73e7415ef4355b4a85e671eabfa7782
SHA-512           : 684b728ba7aa9af101c93e6b39f6329592932077fbcd213875ae443004a6ca33361b6233bd369b00b2386067e6e9e393bd7f40184edc0492ea4cacde2ff0be04
SHA-384           : b5ca1a6b6c8be8b16a3c8f651c17f05c72e6cfc1aab0a3eea9a06b6fb6f991786e046f4d659982e2bffcb46f129ad06b
Full Path         : E:\amiga\ADF_Ex\'Allo 'Allo! Cartoon Fun!_Disk2.adf
Modified Time     : 
Created Time      : 24.11.2016 18:22:14
File Size         : 901.120
File Version      : 
Product Version   : 
Identical         : 
Extension         : adf
File Attributes   : A
==================================================

I can do it in that format if you can process it?

Universal-Rom-Tools commented 7 years ago

I need to work on that probleme of name file being 'truncated'.

I just created an issue about it... Certainly for the tomorrow release ;) (trying to not do more than 1 release by day ^^)

verybadsoldier commented 7 years ago

So, me again :)

I hashed all ADF disks from the Gamebase Amiga 2.1 collection and all SPS disks from 2.0 (they say there no additions in 2.1).

Would be great if you could take a look at it and add them to the DB. I hashed them with F-CRC but it crashed on some ADF files so I added them manually.

Amiga Gamebase 2.1 Hashes.zip

verybadsoldier commented 7 years ago

@Universal-Rom-Tools Sorry for bothering, no hurry, but I thought maybe this was missed :)

neogeronimo commented 7 years ago

they are included and associated now :)

Universal-Rom-Tools commented 7 years ago

Thank you Neo

verybadsoldier commented 7 years ago

@neogeronimo I am really sorry but something went wrong on my end :( The ADF-Set I have posted was incomplete and was missing 83 roms. The SPS-Set was complete though.

Sorry again, but this is the really complete ADF-Set: GameBase Amiga ADF 2.1-proper.zip

Universal-Rom-Tools commented 7 years ago

I just added Them ;)

verybadsoldier commented 7 years ago

@Universal-Rom-Tools Thanks! But sorry to say, something went wrong with the importing regarding character encoding it seems. The text file (UTF-8) contained for example:

Astate - La Malédiction des Templiers.adf

on SS it shows up as:

Astate - La Malédiction des Templiers.adf

So I think the UTF-8 text file was wrongly imported into the DB as Latin-1 (ISO-8859-1).