jpd002 / Play-

Play! - PlayStation2 Emulator
http://purei.org
Other
2.04k stars 248 forks source link

List filtering, Cover Saving, Serial ID identification, On Demand Description #254

Closed Zer0xFF closed 8 years ago

Zer0xFF commented 8 years ago

hehe, I was hoping this would overwrite the other pull request.

anyway, this contains all of my push with added list filtering (it won't list an image(elfs too), unless they're valid).

1) Covers are saved (much faster loading on the next run and needs no connection) 2) using the serial no, compared against db, I can find name and id (thegamedb ID) using that ID I can request the cover from thegamedb, yielding more accurate results. (when serial is not matched, thus ID is missing, we'll still attempt to get cover using name) 3) save game description too.(this is done the 1st time the image is loaded, or preview opened) 4) parse thegamedb to sql and cross match for serials. 5)Clean the code a bit more, there is alot of redundant functions, since I've used/recycled some of @LoungeKatt and also made my own, the re-adapted each to work together, I just pushed this to show progress 6) filter invalid disk images.

Note: this won't compile before you finish adding DiskUtils GetDiskId, my implementation assumed the null method(can be changed if you decide with exception method, but if you look at it as it stands, it makes more sense returning null), I've also kept the I capital to match with function tryGetDiskId.

Also, you should return the serial as is without the ;1 (e.g SLES_XXX.XX), again all of that could be adjusted for, but since the db is this way, it's just more convenient (also milliseconds faster :P ).

There are still few things I'd like to do, but that would be for a future pull, 0) restructure the code (it works perfectly as is, but definitely could be better) 1) the current db will only match 1119 out of the 1600 (note each ID contains multiple serials for different rejions), I can match around an extra 250 with 90% accuracy, however, I'd like to them and the remaining manually, to make sure 100% of their accuracy. 2) place a cross-matched db, instead doing it on the phone, @LoungeKatt has done that but his implimenation is different than mine. 3) add cover clearing option. (@LoungeKatt has done that already) 4) allow user to choose own cover. 5) fuzzy search (won't really be needed once I'm done cross-matching the entire db)

AbandonedCart commented 8 years ago

According to your code if it is the wrong file extension (!IsLoadableExecutableFileName(images.getName())), but the game ID is null (NativeInterop.GetDiskId(images.toString()) == null) it will be removed from the list. Since that is done after the full list is already generated, it will run through an irrelevant search.

Was there something wrong with the method I had added yesterday (https://github.com/AbandonedCart/Play-/commit/cd310b93999d3b0c68024785a81dab498871011e)?

Zer0xFF commented 8 years ago

No, at the time, I didn't understand it (nor have i tested it), but it looks like a better way to do it, now since I've looked at it.

I'm assuming you mean after your new filtering method is called, my method is useless? in that case you're right. I've should have done this much lower, during the listing process, as you've done it, however, since you've done that, when you send in the pull request, you can remove it. it will never be the wrong extension, since its a filtered list, however we can't check elf for serial, thus they're skipped. (is this what you're asking or did I miss something).

LinkofHyrule89 commented 8 years ago

That's great that they can chose a custom image now I'm sure there aren't a lot of situations where this is needed but it's great for when it doesn't find an image or you want custom images for your whole library.

AbandonedCart commented 8 years ago

I was saying I don't follow the logic that you are checking for an image that can't be loaded, then that it does not have a game ID, then removing it. You should simply check that it can be loaded before adding it. I am assuming the intention was to catch homebrew.

Zer0xFF commented 8 years ago

@LinkofHyrule89 ya, with a complete db, I wouldn't image they'd need that option, but there is no harm in having it... just as well, if you recall that coverflow repo, it looked nice, so people might want to remove that PS2 banner and have a pure picture of just the cover character etc.

@LoungeKatt no, it skips homebrew ! and only tests disks images(if invalid they're removed).

LinkofHyrule89 commented 8 years ago

@Thunder07 there are a lot of emulators on WiiFlow that works similar so there are different image sets for games people have created. This being an example https://vgboxart.com/browse-covers/PS2/

Zer0xFF commented 8 years ago

Nice, some of them covers can sell games!!

AbandonedCart commented 8 years ago

catch homebrew = It was meant to prevent removing homebrew from the list when checking the game id

It's too bad they didn't standardize those covers. https://vgboxart.com/view/17559/kingdom-hearts-ii-cover/ looks great, but it can't be used as a cover image in this scenario without looking silly.

Zer0xFF commented 8 years ago

I'm already working with a complete list, thats why I'm removing instead of adding. as I said (and assuming your trying to say), I should have done it earlier as the list is being made instead of after it's complete, as you have done.

Edit: oh, I can be an idiot sometimes...

AbandonedCart commented 8 years ago

https://github.com/AbandonedCart/Play-/commit/baaf829ad5098c8b1a2c190412b2f98cb6fc03f5

That should cover what you are trying to accomplish. It is the method I was using with the added check for valid ELF files. You can always cherry-pick that if you want to make sure it is part of this change. I know a lot of people were having issues with picking that first file.

Zer0xFF commented 8 years ago

done thanks.

LinkofHyrule89 commented 8 years ago

So are you still waiting for the Game Id check to be finished or did LoungeKatt end up doing that?

AbandonedCart commented 8 years ago

@Thunder07 and I both came up with ways to use the native function, but it was a question of whether that would be the final native function or not.

Zer0xFF commented 8 years ago

@jpd002 perfect, I've seen your reply to my other pull at which this should be ready.

jpd002 commented 8 years ago

I've merged the diskid branch in master. The getDiskId function will return null if no disk ID is found. I forgot to make sure the actual id returned is in the "SLES_XXX.XX" format, but I'll do that shortly.

AbandonedCart commented 8 years ago

https://github.com/AbandonedCart/Play-/commit/0d116c4bff6c5b63dca49713f05cb98a10bbeb1a

jpd002 commented 8 years ago

@LoungeKatt Thanks!

@Thunder07 I've tested your changes. I had to rename GetDiskId to getDiskId in getGameDetails.java. I would prefer keeping getDiskId as it more consistent with Java's function naming convention. Everything else works fine. If everything's good with you, I can merge that PR in master.

Zer0xFF commented 8 years ago

@jpd002 that's fine with me and ya as far as this pull is concerned I'm happy enough with it.

jpd002 commented 8 years ago

@Thunder07 Thanks a bunch!

LinkofHyrule89 commented 8 years ago

Good job guys going to test it right now!

Zer0xFF commented 8 years ago

Anytime :)

LinkofHyrule89 commented 8 years ago

So for some reason Marvel vs Capcom 2 isn't pulling the box art unless it's named correctly even though in the logs it shows that it pulled the game id. "mvsp2.iso [SLUS_204.86]" another game Odin Sphere it pulls the art just fine and shows this in the log "SLUS_215.77.Odin Sphere (USA) (En,Ja).iso [SLUS_215.77]" I even renamed it simply OS.iso and it still worked just fine. MvsC2 runs just fine so it's not corrupted or anything.

Zer0xFF commented 8 years ago

As I mentioned, the list is not complete yet, just because it gets the serial, doesn't mean it's getting the the id or cover.

It counts 1120 of 1600 total games, the missing games are matched through name.

LinkofHyrule89 commented 8 years ago

Oh ok well I guess that makes sense is there an easy way to add missing games to the list?

Zer0xFF commented 8 years ago

Make sure your iso is named properly. Check TheGamesDB for the game, if it doesn't exist add it.

LinkofHyrule89 commented 8 years ago

I guess it just doesn't like the way my games are named I used the PS2 game manager to name them and it uses the naming convention GameID.Game Name.iso So like the name is SLUS_204.86.Marvel vs. Capcom 2.iso but it only pulls the box art if it's simply named Marvel vs Capcom 2.iso

AbandonedCart commented 8 years ago

You should write down your entire setup somewhere. There are always those few people that do things a bit different, but so far it has been pretty consistent that you are in that group. It would make it easier to account for the variation in code, or at least present it with that in mind. In this case, most people put the ID in []'s at the end so that is all that was verified. All that's needed is a check for a XXXX_XX.XX prefix.

Zer0xFF commented 8 years ago

I'll say it will be easier to add a reporting system for missing serials than trying to account for naming variations.

AbandonedCart commented 8 years ago

Well now, yeah. This would have been pre-db so it would be there for now

LinkofHyrule89 commented 8 years ago

So basically the reason all my files are named like that is because the OPL Manager as well as the program that the OPL manager is based on names them like that so that they can be read by the PS2 game loader so I would imagine it's the most common format for people's collects of ripped games. If you have an iso that is named in another format it marks it as a bad name and renames it. http://psx-scene.com/forums/f150/opl-manager-tool-manage-your-games-121516/ http://psx-scene.com/forums/f150/opl-cfg-editor-v2-2-a-120657/ The OPL manager has it's own online data base with all the serials box art (both sides), disk art, and backgrounds that it uses the serials to download.

As for missing serials I definitely recommend some sort of system to add games to the data base easily.

Zer0xFF commented 8 years ago

ya it was easy enough to find http://oplmanager.no-ip.info/site/index.php, to bad the cover resolution is really low, its kind useless, it would have probably made our life easier. I actually forgot about OPL, though it was my only way to play games after my dvd reader burned out.

LinkofHyrule89 commented 8 years ago

Yeah that's the bad thing about it all kind of sad really. But might be useful for serials idk.

Zer0xFF commented 8 years ago

nothing much we can do with serials, we already have what's probably is a complete list.. we just need an associated cover.

LinkofHyrule89 commented 8 years ago

Is the database on the git or how can I look at it?

Zer0xFF commented 8 years ago

This contains Serials & Names https://github.com/jpd002/Play-/pull/254/files#diff-8a9e891561633987a7f65187efbe3498 there is another one on your phone /data/data/com.virtual*/databases/thegame.db that has been cross matched, containing ID (TheGamesDB ID), Serial, Name, Description & Cover link you'd need an sqlite explorer/browser/reader to view them.

AbandonedCart commented 8 years ago

For archival purposes, I uploaded their site source and database to my server. Hopefully some good will come out of being able to mess around with it. Maybe a better idea of how they do the search to improve results or something.

LinkofHyrule89 commented 8 years ago

I'm kind of surprised Marvel vs Capcom 2 doesn't have it's id in the database lol. How do I add it to the database?

Zer0xFF commented 8 years ago

I've only added exact matches to the DB, any name even with slight variation haven't been added yet (e.g if one DB had "Marvel vs Capcom 2" and the other had "Marvel vs. Capcom 2", they wont match, just because of that single dot).

Though I've also been able to match such small variations, certain games are being miss-matched because of it... (e.g fifa 2k4 vs fifa 2k5 would match since there is only a single character difference), as such I've given those some 95% accuracy, but I've held them back so I can manually verify them. That won't take all that much (maybe a full day behind the PC) but by the end of it, I plan on having matched all possible games, and added those still missing to thegamesdb database.

As for your question... adding the game to the db right now is useless, since even this will be given an overhaul, but if you feel like helping, I can provide you with the non matched DB (the 300 remaining title). you need to go through, verifying correct name to serial and verify if it exist on thegamesdb (if it exist, you need to get the ID (thegamesdb's ID) and add it back to the db, if it doesn't you need to add it to thegamesdb that will also involve googling for the cover/description, get the ID and add it back). This is what I plan to do myself, but after my exams.

LinkofHyrule89 commented 8 years ago

I was just meaning the serial is missing, the actual game is listed. I bet if we made a Google Doc of the remaining games needed to have their serials added that the community would fill it out in a couple days.

Zer0xFF commented 8 years ago

that's not a bad idea, I can probably have a list up in 5 mins https://docs.google.com/spreadsheets/d/19rnwFDNgw1QeDO2lDnQyIw4_ncJsqXQW52DeH_wvqUs/edit?usp=sharing the list

Zer0xFF commented 8 years ago

@LoungeKatt their GetGamesList.php is better than their GetGame.php API because of their elastic search, I'll probably just switch the name search to it... look at this

http://thegamesdb.net/api/GetGamesList.php?platform=sony+playstation+2&name=SLUS_204.86.Marvel%20vs.%20Capcom%202.iso

although, then, we'll have to preform 2 searches, one with the name to get the ID then a 2nd to get cover and description. Edit: or just update their GetGame.php API to use the elastic search

AbandonedCart commented 8 years ago

I don't follow. What do you mean by you'll switch the name search to it or just update their GetGame.php? Is this going to be running on a dedicated server for Play! or was there some sort of unadvertised takeover?

Zer0xFF commented 8 years ago

ha? no, we're currently using GetGame.php which isn't ideal at best, (god of war, final fantasy x returning ffxii etc), but their GetGamesList.php API uses a different search method https://github.com/TheGamesDB/TheGamesDB/issues/141#issuecomment-92118368 so if we switch from using GetGame.php to GetGamesList.php API the results will most likely be more accurate. however the GetGamesList.php API doesn't return description or cover link, thus we'd need to make a 2nd request with the newly acquired ID (from GetGamesList.php API ) to GetGame.php to get them.

but we can also just update their GetGame.php to use this elastic search, avoiding the hassle of making a 2 request.

Edit: aka the whole point of their github account is for people to help them out with things like this.

AbandonedCart commented 8 years ago

Are you asking that I rewrite GetGame.php or saying you plan to and explaining the reason?

Zer0xFF commented 8 years ago

Bit of both, you said you downloaded their site to mess around, so this is something you can try.

I would do it myself, but as I've mentioned before I've got exams coming up, so I'll hold off until after... At which if you still haven't taking a shot at it, I will.

LinkofHyrule89 commented 8 years ago

@Thunder07 good job with the list hopefully people will full it in.

AbandonedCart commented 8 years ago

Got the elastic search covered in https://github.com/jpd002/Play-/pull/257 but it is using the double search for now. The good news is that it is at least quick because it only pulls the id the first time and goes right into an id-based search.

I also added a secondary check to see if the game was saved under a different region. It will check the primary region first, then cycle through them.

LinkofHyrule89 commented 8 years ago

I've made pretty good progress on this list in the Google Doc using several sources I listed on the post on G+ still quite a bit left but right now I'm on M (Although I did a bunch after M already before I got the new sources).

LinkofHyrule89 commented 8 years ago

@LoungeKatt found this on my travels http://spiffycovers.com/covers/ps2/