Add missing games

[RobLoach]

@lollo78 ran a DB rebuild.

The games not scanned (CRC ! ScummVM ID) are: The Legend of Kyrandia Malcolm's Revenge BBAE99AA | kyra3 Sam & Max Hit the Road 049CB65E | samnmax-it Broken Sword The Shadow of the Templars E5C92D6A | sword1 Broken Sword II The Smoking Mirror CEE47EA9 | sword2 The Secret of Monkey Island F8F7BC27 | monkey-vga-it The Legend of Kyrandia A5D9D73B | kyra1-cd-it Simon the Sorcerer 1 29BBABC3 | simon1-cd-it Indiana Jones and the Last Crusade 582CBFF4 | indy3-vga-it Maniac Mansion D958F338 | maniac-v2-it Lands of Lore The Throne of Chaos 8D979B0B | lol-cd

We should make sure these are built into the DAT correctly.

[lollo78]

Hi RobLoach, like you taught me, inside my locally libretro-super folder I done ./libretro-fetch.sh retroarch and the git pull download the new commits (all OK).

After that, I deleted all the file inside rdb folder and runned ./libretro-build-database.sh All the *.rdb file are recreated, but my ScummVM.rdb is only 547 byte instead of 23,7 KB (like twinaphex upload).

Of course if I try to use it for scanning my collection it doesn't find nothing (it's broken). What I do wrong?

[RobLoach]

This DAT probably needs an update. I could go through it again soon. Thanks...

Kiv just tried this again and found monkey-vga-fr wasn't in. We may want to re-add the languges and graphics. Or not... Still not sure. It's a pretty ugly situation.

[RobLoach]

Don't have the language or the video type in the end. Just use monkey.

[i30817]

And the cd type. Or highrez type? Honestly, you need a better way to generate the dats and the files if it's going to be so incomplete. Have you considered making a tool that parses scummvm.ini and dumps a [game_id].scummvm file on the 'correct' (sometimes they aren't but that's scummvm problem) dirs that scummvm itself detected instead? Could get rid of all those line feed files on the dat too.

Not to mention the plenty of game engines that don't have the correct naming in the dat, just a single 'sci' for all sierra games for example, while scummvm already is naming them 'gk1-cd' and such. I understand that this is because RA scummvm core is very behind, but it feels wrong anyway. Especially if people are supposed to use upstream tables and the ini to figure it out.

[RobLoach]

I think ScummVM just supports launching using the game ID, and not the variant name. Feel free to check the command line params yourself at scummvm --help.

libretro-database-scummvm uses scummvm --list-games along with the extra games.json to build the index. I've tried launching with the cd/language variants myself, and they had no effect. Could still launch from the menu just fine with .scummvm in the specific game dir.

Feel free to try yourself, I couldn't find anything better that worked. Added notes to the ScummVM scanning workflow on the Lakka Wiki.

[i30817]

The reason they have variants is because two version can be launched from the same data often. For example, Ecoquest 2 DOS or Windows from the windows cd, LSL6 DOS or Windows etc.

I'll try to check out what scummvm --list-games does.

[i30817]

Yeah, scummvm has another cmd line switch 'list targets' that lists the games variants that are on scummvm.ini. ./scummvm -t

Then ./scummvm [target] starts it.

It's possible it's a new feature. I compiled scummvm with support for all engines.

To be clear, i think the workflow, when possible, should be:

1. user uses scummvm executable to scan games dir
2. user removes variants he doesn't care for
3. libretro tool can be applied to scummvm.ini to created [game_target_id].scummvm with the target (the actual variant name not the engine).
4. RA scans these.

This makes it easier for the user than creating a file on each of ≃ 130 games dirs (and growing) if they happen to have them all. More, it can be updated easily. While it doesn't really prevent doing it manually.

If you don't want to use the exact variant name but the engine (or whatever is happening now) that's fine for now, you can change it when you update the core to upstream, if you warn users i guess. Since that is probably needed for images for SCI and GOB and CINE and others games, it will happen anyway.

[i30817]

It's possible i'm missing something. The problem is that if you don't have a scummvm.ini, then scummvm won't launch a variant? That makes sense, but i really really suspect that you need to start up with the variant for full compatibility.

Have you considered putting in the full block of a game variant on the .scummvm files (except the path i guess) and modifying the scummvm core to read that info from the .scummvm file given? Probably too intrusive code-wise... but a file with just 'sci' is no good as a id either (this is giving me flashbacks to the dreamcast .gdi files from TOSEC, that are the best dump, but are worse than useless for ID because many files have the same crc32).

Anyway, the point about it being easier for the user to generate the .scummvm files on the right path from the scummvm.ini remains, regardless of the variants being used or not. Creating 130 files manually is no joke.

This is actually a general retroarch problem with the way various cores want 'extra' files to deal with multiple cd/floppy games too, and it's just as bad, but this case is a low hanging fruit because the 'usual' index file doesn't actually exist here and has to be created (unlike say, psx redump .cues). And creating a not unique crc 'id' file is worse than useless for the database as i'm sure you know.

[i30817]

@RobLoach , there is a interesting PR in scummvm right now:

https://github.com/scummvm/scummvm/pull/970

looks like it will be possible in the future to 'detect' a game given a dir from the commandline. This could obsolete the manual creation of the .scummvm files (or even the files themselves if the info can be exposed to the scanner).

Some encouragement to update to upstream if it gets accepted.

[RobLoach]

That's great news. You're right about .scummvm files, we could potentially switch back to the rom-based method through Gruby's DAT.

[tobiatesan]

Um, hi, I noticed you have referenced scummvm/scummvm#970

Is there any way we (I) could make life easier for you guys?

Thanks.

[RobLoach]

Thanks for getting it started! Your CLI addition with --auto-detect will make finding what game to run a lot easier. I haven't had the chance to test it out yet, but may have some time to do that today.

Is there anything you need to help get it pushed forwards?

[tobiatesan]

Thanks for getting it started! Your CLI addition with --auto-detect will make finding what game to run a lot easier.

Thank our GSOC student @yinsimei for that one :)

Is there anything you need to help get it pushed forwards?

I personally would appreciate it if you guys had any suggestions to offer re:the semantics to be implemented.

[i30817]

The only thing i can think of is to make sure the game version enumeration that RA gets back (on stdin right?) Is consistent.

I think that expecting the 'right' game to always be returned first and discarding the other is a expecting too much, I've had experience of the normal scummvm mass detection failing at filtering games and adding game 'versions' that wouldn't even boot alongside the right ones, so i expect the user will have to manually remove the non-working ones.

Still 100x better than telling the user to create and drop hundreds of files in random dirs.

[RobLoach]

There's a PR over in ScummVM that may help this: http://bit.ly/2eED0Yr

Updated the rom-based branch, which uses a file-based method, no longer needing the .scummvm files. Once that PR is merged, we could likely switch to using the files directly.

[i30817]

I saw your post on the the PR @RobLoach , and i think there is a bit of a mismatch. Scummvm detection doesn't do executables often because it doesn't need them, just the data files. Often there is a requirement to slightly change the file/folder structure or filenames too (happens often in cd games).

So the mismatch i see is that this PR expects to be given a (probably absolute) path and return a list of possible games, and you're trying to give it a 'file'.

Are you still thinking of the 'rom-based' detection method too much? Instead of passing files, whose crc is unneeded here, you pass paths and the scummvm core gives back: 'yes it is a game' or 'no it is not a game'.

Files or their crcs are not needed or expected by this detection model that is a blackbox to RA. This also means you can't expect to use a 'exec' as a anchor for metadata like the rest of RA and that might complicate things on your side, but expecting that exec to even exist is wrong (for example i keep all the scummvm games needed data - so no exec - i can separated out of their 'DOS' directories in the right file/folder structure and naming by using hardlinks).

I think this basically means that to do this 'right' you need another way to access metadata than a file crc; that is a mapping that goes from the strings the scummvm core will output for the game version to the metadata.

BTW, this 'string method' would also be useful for the dreamcast tosec set which is full of duplicate .gdi files. A mapping from filename-> metadata (so basically string->metadata) would make the metadata actually show correctly in that set.

[i30817]

After writing this out i think i see your main problem in introducing another method. The main RA code would have to change to deal with directories or files differently based on the core right?

If scanning dreamcast any my advice was followed, the dreamcast set would need to

1. scan as normal for the redump set
2. if nothing is found, scan as the TOSEC set differently with the filename->metadata mapping.

Same problem with scummvm

1. normal scans expect matching crcs
2. but scummvm core needs to be applied to dirs to see if they match.

But it's even worse if the scan is agnostic to the core:

1. All matched 'rom' files have to be crc'ed.
2. All dirs have to be passed to the scummvm core to check
3. All filenames have to be passed to the dreamcast TOSEC check just in case.

Honestly, it might be valuable to pair these better approaches to the other idea for RA for core based scanning based on user pointing the scanner to a dir and saying the core or method that that scan applies to.

I know i'd vastly prefer that method instead of scanning 100's of gigabytes that will be full of false positives and general slowdown if multiple types of scans need to be applied sequentially.

[RobLoach]

Here's how it would work for ScummVM:

1. RetroArch scans the games directory.
2. Detects unique data files in ScummVM game folders and adds it to the menu
3. When launching, RA would ask ScummVM to launch the data file
4. The ScummVM core would load the folder of the data file with --auto-detect, effectively launching the game

With this, we would no longer need the .scummvm files.

Dreamcast/TOSEC are another issue, this is just about ScummVM.

[i30817]

Dreamcast/TOSEC was just another motivating example to show that other methods of mapping the metadata than crc would be valuable besides scummvm.

If you insist in going the same way i think you might have problems with game directories which are 'ambiguous' to if they have 1 game or 2 games. I think Larry 6 cd was one of those cases with a highrez or lowrez version being loadable from the same dir, as well as Fable or Torin which had multiple language dirs.

In my expectation you'd have:

1. RetroArch passes the games directory to the scummvm scanner specialization
2. RetroArch gets back the game versions that scummvm thinks are on that dir and adds all of them to the menu
3. Retroarch uses scummvm [-p <dir>] --game <id> to launch game (if they have autodetect, this will be possible. In fact, i'm requesting it right now in the PR).
4. metadata needs to use a specialized mapping on the database that uses from scummvm_name string as primary key

Notice that launching the dir with 'auto-detect' is always wrong on games that have multiple versions in dir, and maintaining a list of the individual crcs of the games is part of what the PR is supposed to obsolete. I honestly think you're trying to fit a square peg in a round hole here.

[RobLoach]

Retroarch uses scummvm [-p <dir>] --game <id> to launch game (if they have autodetect, this will be possible. In fact, i'm requesting it right now in the PR).

The issue with this is that both RetroArch and ScummVM have no idea what the ID is. That's why the .scummvm files were there.

If you have a better solution, I'm all ears. --auto-detect brings us a whole lot closer to awesome. You're just bringing up hypothetical problems. Test out your claims, see what happens. I don't have Larry 6.

I honestly think you're trying to fit a square peg in a round hole here.

What is proposed here gets us a lot closer to what's needed. We have the capability of making any changes we want, and improving what's there. If you'd like to help out, then please get involved in the core's code, or testing the new feature. Right now, you are not helping.

[RobLoach]

Looks like for Larry 6, libretro-database-scummvm found the following unique roms:

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS)[!]"
    rom ( name "RESOURCE.000" size 5754790 crc e1b0ace7 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS/French)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS/French)[!]"
    rom ( name "RESOURCE.000" size 5776092 crc f83ccbd5 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS/German)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/DOS/German)[!]"
    rom ( name "RESOURCE.000" size 5773160 crc 6f19d651 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS)[!]"
    rom ( name "PATCHES\0.HEP" size 4786 crc 294ec80d md5 187f649b8043de244db794718ad6264a )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS/French)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS/French)[!]"
    rom ( name "RESOURCE.000" size 18538987 crc 2261a6b7 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS/German)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (CD/Hi-res/DOS/German)[!]"
    rom ( name "RESOURCE.000" size 18534274 crc c14354b7 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS)[!]"
    rom ( name "740.MSG" size 2425 crc ea22f1ee md5 e12d81b04998025cefa0ade05553bc7f )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/German)[!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/German)[!]"
    rom ( name "0.HEP" size 2338 crc 5cc62050 md5 e05a1360d2034f688d892677f25b4c5d )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Italian)[u]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Italian)[u]"
    rom ( name "0.MSG" size 12519 crc 04653e4b md5 8b330a6ab2f7b8e69fd27c2f11d22563 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Polish)[u][!]"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Polish)[u][!]"
    rom ( name "0.FON" size 2137 crc 58317f6d md5 c788fc51006648a00c5dcc4b8d3f89a9 )
)

game (
    name "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Spanish)"
    description "Leisure Suit Larry 6: Shape Up or Slip Out! (DOS/Spanish)"
    rom ( name "1242.HEP" size 526 crc f70b040a md5 0c9fd86e7ee23d8dd0d6c8eb7cdebc5f )
)

[i30817]

The issue with this is that both RetroArch and ScummVM have no idea what the ID is. That's why the .scummvm files were there.

Uh, i was assured on this post that at least scummvm does: https://github.com/scummvm/scummvm/pull/970#issuecomment-317373644

If it doesn't, complain. Heck i'll ask for clarification then.

[RobLoach]

That game arg is what the ScummVM libretro core uses currently, along with .scummvm files: https://github.com/libretro/scummvm/blob/master/backends/platform/libretro/libretro.cpp#L176

[RobLoach]

@i30817 It's pretty much a mess.

You are not being productive. Stop right now... I'll put together a branch to help test some of this stuff, in the mean time, we have to come to them with constructive feedback. They're making our lives easier, you should appreciate it.

[i30817]

Ok, got a response. Unfortunately, it clarified that variants can only be used as ids if they are added first. So it's pretty much a given: either retroarch keeps scummvm.ini on its config directory and adds the games on scan, or it can forget about launching variants.

And if it ignores variants, many times it will be launching the wrong game because the bare \<id> can only run the first.

edit: i'm just disappointed. It's not that i don't appreciate your work @RobLoach, but i think it's obvious that the current behavior has several limitations that can only be solved by a redesign that:

1. doesn't fight against scummvm need for a config file for variants.
2. abandons crc checksums for scummvm metadata.

The PR is valuable, because cmd line detection didn't exist before. But it's not nearly enough to adjust that to your code here. You may be able to 'recognize' all the variants (with lots of work looking at scummvm code to find the file crcs), but if launching them sometimes means you're launching the wrong one, that work is flawed.

I wouldn't be surprised if some games with multiple languages always launched english for instance.

Eh, i guess i don't really care too much about the 2) here actually, because the metadata crc works, even if you're doing extra work for info that scummvm --detect will get you. But, if i can not have to add files to recognize the games, that would be better already. And if 1) happens, even better.

[i30817]

Speaking of that, and since libretro-database doesn't have a issue report page; are chd (compressed file format for gdi), deterministic? If so, it would be possible to add their (TOSEC dreamcast set compressed to chd) crcs to the listing to 'recognize' another dreamcast format and workaround the duplicate crcs, since reicast is supposed to support the format?