Updated ScummVM core based on latest sources available if you want it

[diablodiab]

I've made a new updated version of the core based on the existing libretro wrapper and the newest ScummVM sources. There are also some adjustments in the approach that completely separates the wrapper from the original source, hopefully making it easier to keep the core in sync going forward.

If this is of interest you can find it here: https://github.com/diablodiab/scummvm

The libretro-wrapper is added in one commit on top of the base source.

You are more than welcome to incorporate or use it for inspiration towards the official core if you want to. Just wanted to share.

[DrUm78]

But what version of the core are you using? Did you compile one 2.5.0 or are you using 2.1.0 with 2.5.0 engine data (that would still be 2.1.0 code though)?

[triggerthehorizon]

I was using 2.1.0 before with the 2.1.0 engine data. Now I am using 2.6.0 GIT compiled by diablodiab with the 2.5.0 engine data. I haven't been able to run Hopkins FBI through RA android in scummvm since forever, but that same game data works on scummvm standalone in windows and android, and libretro core on windows..

[triggerthehorizon]

Here it is working perfectly fine on Retroarch windows. Same game files and same scummvm extras folder, and it won't work o android...

[DrUm78]

So I don't known why it does not work for you, hard to say without any more info. Another developer may help you.

[triggerthehorizon]

So I don't known why it does not work for you, hard to say without any more info. Another developer may help you.

Thanks for trying to help me, I appreciate it! I also have a French version of Hopkins FBI that works on windows and scummvm standalone on android, yet it crashes in RetroArch ScummVM. This leads me to think it is really the core and not the game files.

Are you using 32-bit version of RetroArch for android by any chance? I am using the 64 bit. Maybe that's the difference?

[DrUm78]

Nope I use a custom 32-bit frontend called SDLRetro (lightweight Libretro frontend) for the FunKey S, totally a different scenario here but yes the issue might come from that difference.

[triggerthehorizon]

@diablodiab Is there any way to identify what is causing Hopkins FBI to crash RetroArch 64 when using the ScummVM core?

[diablodiab]

There seems to be a timing issue with the Hopkins FBI engine on Android. I've applied a workaround for Android that might fix it. Try downloading the Android core again in the link I provided higher up in the thread and see if it works.

[triggerthehorizon]

There seems to be a timing issue with the Hopkins FBI engine on Android. I've applied a workaround for Android that might fix it. Try downloading the Android core again in the link I provided higher up in the thread and see if it works.

You fixed it!!! Thank you so much. Interesting fact, I just realized who you are and I've had your website in my bookmarks for a a while now :) https://lindqvist.synology.me/wordpress/

So will this build be used to update the libretro corewithin retroarch anytime soon?

[mrmatteastwood]

So will this build be used to update the libretro corewithin retroarch anytime soon?

God I hope so. @diablodiab's work is so freaking awesome. With the libretro team going through all cores atm to do QA for Project IO. I hope when they get to ScummVM, they'll implement this version.

[Gamesirvictim]

Could you help me fix this problem on Android. I am trying to open KQ 3 AGS version. by the way it is working on the scummvm stand alone emulator but not RetroArch.

[diablodiab]

Try applying the "fixags.patch" and see if it solves it. If I remember correctly, the issue is related to compiler optimization. It's in the patches folder in my repo.

[triggerthehorizon]

There is also this bug : https://forums.libretro.com/t/scummvm-problem/22828 Where a bunch of games crash after a scene with a fade out. Do you know if this can be fixed?

[Darknior]

Really excellent, i really wish it will be made officially. For now i have stop to use the libretro port, so many new games are not working and some problems .... No AGS support, no 3D games support ... so bad :(

[Gamesirvictim]

I appreciate your help . The problem is I don't know where to find the "fixags.patch" I searched for it but never seen it and even if I found it how do I use it with the core .

[diablodiab]

I assumed you were compiling the core yourself - it's a patch provided with the source code that you can apply before compiling to fix the issue.

Anyway, you can download a compiled version here instead: http://build.bot.nu/nightly/android/latest/

[Gamesirvictim]

I assumed you were compiling the core yourself - it's a patch provided with the source code that you can apply before compiling to fix the issue.

Anyway, you can download a compiled version here instead: http://build.bot.nu/nightly/android/latest/

It working now. TY man I wish you always be happy. I won't forget your help.

[triggerthehorizon]

So should the rest of us update to this version as well?

[diablodiab]

You can give it a try, but I recommend that you keep a backup of the core you were using in case you find that something is broken with the newer versions. I've set up a buildbot environment for myself at http://build.bot.nu which pulls the newest ScummVM sources once per day, applies the libretro backend, and compiles for a number of systems.

[triggerthehorizon]

Thanks diablodiab! Do you know if these builds address this bug? https://forums.libretro.com/t/scummvm-problem/22828

[Gamesirvictim]

You can give it a try, but I recommend that you keep a backup of the core you were using in case you find that something is broken with the newer versions. I've set up a buildbot environment for myself at http://build.bot.nu which pulls the newest ScummVM sources once per day, applies the libretro backend, and compiles for a number of systems.

Great work . The last linked you gave me fixed my problem . I tried this new link and also it is working like the last one . Keep the good work I am a new follower of your work . 👍

[mrmatteastwood]

You can give it a try, but I recommend that you keep a backup of the core you were using in case you find that something is broken with the newer versions. I've set up a buildbot environment for myself at http://build.bot.nu which pulls the newest ScummVM sources once per day, applies the libretro backend, and compiles for a number of systems.

This is freaking awesome. Thanks so much for all your work!!

[ToniBC]

I guess later after testing it will either go to normal RA or the core will be updated. Because no user will see this and it makes little sense.

[inactive123]

OK, so can a PR be sent to this repo to update it? Or how will we proceed? There is a better way to go about it?

[diablodiab]

The isolated libretro backend can be found here: https://github.com/diablodiab/libretro-scummvm-backend

You could choose to make a new fork of ScummVM base, apply the backend files, and then let that be the starting point of a new updated ScummVM core.

[i30817]

This core seems to have inherited some problems from libretro.

If you try to run Gobliiins (gob1 in the scummvm file) you'll get a dialog from scummvm saying it "Error running game: Game data not found" (same thing with 'gob1cd' in the scummvm file).

Then when the core fallbacks to the scummvm launcher you can add the game and you'll (or at least I in my version of the game) will be prompted as to what version of the game you have.

I chose 'Gobliiins CD (v1.02/DOS/English (US)) and that will start fine.

Even after adding it to the menu though, it won't run from the retroarch playlist, which indicates that maybe the 'simple name' is not enough for this game? Am i supposed to put the language in front of 'gob1' somehow in the scummvm file?

Another bug it inherited is gobliins 2 having problems starting:

https://github.com/libretro/scummvm/issues/168

instead of closing or crashing here though, it's a black screen forever.

[mrmatteastwood]

This core seems to have inherited some problems from libretro.

If you try to run Gobliiins (gob1 in the scummvm file) you'll get a dialog from scummvm saying it "Error running game: Game data not found".

Then when the core fallbacks to the scummvm launcher you can add the game and you'll (or at least I in my version of the game) will be prompted as to what version of the game you have.

I chose 'Gobliiins CD (v1.02/DOS/English (US)) and that will start fine.

Even after adding it to the menu though, it won't run from the retroarch playlist, which indicates that maybe the 'simple name' is not enough for this game? Am i supposed to put the language in front of 'gob1' somehow in the scummvm file?

Another bug it inherited is gobliins 2 having problems starting:

168

instead of closing or crashing here though, it's a black screen forever.

I can confirm this for Goblins 1. Getting the same error using ScummVM 2.6.0git (Dec 4 2021, 02:12:52) when attempting to run the game from the RetroArch playlist. When running it from ScummVM's own launcher, it works.

On my machine, Goblins 2 behaves exactly the same way: "Error running game: Game data not found" when attempting to run from playlist, but works when running from ScummVM launcher.

No such issue with Goblins Quest 3, fwiw.

Note: I got all 3 games from GOG, if that helps any.

[i30817]

Turns out i was supposed 'gob1cd-us' for gobliiins 1 (in the .scummvm file that is). I fear i'll have a lot of these bugs.

[mrmatteastwood]

Turns out i was supposed 'gob1cd-us' for gobliiins 1 (in the .scummvm file that is). I fear i'll have a lot of these bugs.

A-ha! Same here!

Now, how to figure out what to put into the .scummvm file for Gobliins 2?

[i30817]

Gobliins2 can also be started. However i wasn't able to find a 'short name' that added to the scummvm file 'launched it by itself'. I was checking the scummvm.ini file after adding the game in the core launcher for clues and tried 'gob2', 'gob2cd', 'gob2cd-us' and 'gob2cd-en' and they all failed more or less. Launching from the core launcher works. Super annoying, since the core launcher is practically unusable in retroarch.

edit: actually gob2cd works i had a little confusion with my copy on write drive.

[i30817]

Baaasically the core is unfriendly in that the scummvm files 'have' to have a corresponding entry in the 'scummvm.ini' file. So you have to 'mass add' all of the games (taking care to eliminate false positives or languages you don't want) to find out the 'real names' of the launcher. And you have to keep the scummvm ini file because those 'real names' refer to entries there, they don't (apparently) exist by themselves in the core.

Annoying.

Though fortunately this doesn't appear to apply to the AGS engine (thousands of games...) and even more fortunately i keep those games separate from the main scummvm directory so i can 'mass add' that and not the ags games.

[i30817]

I think it's time for some regex a script and file mangling. Mass add, remove the few false positives and wrong long names that scummvm has, then process the file to find the enginenames then 'find' .scummvm files in the game dir, then output the 'entry name' to that file. That would work for the ags collection too without more effort (more or less).

So annoying. The core should be able to do this, just read the ini file instead of reading the .scummvm file and finding the directory that corresponds to the scummvm file directory. You'd have problems if you added multiple languages or the game itself supports other 'different versions' and you added them but not otherwise.

edit: in fact it could also add .scummvm files for each entry regardless of language by using the 'long name' as the name of a new NAME.scummvm file.

[Gamesirvictim]

Here is the game name.

[i30817]

I guessed. I just don't want to copy 3000 of them into files to be able to launch them from a retroarch playlist.

[Gamesirvictim]

If I have more than 10 games I would use scummvm stand alone emulator. Why using RetroArch and make my life harder.

[i30817]

I'm also sort of worried about the scummvm.ini file potentially screwing up the 'portable playlists' functionality.

I'm not sure it matters but the fact is if you're using portable playlists the path of your games can change, so the fact that scummvm.ini has the path on the ini file as a hardcoded absolute path makes me wonder if it would stop working once the user moved the games dir or tried it on another platform. It may be true or it may be false since you actually can run scummvm games with a empty scummvm.ini, it may be that retroarch is always passing the current path of the .scummvm file and only the 'core launcher' would be affected by these issues.

Anyway this script process shouldn't be affected if you do it while 'the paths exist in the right place'.

Also if you want to run on both windows and linux, uh, run this on windows so it forbids all windows forbidden characters and not just the linux ones. Or change the script to that other scheme i mentioned, using the directory name as filename of the .scummvm file. However you'd have to guard against multiple 'games' launched from the same dir (the blade runner engine for instance has the 'normal' game and a 'blade runner with restored content' added for a single directory mass scan, goblins adds multiple entries per language, etc.

[i30817]

I opened this bug to see if the utf8 forward slash can be part of the 'turns into _ for thumbnail match' set so i can have the complete scummvm description with (CD/DOS/English) instead of (CD_DOS_English) in linux.

https://github.com/libretro/RetroArch/issues/13496

[mrmatteastwood]

Dude, hats off to you! I'd be way out of my depth with what you just did, so mad respect. I think you're working on a very worthwhile feature, perhaps there really is a way to make that a core functionality? @diablodiab hint hint.

Agreed about libretro-thumbnails, I found it difficult to match games I have in my playlist to thumbnails, and in many cases, I just eneded up getting the box art from the web and manually putting it into my RetroArch folders on my PC using the game names from my playlist. I think a lot of the thumbs in the thumbnails repo don't match the game titles in the ScummVM database.

[i30817]

Agreed about libretro-thumbnails, I found it difficult to match games I have in my playlist to thumbnails, and in many cases, I just eneded up getting the box art from the web and manually putting it into my RetroArch folders on my PC using the game names from my playlist. I think a lot of the thumbs in the thumbnails repo don't match the game titles in the ScummVM database.

@mrmatteastwood , this reminded me that it's actually possible to download the thumbnail files... and then rename them. It would be darkly hilarious if that's the best way to get images 'automatically' with minimal change of error.

'Just' get the real name from scummvm.ini (edit: or playlists) find the highest similar name from the github api, download that ahead of time and rename it to the actual filename (without the forbidden characters anyway, though it might not matter if the file is already local, depending on when the string is transformed, before checking the cache, or before downloading to the cache).

It's a pity this process can't be delayed until retroarch asks without modifying retroarch code or/and turning this into a two step process that modifies playlists with a extra field (and more ra code to 'use this field if it exists to download the thumbnail, because our database is broken', which i doubt would fly)

[i30817]

Opened this one, cross fingers. https://github.com/libretro/RetroArch/issues/13499

I feel it would help a lot of the less popular or fragmented into too many original dump sets thumbnail databases.

[mrmatteastwood]

Nice. This would be an awesome new feature! Fingers crossed they implement it.

EDIT: Noticed typos in the issue title: "Thumbnail Fuzzi-matching Filenmaes" - it would be "fuzzy" and "filenames".

[i30817]

New script. I generalized everything and thumbnail download and the .scummvm file creator script separate again.

#! /usr/bin/env python3

import os
import re
from pathlib import Path

#00-1f are ascii control codes, rest is 'normal' illegal windows filename chars according to powershell
forbidden   =   r'[\u0022\u003c\u003e\u007c\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008' + \
                r'\u0009\u000a\u000b\u000c\u000d\u000e\u000f\u0010\u0011\u0012\u0013\u0014\u0015' + \
                r'\u0016\u0017\u0018\u0019\u001a\u001b\u001c\u001d\u001e\u001f\u003a\u002a\u003f\u005c\u002f\u0026]' 

with open('scummvm.ini') as f:
    text = f.read()

pattern = re.compile(r'gameid=(.*)\n(?:.*\n)*?description=(.*)(?:.*\n)*?path=(.*)')

for m in re.finditer(pattern, text):

    s = m.group(2)

    if os.name == 'posix': #linux
        s = s.replace('/', '_')
    else:
        s = re.sub(forbidden, '_', s) #windows and everything else

    s = s + '.scummvm' #final filename

    p = Path(m.group(3))
    if p.is_dir():
        p = Path(p, s)
        with open(p, 'w') as f:
            f.write(m.group(1))

print('To download the thumbnails, please scan the new .scummvm files with retroarch manual scanner into playlist(s).\n'
    'Then install the thumbnail download script \'pip3 install git+https://github.com/i30817/libretrofuzz.git\'.\n'
    'Then run it with \'libretrofuzz\' and chose the playlist then ScummVM directory to download the thumbnails.')

That's it. Just download the other program with pip like the final output says, and you have a generic program to download thumbnails based on playlists. Then use that parameter (there are many). It's not perfect or anything so if you have better ideas about the normalizer or the scorer please open a bug or PR.

I generalized this when i realized i was doing the same program for different playlists with slightly different approaches because of the set characteristics. For instance for a non no-intro amiga WHDLOAD set which has terrible names like MonkeyIsland2_v1.3_0020, i do libretrofuzz --no-subtitle --rmspaces --before '_' which removes subtitles (for remotes, in the normalizer), spaces and cuts off all substrings after the first '_' for reasons that should be obvious from the name.

[i30817]

I was testing my games collection with this, and versailles 1685 is crashing with a segmentation fault.

Originally it was crashing because i screwed up the setup of game files and it warned of a new variant, but after deleting everything and recreating it started getting further (and it works on standalone ofc).

Then it crashed with segmentation fault. Vulkan seems to crash slightly faster but gl is a difference of half a second and showing a logo for a few frames.

User picked target 'versailles-win' (engine ID 'cryomni3d', game ID 'versailles')...
   Looking for a plugin supporting this target... Cryo Omni3D
Segmentation fault (core dumped)

Either dos or windows version it doesn't matter - it's one of those hybrid dos-windows discs that the scummvm wiki warns about. I have the extra files (and yes, they're the ones from the fork, not from the download from the retroarch extra files menu).

[i30817]

Another thing i noticed is that on Escape from Monkey Island and grim fandango is that the retropad doesn't quite work until you configure the shortcuts in scummvm keymap, and even after that it seems to 'forget' when you close the game and reload. Or something like that i think i managed to make it forget in EfMI too without that but i forget how it happened of if i wasn't imagining things.

[i30817]

Heads up, i opened a PR with my images for the names scummvm produces for the games i have, which are a more or less clear majority with some omissions (ega versions, ags, children's games and some of the more obscure games).

Hope it gets merged. Although the fuzzy matcher program works for a lot of images, it can't work if the game image is not there at all or is too different.

[i30817]

Ok i said it was the last script. I lied:

https://github.com/i30817/libretro-mkscumm

Made a real program like the other program to download the covers.

The advantage of this version is that you no longer need to 'scan' after running it, it creates the playlist by itself. And also, it has the 'real names' in the labels (which retroarch uses to display) so the '_' is no longer a problem.

So you only need to add the games to the scummvm core with 'mass add' from the core, then run this and you'll have a scummvm playlist.

To install use pip3 install --upgrade git+https://github.com/i30817/libretrofuzz.git (the --upgrade switch is just so it keeps downloading if 'you already have it' because pip is kind of dumb and makes install not upgrade files if they're already installed).

then you run it with libretro-mkscumm

--help or the page shows options for if you have more than 1 'scummvm' playlists because you want to divide, say, the AGS games from the rest or something, you can use --filter somepath --playlist someplaylist.lpl to only scummvm.ini that start with the filter to a someplaylist.

In windows you have to provide the path the retroarch.cfg file, like the other program. This is because I don't know the default install path in windows.

[i30817]

Ugh. The versailles bug i mentioned only appears to occur with the nvidia driver, with the intel driver it seems to work. edit: spoke too soon, now when i retry it crashes. I don't even...

Also i had a major bug on the libretro-mkscumm with the regex which was causing the contents of the .scummvm files to be wrong, just upgraded the version. Should be the last version of the script, knock on wood.

[mrmatteastwood]

@i30817 Just wanted to say kudos and hats off for all your hard work, man! I personally won't need these programs/scripts for the small handful of SCUMMVM games I have, but damn this is impressive.

Really hoping the RetroArch team takes note and integrates both your work and @diablodiab's brand-spanking-new ScummVM core.

[i30817]

I released 1.0.0 and i don't expect more changes. The changes since the last post were:

1. handle the case where the game entries are not available (for instance, a external drive that is disconnected). In that case you can't actually create the '.scummvm' files but previously the playlist would be created without items, now they're still there and it asks to run again once the drive is connected.
2. I tested out what happens in a playlist with 'scan_content_dir' set to the empty string and that is a much better default than set to 'base_content_directory'. What happens is that the playlist refresh option is disabled, which is much better than having it enabled and nuking the 'filters if any used'. Just a obscure corner case.
3. There was a bug where i was using 'commonprefix' instead of 'commonpath' to find the candidate 'scan_content_dir'. This could, unlikely but could, end up with 'scan_content_dir' that was a dir that didn't actually exist, if the 'common prefix' of all the game directories ended up in the middle of a dirname or filename.

And code clarification/comments/refactoring of course.

[i30817]

I released 2.0.0. It's just a clearer readme and the ability to not have to setup the extra path in the scummvm core GUI (i don't download the files, the program checks if the dir exists, and is not empty and stops if it isn't there, and if it's there, it fills in the 'extrapath' in the scummvm.ini).

It's useful to have a sanity check, because these files are necessary for many many games to actually work in scummvm. The documentation also hints that you can use the 'core system files downloader' if you're using the normal core and to download it from upstream master if you're using this core (and points to the buildbot that was linked here too).