Updated ScummVM core based on latest sources available if you want it

[diablodiab]

I've made a new updated version of the core based on the existing libretro wrapper and the newest ScummVM sources. There are also some adjustments in the approach that completely separates the wrapper from the original source, hopefully making it easier to keep the core in sync going forward.

If this is of interest you can find it here: https://github.com/diablodiab/scummvm

The libretro-wrapper is added in one commit on top of the base source.

You are more than welcome to incorporate or use it for inspiration towards the official core if you want to. Just wanted to share.

[hizzlekizzle]

I've been watching this issue since it opened, and I talked with @LibretroAdmin about it at the time (and since). As you mentioned, we don't have a dedicated maintainer for scummvm (either version), so it's really a matter of the devil we know vs the one we don't. To try and correct that, I tried building this one and had no problem building the snapshot it's based on, but it failed for me after a rebase to the latest master. That might be my own fault (i.e., something I'm doing wrong), but no one else is stepping up to investigate it further, so there it sits. We have limited bandwidth for dealing with this kind of stuff, so we need someone to pitch in there.

So, by all means, if someone can spin up a straightforward repo with this backend that compiles successfully with recent master on the major platforms (win/lin/mac/android/ios) that we can add our buildbot CI scripts to, we'll switch over to it (or at least add it as a 'current' option with the existing core as a datestamped snapshot for the "obsolete" platforms). This is probably an easier path to take than setting up a parallel buildbot infrastructure, adding to RetroArch the ability to federate updates, and then having everyone add that additional location.

@i30817 dealing with upstream projects is frequently a touchy subject in both directions, not just with us but across the open software world.

[dami8a]

Hello everyone, thank you very much for the effort updating this core! This is the list of games I tried so far and now they work:

• Bizarre Adventures of Woodruff and the Schnibble
• Escape from Monkey Island (very slow)
• Goblins 2
• Goblins 3
• Grim Fandango
• Last Express (buggy)
• Longest Journey (hd version not supported)
• Phantasmagoria: A Puzzle of Flesh
• Sanitarium
• Toonstruck

[LibretroAdmin]

Just give us a pointer as to whichever newer repo is out there that works with more games but is not yet available on every platform we support so far, and we'll see where we can go from there.

[i30817]

All 3d games are slow, since they're in software mode. Hell, all games turn up the fan because of that really.

BTW, myst III 'works' but can't move the cursor like upstream, for some reason; and 'wide screen mode' (more like 'cut off the borders and zoom in' mode i think) is broken. Versailles 1632 and a whole bunch of wintermute games work (the 2d ones), lots of ags games etc.

Mind you some engines are as broken as ever upstream, because they don't have a dedicated engine coder anymore (dragonsphere/return of the phantom are some where the engine is a terrible state).

[mrmatteastwood]

Just give us a pointer as to whichever newer repo is out there that works with more games but is not yet available on every platform we support so far, and we'll see where we can go from there.

@LibretroAdmin The latest builds by @diablodiab are on http://build.bot.nu/nightly, although be advised that the current Linux build from Aug 15th won't load.

@diablodiab Nothing happens when selecting it from the "Load Core" menu in RA, even when using RA's latest nightly build. Here's a log from RetroArch.

[INFO] RetroArch 1.10.3 (Git 532840c4b8)
[INFO] === Build =======================================
[INFO] CPU Model Name: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
[INFO] Capabilities:  MMX MMXEXT SSE SSE2 SSE3 SSE4 SSE4.2 AES AVX AVX2
[INFO] Built: Sep  3 2022
[INFO] Version: 1.10.3
[INFO] Git: 532840c4b8
[INFO] =================================================
[INFO] [Input]: Found input driver: "x".
[INFO] [Environ]: SET_PIXEL_FORMAT: RGB565.
[INFO] [Core]: Version of libretro API: 1, Compiled against API: 1
[INFO] [Audio]: Set audio input rate to: 48000.00 Hz.
[INFO] [Video]: Set video size to: fullscreen.
[ERROR] [Wayland]: Failed to connect to Wayland server.
[INFO] [Vulkan]: Vulkan dynamic library loaded.
[INFO] [Vulkan]: Found vulkan context: "vk_x".
[INFO] [Vulkan]: Detecting screen resolution: 5760x1440.
[INFO] [X/Vulkan]: Window manager is Mutter.
[INFO] [XINERAMA]: Xinerama version: 1.1.
[INFO] [XINERAMA]: Xinerama screens: 3.
[INFO] [X/Vulkan]: Using Xinerama on screen #0.
[INFO] [X/Vulkan]: X = 0, Y = 0, W = 2560, H = 1440.
[INFO] [X/Vulkan]: Using windowed fullscreen.
[INFO] [Vulkan]: Found GPU at index 0: "NVIDIA GeForce GTX 1080 Ti".
[INFO] [Vulkan]: Found GPU at index 1: "llvmpipe (LLVM 13.0.1, 256 bits)".
[INFO] [Vulkan]: Using GPU index 0.
[INFO] [Vulkan]: Using fences for WSI acquire.
[INFO] [Vulkan]: Using GPU: "NVIDIA GeForce GTX 1080 Ti".
[INFO] [Vulkan]: Queue family 0 supports 16 sub-queues.
[INFO] [Vulkan]: Got 3 swapchain images.
[INFO] [Vulkan]: Using resolution 2560x1440.
[INFO] [Vulkan]: Using RGB565 format.
[INFO] [Vulkan]: Loading stock shader.
[INFO] [udev]: Pad #0 (/dev/input/event5) supports force feedback.
[INFO] [udev]: Pad #0 (/dev/input/event5) supports 16 force feedback effects.
[INFO] [Joypad]: Found joypad driver: "udev".
[INFO] [DBus]: Suspended screensaver via DBus.
[INFO] [Video]: Found display server: "x11".
[INFO] [PulseAudio]: Requested 24576 bytes buffer, got 18432.
[INFO] [Display]: Found display driver: "vulkan".
[INFO] [SRAM]: SRAM will not be saved.
[INFO] [Playlist]: Loading history file: "/home/thebrightside/.config/retroarch/content_history.lpl".
[INFO] [Playlist]: Loading history file: "/home/thebrightside/.config/retroarch/content_music_history.lpl".
[INFO] [Playlist]: Loading history file: "/home/thebrightside/.config/retroarch/content_video_history.lpl".
[INFO] [Playlist]: Loading history file: "/home/thebrightside/.config/retroarch/content_image_history.lpl".
[INFO] [Playlist]: Loading favorites file: "/home/thebrightside/.config/retroarch/content_favorites.lpl".
[INFO] [PulseAudio]: Pausing.
[INFO] [X/Vulkan]: Resized fullscreen resolution to 2560x1440.
[ERROR] Failed to open libretro core: "/home/thebrightside/.config/retroarch/cores/scummvm_libretro.so"
[ERROR] Error(s): /home/thebrightside/.config/retroarch/cores/scummvm_libretro.so: undefined symbol: _ZN4Pink6Action11initPaletteEPNS_8DirectorE
[INFO] [PulseAudio]: Unpausing.
[INFO] [Config]: Saved new config to "/home/thebrightside/.config/retroarch/retroarch.cfg".
[INFO] [Core]: Content ran for a total of: 00 hours, 00 minutes, 00 seconds.
[INFO] [PulseAudio]: Pausing.
[INFO] [Core]: Unloading core..
[INFO] [Core]: Unloading core symbols..
[INFO] [XINERAMA]: Xinerama version: 1.1.
[INFO] [XINERAMA]: Xinerama screens: 3.
[INFO] [XINERAMA]: Saved monitor #0.
[INFO] [Video]: Does not have enough samples for monitor refresh rate estimation. Requires to run for at least 4096 frames.

[schmurtzm]

Just give us a pointer as to whichever newer repo is out there that works with more games but is not yet available on every platform we support so far, and we'll see where we can go from there.

@LibretroAdmin

Hi, the compatibility list has been greatly improved between version 2.1 of the official core and the version 2.7 DEV currently supported by the work of diablodiab/libretro-scummvm-backend. About 298 games in v2.1.1 vs 385 in v2.7).

For example you can run Grim Fandango on Miyoo Mini (a weak linux arm handheld) or Broken Sword 2.5 which are Windows games that other RA cores can't run at all. Other example "spiderman - sinister six" or "Bruce-Coville s My Teacher Is an Alien" which runs perfectly on ScummVM.

So without hesitation you can use the main branch of diablodiab/libretro-scummvm-backend to create a new official core which support stable 2.6 release and the current 2.7 dev version of ScummVM. Even if it looks strange to incorporate a multiple engines system in a multiple cores system😵 , this core deserve its update thanks to the possibilities that it offers in terms of compatibility and performances.

[hizzlekizzle]

This is the repo we should be using, right? https://github.com/diablodiab/libretro-scummvm-backend

It looks like they're missing some #include statements upstream and that's causing the builds to fail, but I've been successfully sed-ing them in in my test recipes pulling from latest upstream git: https://github.com/hunterk/libretro_builds/blob/master/.github/workflows/linaarch64_scummvm.yml#L40

[ner00]

So without hesitation you can use the main branch of https://github.com/diablodiab/scummvm to create a new official core which support stable 2.6 release and the current 2.7 dev version of ScummVM.

Shouldn't you be using the upstream instead? https://github.com/scummvm/scummvm

[schmurtzm]

This is the repo we should be using, right? diablodiab/libretro-scummvm-backend

It looks like they're missing some #include statements upstream and that's causing the builds to fail, but I've been successfully sed-ing them in in my test recipes pulling from latest upstream git: hunterk/libretro_builds@master/.github/workflows/linaarch64_scummvm.yml#L40

Yes that's right (I've edited my last message with the correct link to the backend)

Shouldn't you be using the upstream instead? scummvm/scummvm

My mistake on the repo link, sorry. Yes use the upstream. The current scummvm is currently very active with many many fixes and enhancements. It compiles well and works well with the diablodiab's backend on the very last 2.6 and 2.7dev builds (tested today). (even if diablodiab/scummvm has may be some interesting commits ahead, I don't know about that, @diablodiab could probably confirm 😉 )

[ner00]

Yes use the upstream. The current scummvm is currently very active with many many fixes and enhancements. (even if diablodiab/scummvm has may be some interesting commits ahead, I don't know about that)

Okay, just wanted to clarify because your initial suggestion lead to believe that https://github.com/diablodiab/scummvm was somehow the better fork compatibility-wise even though it is way behind upstream. I've not compiled in a few months so, I'm not sure if upstream changes might have cause the same build issues for Windows as the ones reported for Linux by @hizzlekizzle

[schmurtzm]

Okay, just wanted to clarify because your initial suggestion lead to believe that diablodiab/scummvm

It was a mistake on the repo link from myself, yes I was talking about diablodiab/libretro-scummvm-backend. Of course it deserve more recent compilations tests because I've compiled the last 2.6 and 2.7 only for linux but in the past comments here it seems to be OK with other platforms.

It's not the easiest core to manage due to the use of 2 different repo with a lot of options for including engines or not... On this repo https://github.com/StupidHoroscope/ScummVM-MiyooMini , the author has added the backend as a submodule, which seems handy. But I trust the RA team to find the most maintainable solution on this subject (I'm not an expert to evaluate to good way to do to maintain it 😉 )

[i30817]

It looks like they're missing some #include statements upstream and that's causing the builds to fail, but I've been successfully sed-ing them in in my test recipes pulling from latest upstream git: https://github.com/hunterk/libretro_builds/blob/master/.github/workflows/linaarch64_scummvm.yml#L40

The missing includes is because scummvm expects 'ports' to include the version of system includes they need themselves in a dedicated file (portdefs.h). For some reason, this only started failing a all engine build recently.

I added +#include +#include

but the backend upstream only added limits.h (although i'm sure a all engine build failed here without stddef.h).

[hizzlekizzle]

oh TIL. clever. I'll try adding them there instead of sed-ing into the actual files. FAKEDIT: oh, ok. weird. I'll look into it on my end, as well.

[i30817]

climits.h and limits.h difference is because scummvm is a c++ project, so you probably should use the c++ versions (or maybe not, if the file doesn't).

I don't actually know which is which, and if you should include the .h in the name or not. I guess just follow the file convention or what the original include (in another file than portdefs.h) used.

edit: the file that '#include portdefs.h' actually.

[Darknior]

Okay, just wanted to clarify because your initial suggestion lead to believe that diablodiab/scummvm

It was a mistake on the repo link from myself, yes I was talking about diablodiab/libretro-scummvm-backend. Of course it deserve more recent compilations tests because I've compiled the last 2.6 and 2.7 only for linux but in the past comments here it seems to be OK with other platforms.

It's not the easiest core to manage due to the use of 2 different repo with a lot of options for including engines or not... On this repo https://github.com/StupidHoroscope/ScummVM-MiyooMini , the author has added the backend as a submodule, which seems handy. But I trust the RA team to find the most maintainable solution on this subject (I'm not an expert to evaluate to good way to do to maintain it wink )

It is really cool to read that it is possible. But do you think i will become official one day and the official repo will be updated for every one ?? Scummvm is really an impressive engine and it is so bad to let it in an old version :( And if i want to compile a version updated with last ScummVM source code, what version do you think i must use ? Can you give me the GIT because it is to integrate in Batocera and replace the original git url Thanks a lot

[ToniBC]

Hello, I tried this new core version, but for example the Legend of Kyrandia games do not load, it asks for the kyra.dat file that is present in extras, with the path in the interface in extra and in the game options, but he keeps asking. I tried to put the latest nightly from kyra.dat, but nothing, I can't get these games to work and I guess others may have a similar error.

In the standalone 2.7.0 nightly they all work without problems.

This is normal? Do I need to make any more adjustments?

Thanks.

[schmurtzm]

it asks for the kyra.dat file that is present in extras

It works on my side, check that your "extrapath" in scummvm.ini really point in the path where your kyra.dat is and check if you kyra.dat is up to date.

[ToniBC]

The scummvm.ini file lists the extrapatch directory correctly.

If I download the core with the updater, version 2.1.1 and the data of that core, the game works without problems.

When updating the core to version 2.7.0git 224d516, which is the last one published, and the data from the same buildbot, the kyra.dat error continues to occur.

It seems to only work with the correct version of kyra.dat which is not the one in the extras zip.

Could someone put the link of the correct extras?

Version 2.1.1 with the extras of 2.7.0 gives the same error, so the problem is that version is incorrect, but we have already tried many others and none of them work.

I downloaded everything from here. http://build.bot.nu/

What was the site where you indicated to download it.

[i30817]

Forget it for a while, i'm building the core myself (along with the extra data, to keep it always up to date) and i just crashed the old core i had - a month old or so.

Unfortunately the gdb trace is useless, probably because i built it in release mode. Now i'm building in debug and with the recent upstream code.

BTW I still need a extra include in portdefs.h to build the engine for the game 'In cold Blood' in spite of my other reported missing include got fixed there. I wonder why it doesn't break for other people? edit: actually the fix was committed yesterday.

[i30817]

Ok, a debug build later, this is the segmentation fault in current master with Lok:

Thread 1 "retroarch" received signal SIGSEGV, Segmentation fault.
0x00007fffc6af1c83 in Kyra::KyraEngine_LoK::delay (this=0x55555a3fd7a0, amount=33, update=false, isMainLoop=false) at engines/kyra/engine/kyra_lok.cpp:584
584         ct = skipFlag() ? ct + _tickLength : _system->getMillis();
(gdb) bt
#0  0x00007fffc6af1c83 in Kyra::KyraEngine_LoK::delay (this=0x55555a3fd7a0, 
    amount=33, update=false, isMainLoop=false)
    at engines/kyra/engine/kyra_lok.cpp:584
#1  0x00007fffc6b242bc in Kyra::Screen::fadePalette (this=0x555558d9c000, 
    pal=..., delay=84, upFunc=0x0) at engines/kyra/graphics/screen.cpp:842
#2  0x00007fffc6b2401c in Kyra::Screen::fadeFromBlack (this=0x555558d9c000, 
    delay=84, upFunc=0x0) at engines/kyra/graphics/screen.cpp:803
#3  0x00007fffc6bc0981 in Kyra::KyraEngine_LoK::seq_introLogos (
    this=0x55555a3fd7a0) at engines/kyra/sequence/sequences_lok.cpp:176
#4  0x00007fffc6bc0147 in Kyra::KyraEngine_LoK::seq_intro (this=0x55555a3fd7a0)
    at engines/kyra/sequence/sequences_lok.cpp:113
#5  0x00007fffc6af0af5 in Kyra::KyraEngine_LoK::go (this=0x55555a3fd7a0)
    at engines/kyra/engine/kyra_lok.cpp:360
#6  0x00007fffc6af36e6 in Kyra::KyraEngine_v1::run (this=0x55555a3fd7a0)
    at ./engines/kyra/kyra_v1.h:205
#7  0x00007fffc53d2319 in runGame (plugin=0x55555a31c980, 
    enginePlugin=0x55555a021a30, system=..., debugLevels=...)
    at ../../../../base/main.cpp:318
#8  0x00007fffc53d40e3 in scummvm_main (argc=4, 
    argv=0x7fffcb3c0fa0 <retro_wrap_emulator()::argv>)
    at ../../../../base/main.cpp:619
#9  0x00007fffc53c7ac9 in retro_wrap_emulator ()
    at ../../../../backends/platform/libretro/libretro.cpp:95
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x00007fffc53d0d11 in co_init ()
    at ../../../../backends/platform/libretro/libretro-common/libco/amd64.c:121
#11 0x0000000000000000 in ?? ()
(gdb) q
A debugging session is active.

    Inferior 1 [process 127612] will be killed.

Quit anyway? (y or n) y

[i30817]

I guess either system is null or 'ct' whatever it has broken + operator or is null.

edit: no, either being null seems impossible, since _system->getMillis(); is used at the start of the function that fails already to initialize ct.

I think it's _tickLength that is null, since it's not initialized in the function.

Or anywhere as far as i can see (it's not in the h file for that cpp either). What's this...

edit: kyra_v1.h. Probably.

[i30817]

I was wrong, as usual:

Thread 1 "retroarch" received signal SIGSEGV, Segmentation fault.
0x00007fffc6af1c83 in Kyra::KyraEngine_LoK::delay (this=0x55555a0b5b80, amount=33, update=false, isMainLoop=false) at engines/kyra/engine/kyra_lok.cpp:584
584         ct = skipFlag() ? ct + _tickLength : _system->getMillis();
(gdb) p 'kyra_v1.h'::_tickLength
No symbol "_tickLength" in specified context.
(gdb) p 'kyra_v1.cpp'::_tickLength
No symbol "_tickLength" in specified context.
(gdb) p _tickLength
$1 = 16
(gdb) p ct
$2 = 0
(gdb) p _system
$3 = (OSystem *) 0x555556660dd8
(gdb) p _system->getMillis()
Cannot access memory at address 0x1e0

i suspect something is uninitialized on the libretro_os.cpp initBackend() function.

There is a virtual uint32 getMillis function there, but i'm not a c++ programmer so i can't tell you how virtual will matter and how the hell the program can't 'access memory' of a virtual function.

[i30817]

Think this was fixed meanwhile, i'm doing a regression test (for my own curiosity) to see if i can check which fixed it by going back the kyra engine commits.

edit: can't find the crashing commit again... This is annoying. Trying to build with all engines instead of just kyra now.

edit2: no difference. Almost like there was a build artifact (i do change the dat files on each build)...

[mrmatteastwood]

The scummvm.ini file lists the extrapatch directory correctly.

If I download the core with the updater, version 2.1.1 and the data of that core, the game works without problems.

When updating the core to version 2.7.0git 224d516, which is the last one published, and the data from the same buildbot, the kyra.dat error continues to occur.

It seems to only work with the correct version of kyra.dat which is not the one in the extras zip.

Could someone put the link of the correct extras?

Version 2.1.1 with the extras of 2.7.0 gives the same error, so the problem is that version is incorrect, but we have already tried many others and none of them work.

I downloaded everything from here. http://build.bot.nu/

What was the site where you indicated to download it.

Hey team, I'm running into similar problems on the latest stable RA build (1.12.0) on Linux Mint 21. I downloaded scummvm_libretro.so from here: https://github.com/hunterk/libretro_builds

...and the extras/themes (ScummVM.zip) from here: https://build.bot.nu/assets/system/

Problems:

1. Unable to set any theme for the ScummVM core (exactly the same as described here: https://github.com/diablodiab/scummvm/issues/9)
2. Severe audio stuttering and micro-freezing during the intro of Legend of Kyrandia 1

Questions/Observations Should I download the extras/themes from elsewhere? Issue #2 occurs regardless of what value I set for audio latency and whether I have the speed hack enabled or disabled.

EDIT: When selecting "Close Content" from RA's Quick Menu while in-game, at least during Kyrandia 1 (need to check others), RA completely freezes and needs to be killed from Linux.

[hizzlekizzle]

1. I was able to change the theme using the latest stuff from the upstream repo. I collected them here: https://github.com/hunterk/libretro_builds/raw/master/scummvm_system_files.zip
2. I got some sound stuttering, as well. Could suggest needing a larger audio buffer (here? https://github.com/diablodiab/libretro-scummvm-backend/blob/main/libretro.cpp#L474)

I did not get the freeze issue in Linux.

[ToniBC]

With that core build and those extras, the Kyra.dat no longer fails, I'll test it better tomorrow.

The audio problems @hizzlekizzle of which game is it?

[hizzlekizzle]

I was referring to mrmatteastwood's number 2. I didn't get it severely, but it definitely does some crackling there on my machine. I didn't check for framedrops or fast-forward headroom yet, though.

[mrmatteastwood]

1. I was able to change the theme using the latest stuff from the upstream repo. I collected them here: https://github.com/hunterk/libretro_builds/raw/master/scummvm_system_files.zip
2. I got some sound stuttering, as well. Could suggest needing a larger audio buffer (here? https://github.com/diablodiab/libretro-scummvm-backend/blob/main/libretro.cpp#L474)

I did not get the freeze issue in Linux.

I wrote a reply 2 days ago but just realized it's not here. I might have forgotten to send it? Odd. Anyway:

Yes, the theme works for me too with the link you posted. Thanks!

Re. the audio buffer, is this something you guys can take care of, or should I open an issue on @diablodiab's GitHub?

Here's a video showing the audio dropouts on my desktop PC, game starts at 0:45. They're worst during the first minute or two, but dropouts still occur later in the game as well, e.g. in the scene with Kallak and Malcolm. It's similar, though a bit worse on my laptop. They also occur when I turn off the fancy shader. You can see from the noise filter that the game actually micro-freezes.

https://youtu.be/Esje0_IlCAs

The freeze no longer occurs on my system either, so it must have been a one-time thing on my machine.

[hizzlekizzle]

It's probably worth mentioning to diablodiab, yeah. I'm not sure if it's actually a buffer issue, but that's the first thing that jumped out at me, and it's a hardcoded number, which may indicate that it's an arbitrarily selected number, based on a "should be good enough for everything" guestimation, in which case, a simple bump-up should fix it. That's all conjecture, though.

And yes, if it's actually freezing and not just crackling, that may be the wrong direction altogether and we should probably verify whether it's a simple frame-drop (for whatever reason; performance, etc.) or if it's actually blocking/stalling on something.

[mrmatteastwood]

Issue submitted. I hope @diablodiab is still around, it seems like he hasn't been active in quite a while, and the latest builds on his build bot are from August 15th...

[ToniBC]

Yes, the same thing happens to me with the sound. The problem is that I still don't understand why this core or variant is not in the buildbot. If we have variants of puae, Stella, Snes9x, bsnes, etc... I don't understand why we don't have a 2012 scuumvm (or whatever year) and a Current Scuumvm.

[hizzlekizzle]

this one builds differently. we need to work out how to integrate it with our existing CI infrastructure. That's part of why I created my github actions recipe.

None of our existing cores have to fetch upstream, fetch the backend, add the backend to the upstream code and then build.

[mrmatteastwood]

Just a quick note, I realized I had reported the sound dropout issue on the wrong repo. Copied it to the right one now (which is also where diablodiab is active :-)).

[diablodiab]

@BoardsCanada and Hunterk: I was told that you wanted to talk with me about getting the libretro-scummvm-backend on the buildbot. At the moment, I have very little time due to work related issues, so it's difficult to prioritize, but...

The work I did with restructuring the ScummVM libretro backend was always intended as a contribution to the official work on the libretro core, for you to adopt if you wanted it. I don't have time to maintain it myself at the moment, so the ideal scenario for me would be if you want to to create a new repo based on the current state of the libretro-backend and allow others to take over and make contributions and improvements here in a more official repo.

If this is what you wanted to talk about, then by all means please proceed :)

[LibretroAdmin]

Sure, make that repo, then transfer it over to me so I can move it to the libretro org. From there, I will give you admin access over it as well. So you won't lose anything in the process. And we'll have an easier time gathering contributors that can contribute to it.

[schmurtzm]

That's an awesome news ! :) Very happy to see that it will happen ! Last month I have worked on libretro ScummVM version for Miyoo Mini (a little linux handheld with low cpu) I've tested and compared the official core and the one from diablodiab. I can tell that ScummVM from Diablodiab is awesome : it make possible to play a lot of games in good condition and it will be a lot better for following the rythm of development of ScummVM (which is quite impressive).

During my exploration I've discussed with the ScummVM team to clarify some points on the way how the ScummVM core is working, they don't support the libretro version at all but they've given me some explanations. So I've discovered some interesting points that I'll expose quickly here for the future maintainer of the core. It's a little anticipated and it will deserve dedicated issues of course but I have to expose it 😄

Two essential points (which are related to the current libretro core and the future diablodiab's core) :

• The most important : the way how the scummvm core runs the games is wrong from the beginning 😅 . The shortcuts with .scummvm files runs a command line that will never use the configuration file (scummvm.ini) from scummvm as it should. For example the consequences are that if you run a game like grim Fandango, the flag which indicate that the game files have been checked will never be set and at each start it will check all the files (long process). Futhermore I emphasize the fact that creating .scummvm files is a complex and long task for the user (they have to check an external website with compatibility list). So it must change to use the powerful game recognition included in ScummVM. There is two easy steps to make it simple and more efficient :
  • use scummvm -p "$gamedir" --add --recursive to detect and add all the present games in the scummvm.ini file
  • use a new shortcut file which contains the name of the game section of the scummvm.ini.   There are numerous advantage to this method :
  • ScummVM works as it should by using its internal game recognition
  • The creation of the shortcuts and the import of games can be automated
  • The ScummVM main GUI (called launcher) can be used with this configuration, it allows to configure game settings   All the information and details about this are in this issue : https://github.com/diablodiab/libretro-scummvm-backend/issues/24, we already use the new core in this way with our Onion's community and the users are really happy of the way it works because it transform the experience of adding games for the user in something very easier than the initial way.    
• The second point is about the audio stack which is not as good than the standalone version at all. On low powered devices you'll have quickly sound stuttering on games like Broken Sword 2, Gim, Blade Runnner.... I've made a workaround here which improve a lot the result and allows to play the previous mentioned games on the Miyoo Mini (which is an achievement!) but I think that the core deserve a new sound stack which is a bit beyond my area of expertise 😅 . An issue about this is here.

For now the core from diablodiab can be used as is because it already improve a lot the game compatibility compared to the current libretro core but if in the future these 2 points can be taken into account it will be awesome. I'm fully available to help if you need more details, you can find me here or on Retroarch Discord and ScummVM Discord). Looking forward to see diablodiab's core available for everyone 😀 !

[i30817]

For the previous post i use this program to do it for me after adding all the stuff in the core manually (from starting the core from the core menu): https://github.com/i30817/libretro-mkscumm

I don't like to add recursively without checking because the scummvm detector is not perfect and will introduce 'bogus' games even with all the data files correctly named and positioned (which sometimes needs changes from the original game files). Especially if you use the mass add function, which is the only real way not to spend a lot of time.

Then after removing all the buggy entries, and using my program with those games added, the ids inside the .scummvm files that are created are the correct ones instead of the 'generic' ones that the libretro readme mentions, have the real game name as understood by scummvm, as far as possible (forbidden windows characters removed) and also the playlist(s) have correct names for the games (and i can make other playlists for the same core easily too, based on the filepaths), because the label json property is what is used in RA, not the filename.

Since i've also been adding the games i have to the libretro-thumbnails scummvm project, the thumbnails mostly appear, and most of those that don't can be adapted with my other project https://github.com/i30817/libretrofuzz

Although there are some engines where i'm not adding thumbnails yet (AGS, some others). More people contributing would eventually close this gap.

Personally speaking the most annoying thing about these cores (either) is the lack of hardware acceleration. Running any game on these cores compared to running them on upstream will noticeably turn up a laptop fan, if yours has a fan. Much worse if the game has 3d elements. That may be one of the causes of the 'audio stuttering' even.

[schmurtzm]

For the previous post i use this program to do it for me after adding all the stuff in the core manually (from starting the core from the core menu): i30817/libretro-mkscumm

I don't like to add recursively without checking because the scummvm detector is not perfect and will introduce 'bogus' games even with all the data files correctly named and positioned (which sometimes needs changes from the original game files). Especially if you use the mass add function, which is the only real way not to spend a lot of time.

Then after using my program with those games added, the ids inside the .scummvm files are the correct ones instead of the 'generic' ones that the libretro readme mentions, and also the playlist(s) have correct names for the games (and i can make other playlists for the same core easily too, based on the filepaths).

Generally, i'm unsatisfied with how the scummvm core handles adding games for another reason: it requires a additional file in the game directory.

Not only is this a bad idea if your games are zipped or something, it's also completely unnecessary now that the playlist fileformat is json. The 'id' of the section in the scummvm.ini file could be in the playlist as a new filed in the game entry, and the path pointing to a directory instead of the .scummvm file.

I don't know if this has to be implemented by libretro or the core, but it should be to finally get rid of the need for .scummvm, if not for the need to add manually to the scummvm.ini file, which as mentioned, is necessary for correct function.

Doing it this way exclusively would screw up the manual or automatic scanner, so i understand why it wasn't implemented, but i'd like the opinion, for projects like mine that aren't completely generic and actually use the scummvm.ini file as it's supposed to be used.

I agree on this point : the creation of additional files shouldn't exist : to list games in the frontend ideally it should read the scummvm.ini file. But it's not the way how the frontend work (and it is the same for all retrogaming software) : they need a file with a specific extension to launch a game.

From my point of view the scummvm detector must be use by the users : this is efficient and easy way. If you see buggy entry it deserve an issue on the ScummVM repo to be fixed and not a workaround by making it manually. This is the way to improve it. Obviously it is constraining but we are facing a special case (ScummVM is not an emulator and it is a multi-egnines software embedded in a multi core software 🤪 ).

The most important here is not how to add the games, you can do that as you want from a command line or from the Launcher UI of ScummVM BUT after that the game must be launched in a way to use this entry of the scummvm.ini file. And it's not the case of the .scummvm shortcut files for now.

[i30817]

I removed that part of the comment (and added some others).

The reason i removed it is that the files actually have a 'advantage' and it doesn't actually cost me much to add them and use that method.

The advantage is the the RA scanner and manual scanner understands them. I know 'you can always create them again' and that's what i do, but if you delete/move a game and don't want to go through the annoyance of using libretro-scummvm-playlist again, you can just use the playlist management to 'rescan' a playlist base directory and it will mostly work (the names will be screwed, remember my program eliminates forbidden windows characters from .scummvm filenames).

So although i think the files are a bad design, it doesn't actually affect my workflow that much and has at least 'some' utility for now.

Doing the files manually though, that's horrible and error prone because of the ids need to match scummvm.ini ids to work 'correctly', especially if you have a near complete or complete scummvm collection. That's the main reason i created libretro-scummvm-playlist anyway (you can install it with pip right now).

[schmurtzm]

Stop to edit your messages, I see it always changing when I write an answer 😅 We have to stay focus on the core here, you are mainly talking about the frontend and the playlists, on my side I talk about how the core is launched currently ;)

[mrmatteastwood]

Just wanna chime in real quick to say BOOYAH and KUDOS to you guys for being on top of this!

An up-to-date ScummVM is at the very top of my wishlist for RetroArch and though I have no programming experience, I'd like to contribute in any way I can, e.g., by testing on Windows and Linux (Mint), or by editing the documentation for ScummVM once the new core is ready. I did the whole doc for DOXBox Pure with @schellingb. » https://docs.libretro.com/library/dosbox_pure/

@schmurtzm mentioned audio issues, let me add that Kyrandia 1 is having a weird stuttering issue where it seems like not only the audio stutters, but the whole game micro-freezes. I have a comprehensive bug report about this here:

https://github.com/diablodiab/libretro-scummvm-backend/issues/27

It happens on both my PCs on Linux Mint and on Windows (on the laptop only, no Windows on the desktop). Issue does not occur with the ScummVM core currently present in the RA repo (2.1).

[i30817]

They're basically one and the same. The playlists point to the (invented) scummvm files to have a 'id' to launch the game.

The main problem you're complaining about is that the users creating the scummvm files inevitably use the wrong ids because they're either not adding the games through the scummvm GUI or because they just don't care and don't use the ids that are in the scummvm.ini file. That program solves this issue if you use it, creating those files for you and a playlist pointing to them after you the the GUI mass add.

For RA to do the same without a external program, it would be ... problematic.

For the first issue, 'creating the scummvm.ini file' it would essentially have to proxy the 'scanner' to not use normal methods when you start scanning a directory for that core, but call the core functions. Not sure this will ever fly. But users can do it manually if they're directed to do so.

For creating the 'scummvm id files' (or the alternative of another json property in the playlist) it would have to hook up when a game is added to the scummvm.ini file (not a easy thing to do without adding code to the core or some non-portable file watching). Maybe when the core closes the scummvm.ini could be scanned to build a playlist 'automatically'.

I mean, it's possible, but i kind of doubt it because the information would be going from 'core->frontend' when usually the api is the opposite.

And to clarify i mean 'I doubt it would be accepted in the project', not that it's impossible. Making the playlist creating being dependent on the core changing scummvm.ini would essentially mean 'giving up' on the scanner for the scummvm core and making the core dictate the playlist creation, so even if it's 'the least error prone way', I'll believe it when i see a PR merged.

[schmurtzm]

You continue to edit your messages all the time :grimacing:

They're basically one and the same. The playlists point to the (invented) scummvm files to have a 'id' to launch the game.

No, you misunderstand. Please read my issue carrefully 😉 I 've just explained that the game should not be launched with his ID like this For example : The right internal command line to run Grim is : scummvm grim-win and with .scummvm file you will obtain scummvm -p SCUMMVMpath/GRIM grim:grim which is wrong due to the points that I've explained.

[i30817]

That's because you're creating your files wrong, aka you're one of those users that is doing something wrong.

Here is my grim fandango file based on the scummvm.ini which is created by the core after i add the game in the scummvm GUI:

i3@sleipnir:/media/i3/Mordred/Games$ cat ScummVM/grim_fandango/Grim\ Fandango\ \(Windows_English\).scummvm 
grim-win

[i30817]

Besides to launch a game from retroarch manually you must do this:

i3@sleipnir:/media/i3/Mordred/Games/ScummVM/grim_fandango$ retroarch -L scummvm Grim\ Fandango\ \(Windows_English\).scummvm 

retroarch -L corename file

If you try it like that, grim fandango will 'verify' the files the first time, but not the second, which is what you were complaining about.

I have no clue from where you think 'grim:grim' comes, or why would you try to start scummvm instead of retroarch to use a retroarch core.

[schmurtzm]

You're not really aware of how ScummVM work currently... and this conversation must be done outside this issue. Please read the scummvm documentation to understand from where grim:grim comes. Then you can come on discord to discuss with me and to understand each other well before interfering here more 😅 You can also read my conversation with ScummVM team from here. I'm not complaining of anything but just give some advice to the future maintener from my experience.

[i30817]

I read it now, and i'm telling you that you that that kind of start initializer will not work in this core.

Only names which exist in the scummvm.ini files are (or rather should be) accepted by the core, not the 'engine:game' variant.

What's the point with that anyway? You can't automate that and without the scummvm.ini entries you're not even sure if you're choosing the right 'game' right-side name (entries that already have the engine anyway).

I guess i'm asking 'why would you even want this'. Are you trying to eliminate the need for the scummvm.ini?

[schmurtzm]

Only names which exist in the scummvm.ini files are accepted by the core, not the 'engine:game' variant.

You're wrong one more time ! If we talk about the diablodiab's core you have to specify 'engine:game' or game like "Indiana Jones and the Fate of Atlantis" will not run. Read this issue to understand. But anyway using the game ID it is not the recommended way at all. Please stop interfering here, come talk to me on discord (I don't find you), I'm sure that we have a lot to share here once we will be agree on these points ;)

[i30817]

That issue is complete nonsense. I can start fate of atlantis fine with a .scummvm file and if you want a video i'll post it in your issue. But even if we ignore your assertion that the scummvm file doesn't work, you're then complaining that 'engine:game' breaks (a unsupported way to use the core anyway).

This is not surprising! Because ':' is a illegal windows character in command lines (it's the path separator in windows). You have to quote it.

[schmurtzm]

That issue is complete nonsense. I can start fate of atlantis fine with a .scummvm file and if you want a video i'll post it in your issue. But even if we ignore your assertion that the scummvm file doesn't work, you're then complaining that 'engine:game' breaks (a unsupported way to use the core anyway).

This is not surprising! Because ':' is a illegal windows character in command lines (it's the path separator in windows).

😞 That's because your core is not up to date. With an old core there wasn't doublon with atlantis. I understand that you are very invested with your projects on playlists but I will not answer here anymore, it breaks the initial discussion. Come talk to me , we will expose our point of view when we will be agree.