Updated ScummVM core based on latest sources available if you want it

[diablodiab]

I've made a new updated version of the core based on the existing libretro wrapper and the newest ScummVM sources. There are also some adjustments in the approach that completely separates the wrapper from the original source, hopefully making it easier to keep the core in sync going forward.

If this is of interest you can find it here: https://github.com/diablodiab/scummvm

The libretro-wrapper is added in one commit on top of the base source.

You are more than welcome to incorporate or use it for inspiration towards the official core if you want to. Just wanted to share.

[i30817]

That said @diablodiab , is it possible for the buildbot assets dir to have a zip of the dist/engine-data dir from upstream? Github doesn't have a sane way to download directories (i suspect it is on purpose), and the utilities for it on the web require to give up github authorization tokens, which is unsafe.

The only alternative is to download the master zip, which is huge.

[hizzlekizzle]

@i30817 you can see about adding a script here: https://github.com/libretro/libretro-system-files

I believe it's already being used for dolphin's Sys directory and ppsspp's assets, so I assume it would work here, as well.

[i30817]

I believe it already exists in retroarch master. However, it's downloading from the libretro fork, which is out of date to master, which @diablodiab buildbot tracks.

Some games would fail to start with the old files from that script and the nightly build, and some games would fail to start with the new files and the libretro core.

Makes sense to ask/contribute for a upstream version of the script i guess, if the naming of the result is explicit.

edit: opened a issue for that.

I'm worried that users will want to use both cores, because users will be users, and if they share a dir for the files... Though i don't see a alternative, maybe it's best to let them get burned like this. edit2: actually since the cores share the same name, this probably isn't that big of a problem. Downloading another to use will require to download and overwrite the core system files always.

[i30817]

A small update to setup the saves directory in scummvm.ini when creating the playlist.

This is necessary because the core breaks if the autosave is set to on (i set it to the retroarch.cfg save directory).

edit: and now that the scummvm.ini file was written to, the regex needed a update because of spaces between ' = '.

[i30817]

Phantasmagoria 1 and 2 is crashing in recent compiles. Not sure if the problem is from upstream or not (it worked before).

Also Swords of Xeen crashes when you start a new game with a assertion error, but that's from upstream.

Themes do not work, download from upstream or the libretro-scummvm repo.

Taking a screenshot says it takes it, but it never appears in the pictures folder in this core.

[i30817]

Ok i tested it by building myself with upstream scummvm (the latest).

Good news, is that everything works.

Bad news is that you really really need to build the scummvm.zip as this says: https://github.com/diablodiab/libretro-scummvm-backend#updating

Both should be consistent, same build of scummvm_libretro.so using the same data.

edit: apparently in between my posting this message and checking again the buildbot assets dir got a new folder, seems like that contains the data. So it's fixed now and the documentation of my utility now mentions that link for people using that core.

edit: has the data now, everything is fine now.

[i30817]

@diablodiab here are problems that i noticed that exist in the core - even with the latest scummvm code - that doesn't upstream (at least with your libretro-scummvm-backend ). I don't know if these bugs are for using upstream master for the core and testing with the release deb for scummvm, so take them with a grain of salt - they might just be normal breakage of upstream during the dev cycle.

One problem is that on some games, the game mouse cursor is corrupt with a blue 'box' of corruption (with transparency artifacts). This happens for instance in Myst Masterpiece edition or 11th hour in the first screen the cursors appear.

The engine that powers 7th guest/11th hour in the core 'fast forwards' videos without sound if you press escape. This would be interesting if it was a feature, but the .deb installed scummvm does no such thing, it just forces you to watch the video.

I wanted to see if the corrupt text in the 'skull main menu' of 11th hour existed in the release scummvm but that engine is not enabled yet, so no luck on that.

edit: actually i'll just try to build scummvm upstream and see if these still apply if you wait a little bit.

I also reported launcher 'mass add' bugs to upstream and reported a assertion error in swords of xeen (that one was fixed promptly). It's a pity the scummvm bug tracker is so bad to use, you can't even tell from the bugs page you have updates to your bugs without checking email.

[i30817]

Ok i tested it.

The cursor bug only happens in the core build - probably related to differences in the opengl tinygl.

The videos do fast forward in upstream, but the difference is that it's a bit slower (i think), and the music channel does not fastforward and still plays. What is disabled is voices. Which is interesting, because it appears to be a case where retroarch sound handling shoots itself in the foot (maybe different mixer capabilities?).

The text in the first screen of 11th hour is fine in upstream, so this is another graphical bug that is core only.

[i30817]

Now libretro-mkscummvm notices if the soundfont is in the extra path and adds its path to the ini file if missing and notices if there are m32 or CM32L roms in the retroarch system dir (because of dosbox for instance) and hard links them in the 'extra' dir too. It's a bit of streamlining, like the already existing paths to the saves and extra dirs into scummvm.ini. It doesn't touch the default devices. Or adds the themes directory either (scummvm modern looks kind of aliased here for reason).

edit: i added the themes directory after all, no reason not to.

[i30817]

I'd like to correct that the video problem i mentioned in the 11th hour was a false alarm (or maybe a bug) that i can't reproduce now, and upstream behaves the same now (no sound in live action intro movie).

Everything else is corrected by a patch in this bug report: https://github.com/diablodiab/libretro-scummvm-backend/issues/3 (with the corrections i mentioned, or whenever the author edits the text).

[i30817]

I did find a new problem. The hardware renderer (opengl or tinygl for some engines) is not available. This is very obvious on residual engines where playing grim fandango, EMI or the longest journey will kick up your cpu fan and still look worse than upstream (TLJ and grim fandango have those 'stars' bug around the 3D characters in certain perspectives that is common in software renderers and the antialiasing is worse, though i'm not the best at noticing those things).

edit; although, upstream has by default, a non-correct aspect ratio for the 3d games, unlike the core. Maybe some day upstream will update those engines to show more of the screen when it can be scrolled on those games instead of distorting the image, and default to black columns when it can't.

[mrmatteastwood]

Hey @diablodiab I have a version of this core from Dec 6, 2021, the version number is 2.6.0. By any chance, is there a kyra.dat for this version somewhere?

[i30817]

The data files are in the buildbot with the core, or you can build and install them yourself if you have the code (although that's a bit complicated).

[mrmatteastwood]

Nice, thanks! These seem to work with version 2.1.1 of ScummVM, which comes with RetroArch. However, as soon as I manually replace it with version 2.6.0 by @diablodiab, I can no longer run the Kyrandia games. ScummVM tells me the necessary kyra.dat is missing. :-/

[i30817]

If you're building, you should build that zip (and extract it) too. If you're using the cores from the buildbot core zip (in another directory there) you can use that ScummVM assets zip. The assets are constantly changing in upstream scummvm, so that's probably why it complains on one version and not the other. The core and the assets should match, or things might (and almost certainly) get broken (this also can happen to other cores, but it happens a lot in scummvm because that project is constantly externalizing new data and sometimes changing the representation of it).

Also you need to have it in scummvm.ini (on the retroarch system dir), but my tool at https://github.com/i30817/libretro-mkscumm takes care of that.

If you want to build it yourself, first i recommend using the 'backend' repository (read the readme for instructions) and upstream scummvm. I also add a patch that is on one of the bug reports to make myst and 11th hour cursors correct.

Then make clean, make, copy the core to the retroarch core dir, and create that assets zip by calling bundle_aux_data.bash. After you have the zip, you can extract it into the retroarch system/scummvm dir.

This is more complicated than what i'm willing to tolerate doing regularly, so i kinda use some automation for it ( I use a program called 'navi' and two custom scripts).

If you want to use this nonsense, first you have to install navi, then place the git-branch-pull script on the path somewhere (i use ~/.local/bin), and the navi.build file in ~/.local/share/navi/cheats. Then make the scummvm git directory be in ~/Documents/Projects/scummvm (because i use that directory) with the backend git inside it on the directory that is expected, and you'll be able to build it by calling navi and choosing 'scummvm core' (probably). If you want the patch to make 11th hour cursor not broken, which is not merged yet for some reason, you can apply it to a new local branch and the navi script will give you the option to chose it when you build.

Unfortunately, if you build it youself, for some not very clear reason, you need to make clean on every new build otherwise you might get runtime errors when trying to start the scummvm core / start a scummvm game. Some kind of make cache error i think. The navi.build file takes care of that, but should have not been needed (like it isn't for the retroarch build in that same file).

BTW, navi is a terrible 'language' but unfortunately it's the first i found that did the job. I suspect the competition is just as bad. Since navi kind of sucks, if one of the rules 'fails' (has no possibilities), it continues - I think this can only trigger if your internet connection is the kind that requires some prompting and fails the 'first time'. This usually means that the navi rules i have there will build, but will build the old version, because the git pull failed or something.

And that's the story of how i'm using upstream scummvm in retroarch without it being 37 thousand commits behind and without needing to create several hundreds of .scummvm marker files manually (with libretro-mkscummvm)

Mind you, you still don't have opengl acceleration in the scummvm core unfortunately - which is annoying because it's much less fan intensive.

[ner00]

There is also this bug : https://forums.libretro.com/t/scummvm-problem/22828 Where a bunch of games crash after a scene with a fade out. Do you know if this can be fixed?

It's a bummer because this and other bugs have been fixed in the official libretro core branch, but then this one still has that bug and other like crashing when loading Beneath a Steel Sky, but not crashing when logging is enabled which is kind of weird. If I use this core version I get extra engine support, but at the cost of losing other games like BASS which now crashes RetroArch.

[i30817]

I run BASS in retroarch fine (ie: there is no such crash although i didn't bother to wait for the whole intro movie to end and pressed esc to get ingame after seeing it start), and i'm running the diablodiab version. Make sure that the data files match the version of the core you're running (ie, if you're using the buildbot core, use the assets from the buildbot, there in another directory), and also make sure that the save directory is set on the scummvm.ini that retroarch is using (or use my tool to setup the scummvm playlist libretro-mkscummvm, that besides helping with the playlist, sets some essential scummvm.ini settings).

If you're building, things are slightly more complicated, as my previous post mentioned.

The save directory is needed because scummvm has autosaves, and has a bad reaction if it can't save them, besides, you know, not being able to save if you don't have it.

here have a terrible 900 kb video linked to see how it works here:

https://user-images.githubusercontent.com/757900/173203234-0d520185-cb98-44dc-aa9f-9b8a7e52fba9.mp4

[ner00]

Windows x86_64: http://lindqvist.synology.me/windows/scummvm_libretro.dll.zip

@i30817 Are you using the core linked above?

That one crashes for me with BASS:

https://user-images.githubusercontent.com/2269603/173204250-de3a8046-e63d-4b37-a8c0-3f475b44d0f6.mp4

Unless I enable logging, in which case I get the buggy issues:

https://user-images.githubusercontent.com/2269603/173204308-355a9ada-0c06-4139-b0de-ef78922d0e30.mp4

[i30817]

Do you have the path to the extras set up in the options button in the menu that is hidden by that curvature filter on your video? I am using that core but i built it myself. And i'm in linux x86-64.

Wait there was a bug fixed recently for windows: https://github.com/diablodiab/libretro-scummvm-backend/issues/6

That said it 'either crashed or had severe results' (and one of the game was BASS). It was a windows only bug, but it was fixed in a recent commit, so try to check the if there was a new build of the core. The bug was fixed 6 days ago, so if the build is newer it should have it fixed.

[diablodiab]

I've just tested with the Windows build from one hour ago, and I do not have the issue you showed in the video with logging disabled. It might have been fixed by the libco update 6 days ago, especially if you say that this issue is not present in the official libretro core.

Try to download the latest version and see if it works now.

[ner00]

Try to download the latest version and see if it works now.

Isn't v2.6.0git 5f42cb19e6 the latest? Where do I get a newer version from?

[mrmatteastwood]

Hey @diablodiab, would you be willing to share the download link for the latest versions again? I looked through this thread for links and the ones I found pointed to old versions, like Nov. 2021.

[ner00]

@mrmatteastwood same

[diablodiab]

I meant the newest version on my build bot: http://build.bot.nu/nightly/

[mrmatteastwood]

Ah yes, nice, thanks! :-) If I find some time tonight, I will double-check this myself (Linux Mint 20.2).

[mrmatteastwood]

Quick note from my side: the latest version of the core doesn't run on RetroArch 1.10.3 on Linux Mint 20.2. I manually place scummvm_libretro.so in the cores directory, overwriting the previous one, then select "Load Core" in RA and attempt to load the ScummVM core. It immediately goes back to the main menu and it says "No Core" on the bottom left.

[ner00]

I meant the newest version on my build bot: http://build.bot.nu/nightly/

Thanks @diablodiab I did try the latest from your buildbot and it worked great on RetroArch 1.10.3, (Win 10 x64), so thanks again. Please consider adding that URL as a footnote in your opening comment, might help... I missed this, probably because it's already buried in the discussion.

I'd also like to take this opportunity to thank you for providing and refining the method for integrating the libretro back-end into the official scummvm source, that's great for people to have the ability to compile up-to date versions, even though some patching might be required here and there.

On that last note, a couple questions.

1. I compiled it myself but my build doesn't show the commit number after the version, it just says v2.7.0git, while yours says v2.7.0git 956637b

2. How do you suppress the orange ScummVM splash-screen from your builds? My build gets the orange splash-screen while yours doesn't. Here's a quick comparison (even though it's self-explanatory):

your (diablodiab) build

https://user-images.githubusercontent.com/2269603/173247911-449ec225-65c2-413b-937f-bf2bb93894e7.mp4

my build

https://user-images.githubusercontent.com/2269603/173247938-e84d9c08-b6f3-4c84-86f8-7703c266f4a4.mp4

Thanks!

[diablodiab]

Oh, regarding the splash screen that's an oversight on my part. I had it disabled for my own personal use, but it is not good karma to include that hack in a release out of respect of the ScummVM team. I wil have to enable that again.

[ner00]

I had it disabled for my own personal use

That's exactly my point, even though libretro doesn't follow that, I completely understand your position since you have a public distribution of the core, but that's not my case. My builds are for personal use and not for public release/distribution. If you feel strongly about it, can you at least tell me where to look without being too specific?

[diablodiab]

You can just search the code for "splash", it's pretty clear where the splash screen is shown. There is also a message from the devs about the splash screen at the code.

[ner00]

@diablodiab ok, thanks, it's not very important regardless. As for my build not showing the commit hash, any clue why?

EDIT: This was an issue with my environment, I had cloned the sources as a standard user but was building as root, git apparently doesn't like that so I had to set the folder as trusted: git config --global --add safe.directory ~/scummvm/build/backends/platform/libretro

[diablodiab]

Quick note from my side: the latest version of the core doesn't run on RetroArch 1.10.3 on Linux Mint 20.2. I manually place scummvm_libretro.so in the cores directory, overwriting the previous one, then select "Load Core" in RA and attempt to load the ScummVM core. It immediately goes back to the main menu and it says "No Core" on the bottom left.

I've just tested it on Linux Mint 20.2 in a virtual machine, and I was able to run the x86_64 core without issues. Is this the core you were trying, and is it still not working? (try redownloading).

[mrmatteastwood]

A-ha! The latest build from your buildbot works. Nice, thanks! btw, I like that you put the ScummVM splash screen in at the beginning.

I'm still a bit hazy about what to do to get the Kyrandia games to run. It says it's missing the KYRA.DAT file, and I learned before that one needs to get that (and some other files) specifically for the exact ScummVM version that one is running, so, your current build being 2.7.0. I don't know where to get that particular file from, though...

[i30817]

A zip with data files are in the buildbot too (you can also build them yourself, which is what i do). I just tested kyrandia 1.

You also need to specify the path to those data files in the scummvm gui or ini (start the core, select the options and check the paths to fill it in the scummvm gui). Or use my tool mkscummvm, which does the playlist and setup the paths if you place the data files in the 'standard' directory for retroarch (system/scummvm/extra iirc).

[mrmatteastwood]

Yes! This is awesome! I had no idea the extra files were on the buildbot. I tried them with the latest ScummVM build (downloaded justa few hours ago). I deleted the scummvm.ini, all core options, and completely set up the themes and extras folder from scratch to start with a clean slate.

ScummVM works with most games, but a few things make it crash:

1. Enter RetroArch menu during Kyrandia 1 gameplay, go to Latency, change audio latency, then go back and select "Close Content" or "Resume". Usually crashes RetroArch completely. 80% repro rate.
2. Attempt to run Legend of Kyrandia 1 from RetroArch playlist. 100% repro rate.
3. Attempt to run Legend of Kyrandia 1 from ScummVM's own playlist. 30-40% repro rate.

Kyrandia 1 has very choppy music, it crackles and pops like crazy. Enabling the speed hack doesn't fix it, and changing the audio latency crashes RetroArch (cf. point 1 above).

[i30817]

about '2' or '3', that doesn't happen to me. For what it's worth, in scummvm.ini my kyrandia 1 section is this:

[kyra1-cd]
description=The Legend of Kyrandia (CD/DOS/English)
extra=CD
path=/media/i3/Mordred/Games/ScummVM/legend_of_kyrandia_1
engineid=kyra
gameid=kyra1
language=en
platform=pc
guioptions=midiPCSpk midiAdLib midiMt32 vga lang_English

Note, i use my tool libretro-mkscummvm to make the playlist, so that tool creates the 'The Legend of Kyrandia (CD_DOS_English) .scummvm' file for me without the 'forbidden characters' (but displays them in the playlist because the playlist name and the file can be different).

I did notice some crackle in kyrandia 1 though.

[mrmatteastwood]

Hmmm, that's exactly identical to mine (except the path, of course). Are you running RetroArch on Linux or Windows? I'm on Linux Mint 20.2.

Do you get music stuttering in Kyrandia 1 during the intro?

[i30817]

Linux (ubuntu 20.04) and yes.

[mrmatteastwood]

Thanks! When I find some time, I'll reinstall RetroArch completely from scratch and try again.

[i30817]

I don't think you should take those measures. I doubt it will help. Maybe use a debugger to try to find what is crashing.

[mrmatteastwood]

True. For now, I did a quick run from terminal:

thebrightside@CHAINSAWBOI:~$ retroarch
User picked target 'kyra1-cd' (engine ID 'kyra', game ID 'kyra1')...
   Looking for a plugin supporting this target... Kyra
Segmentation fault (core dumped)

[i30817]

Note LoK has some suspicious pauses during the intro that may cause crackles. One just before the westwood logo, one just before the westwood logo disappears, one just before the Legend of Kyrandia logo appears, etc. The pattern seems obvious.

I only notice the crackle with my earbuds, which are kind of ... bad... but the pause is obvious even without them.

[mrmatteastwood]

Here's a debug log for the LoK crash: https://pastebin.com/LEAUSK8n

My PC's full specs: https://termbin.com/qdmt

Video showing the game crashing and also the audio dropouts: https://youtu.be/xROgG8oQUU4

[i30817]

Mmm. Try to turn 'threaded video' off in the video options. Maybe it will affect it, maybe it will not. I've had some 'success' avoiding bugs in buggy cores because of that before, so who knows.

[mrmatteastwood]

Good thought, thanks! Alas, I already had it set to "Off". Switching to "On" made no difference.

Forgot to mention that the issue kept occurring after I deleted retroarch.cfg and used a fresh setup with only a small handful of custom settings.

Just spitballing: Perhaps it has to do with me being on Linux Mint 20.2, or perhaps me using the "edge" version of that distro which is running kernel 5.15. I hope that Linux Mint 21 comes out in the next few weeks, and I look forward to upgrading (will use the "edge" version again if available).

[dem1980]

Hi, i would like to compile myself for specifics devices (miyoomini or gcw0) from "debian 20.04 lts". What is the correct command line ? I use "make -platform=miyoomini" or "make -platform=gcw0" but the compiled file is always the "unix" target file. ps : i'm using the fork https://github.com/StupidHoroscope/libretro-scummvm-miyoo-backend for miyoomini support Thank you

[i30817]

try with two '-' as in make --platform=miyoomini

Not sure if it's that the problem, but unix commands 'long form' are usually prefixed with two -- and short only -.

[0x90shell]

Apologies if this is stated somewhere in thread, but why aren't these latest builds being incorporated into RA?

[mrmatteastwood]

I think everybody is wondering that. My personal best explanation is, the RA devs either aren't aware or there's nobody available to take care of it. Does anybody know any RA dev(s) who we could tag here?

[i30817]

You'd probably have better results asking for a way to have the online updater use more sources for the downloads. Cores that get updated regularly are cores with dedicated maintainers. And since there is no dedicated maintainer in this core, it's better to switch to the version that is maintained, even if the maintenance is 'we just track upstream and nothing else', ergo, point the updater to another buildbot.

The backend has a much much better solution of 'don't even include the upstream code so no one can actually break it then 'fix it' in a way that breaks upstream compatibility'. Either it works or it's broken and no edits of 'upstream' code can do anything about it, just the libretro section. At the obvious cost of not being able to port the core to obsolete console compilers, and bad integration of 'extra' features of libretro, like runahead, cheats or savestates - those never worked with the scummvm core anyway.

Meh, i already had a issue closed on the dolphin libretro for this, so i'll just counsel you not to call certain people, because they're more likely to close the issue than face the reality, or even tolerate talking about it.