[Feature] Regular Linux build support

[parkerlreed]

Is your feature request related to a problem? Please describe.

Linux build currently assumes the docker solution.

Describe the solution you'd like

Support for the classical launching PHP router and QEMU Gamezip server based on services.json

Describe alternatives you've considered

Currently have a script that is loading everything and hard-coded to a specific game.

#!/bin/bash
unset LD_PRELOAD
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export PATH="$SCRIPT_DIR/FPSoftware/FPWine-8.0:$PATH"
if [[ ":$PATH:" == *":$HOME/.local/bin:"* ]]; then
    echo lol
else
    export PATH="$HOME/.local/bin:$HOME/.local/podman/bin:$PATH"
fi
cd $SCRIPT_DIR/Legacy
php -S 127.0.0.1:22600 router.php &
cd $SCRIPT_DIR/Server
qemu-system-i386 -machine pc-i440fx-5.2 -m 128 -net nic,model=virtio-net-pci -net user,hostfwd=tcp:127.0.0.1:22500-:80 -qmp tcp:127.0.0.1:22501,server,nowait -qmp tcp:127.0.0.1:22502,server,nowait -drive file=alpine.qcow2,if=virtio -serial stdio -display none -loadvm quick &
wine start /wait /unix /home/deck/Games/Flashpoint/FPSoftware/Shockwave/PJ101/SPR.exe http://www.candystand.com/games/slipstream/slipstream_1080.dcr --setTheRunMode Plugin --forceTheExitLock 0 --disableGoToNetPage --noDirectX7 --newScriptText "on alertHook me, err, msg --newScriptText return(1)" --do "set the alertHook to script(newScriptMember)" --do "set the resizable of the stage to 0" --do "set the visible of the titlebarOptions of the stage to 0" --do "set the border of the appearanceOptions of the stage to #none" --do "set the rect of the stage to the desktopRectList[1]" --do "set the drawRect of the stage to the desktopRectList[1]" --do "set the fixStageSize to 1"
killall php
killall qemu-system-i386

Additional context

Script works well enough but CLIFp functionality would be much preferred.

[oblivioncth]

Sorry for the delay, I've been busy and also accidentally nuked my Ubuntu dev environment not that long ago and hadn't gotten around to making a new one.

When I initially added support for this to CLIFp I only kept in mind making it work with the docker approach since that is the only way Flashpoint is designed to run natively on Linux out of the box, with the alternatives being using the Windows version directly with Wine or in a VM, in which case you'd just use the Windows version of CLIFp.

I hesitate to stray too far from this since Flashpoint is already intrinsically jank enough, with its major updates almost always breaking CLIFp, and its quirks requiring some annoying intricacies in CLIFp's implementation.

Since you're running a modified version of Flashpooint that's essentially a Frankenstein of the Windows/Linux versions (primarily since you're using the QEMU server) I wasn't exactly sure at first what you wanted me to support; however, after re-reading your issue I think I may understand what you mean.

To clarify, as things stand now there are two separate, but related hurdles when it comes to using CLIFp in this manner:

1) Actually re-configuring FP on Linux to run via QEMU (i.e. installing QEMU, copying over the alpine image, modifying services.json, etc.), and performing any other modifications as you see fit

2) Dealing with the fact that, as you've noticed, CLIFp assumes docker is being used on Linux. IIRC in reality this actually only the case in one section of the code base, as CLIFp does fully respect services.json otherwise. The vanilla launcher starts the docker image based server fairly early on and then takes a little bit to initialize itself. By the time it's finished, the docker image is completely up and running; however, with CLIFp this isn't the case. Because it is much more minimal, just starting up the services as they're listed would result in CLIFp starting the selected title before the docker image is ready, so I had to add an extra step where CLIFp explicitly waits for the docker image to come-up before proceeding. The issue is that it does this unconditionally on Linux, so if you've modified Flashpoint to not use docker, CLIFp now inadvertently can't be used because it will always terminate with an error since it will never see the docker image start.

I believe you're just asking me to resolve (2): Alter CLIFp's procedures on Linux so that they more strictly adhere to the contents of services.json . Essentially, check if docker is actually being used as the server and only then add the CLIFp specific docker wait task to the task queue. This way, even though it's non-standard, if you have modified your copy of Flashpoint to successfully run with without docker, CLIFp won't get in your way and in theory will work with that modified copy.

Is this correct?

[parkerlreed]

I believe so. The thing with the Linux version is you can take the Windows Flashpoint Infinity and remove its Launcher and place in a regular Linux build of the launcher, which is what I've done here.

I've avoided the actual Linux version because you know docker and Steam Deck don't exactly mix well.

This way you have a Linux launcher that loads a native Linux qemu and php but uses wine for the games as normal.

Essentially you have the Windows services.json with the only custom modification being to replace the qemu-system-i386.exe reference and removing the exe.

I agree supporting the non standard Linux way is a stretch but for the systems like the Deck where docker isnt really feasible, I think it would be a good addition.

[parkerlreed]

I tried adapting Linux Flashpoint's docker approach to podman which can run "rootless" and in the users home directory. podman IS a drop in docker compatible program but it chokes on the virtual network setup.

My current method of obtaining qemu and php because of the Steam Deck immutable fs is to use distrobox and podman to make those binaries available completely without modifying/read write rootfs.

https://gist.github.com/parkerlreed/4bd1f5fa38f7ffa72f9ceacb7d7f636d

[oblivioncth]

Huh, I'm surprised I haven't heard of podman yet, though I suppose it does seem to still be up-and-coming. Shame the docker image isn't completely compatible with it though, I guess it would need some tweaks upstream. I wanted to glance at that out of curiosity but I it looks like the keep the docker image configuration on a private GitLab instance.

Anyway, what you're doing makes sense given the limitations you're faced with. I'm honestly surprised that it works as well as it does with so relatively few modifications, though I imagine some of the more obscure platforms may be unknowingly broken if you haven't tried them all.

Regardless, try out this artifact whenever you feel like it: https://github.com/oblivioncth/CLIFp/suites/10805455887/artifacts/543528439

It has the change such that it won't try to wait on docker if the docker image isn't being used as the server, and instead follows normal procedures otherwise. In theory, given that you've configured Flashpoint itself to work properly (as it appears you have), I imagine it should work. There's a lot of moving parts here so I wouldn't get your hopes up too much, but I'm fine trying to tweak things a bit more as long as the changes aren't massive if necessary and it's relatively clear what needs to be done.

Assuming that's all that was needed to at least have it functioning for this use case in a basic capacity, I'll merge the feature branch and push a new release with the change.

[parkerlreed]

:D

Flash worked IMMEDIATELY

ps aux

deck       75915  0.2  0.1  79248 22224 ?        S    10:56   0:00 /usr/bin/php -S 127.0.0.1:22600 router.php
deck       75980  4.7  0.9 2409120 147496 ?      Sl   10:56   0:02 /usr/bin/qemu-system-i386 -machine pc-i440fx-5.2 -m 128 -net nic,model=virtio-net-pci -net user,hostfwd=tcp:127.0.0.1:22500-:80 -qmp tcp:127.0.0.1:22501,server,nowait -qmp tcp:127.0.0.1:22502,server,nowait -drive file=alpine.qcow2,if=virtio -serial stdio -loadvm quick -display none

Shockwave gets close but bails out in the Projector for some unknown reason

I will try to debug that on my own. It's definitely related to how clifp is loading it but may be something solvable in the configuration.

Thank you so much!

[parkerlreed]

though I imagine some of the more obscure platforms may be unknowingly broken if you haven't tried them all.

YES lol. Flash, Shockwave, HTML5 are good bets for working. Java applets work with your own sh. Everything else is a tossup. It's the great wild west!

[parkerlreed]

Is there a way to get verbose log from clifp?

I think flash only worked because it was locally cached by the Projector. New games are having trouble loading just like Shockwave.

If I had to guess the game is being started before php and qemu (mainly) have a chance to get running. Seems too quick to launch a game when qemu may take 4 seconds to load and the game is up within 2

[oblivioncth]

Could be something like that going on. That makes me worry that anyone with a toaster that's tried out CLIFp on desktop has never gotten it working if the Steam deck is too slow and experiences a race condition with QEMU and the game, yikes. Might have to see if like with docker there's a way to query QEMU to see if the VM has completely started and then wait on that.

CLIFp.log, which gets generated in your working directory is all there is, though it's fairly detailed.

Edit: Missed your initial response. Glad it seems like it's just on the cusp of working. Of course, it still could be tricky given I can't debug on the system directly, but hopefully that means only a few tweaks are needed to get it the rest of the way there.

[parkerlreed]

Ok good news. New Flash was my own doing. I had a mapping in the Launcher config pointing flash player to a local script for ruffle. Reverted this to regular Windows flash player and it launched as expected with no race conditions.

Still working on Shockwave.

Thanks a ton again!

[parkerlreed]

Funny part is nothing looks out of the ordinary with the launch command

 - <11:42:40> [TExec] Started Blocking process 'Slipstream': wine {"start", "/wait", "/unix", "/home/deck/Games/Flashpoint/FPSoftware/Shockwave/PJ101/SPR.exe", ""http://www.candystand.com/games/slipstream/slipstream_1080.dcr" --setTheRunMode "Plugin" --forceTheExitLock 0 --disableGoToNetPage --noDirectX7 --newScriptText "on alertHook me, err, msg" --newScriptText "return(1)" --do "set the alertHook to script(newScriptMember)""}

As compared to Launcher

                                    command:         "wine start /wait /unix "/home/deck/Games/Flashpoint/FPSoftware/Shockwave/PJ101/SPR.exe" "http://www.candystand.com/games/slipstream/slipstream_1080.dcr" --setTheRunMode "Plugin" --forceTheExitLock 0 --disableGoToNetPage --noDirectX7 --newScriptText "on alertHook me, err, msg" --newScriptText "return(1)" --do "set the alertHook to script(newScriptMember)"" ]

[parkerlreed]

Aha I was able to recreate it outside of clifp

In the output above, clifp is putting extra quotes around the URL and arguments. That's what causes wine to throw that error.

(2)(deck@steamdeck Flashpoint)$ wine start /wait /unix "/home/deck/Games/Flashpoint/FPSoftware/Shockwave/PJ101/SPR.exe" '"http://www.candystand.com/games/slipstream/slipstream_1080.dcr" --setTheRunMode "Plugin" --forceTheExitLock 0 --disableGoToNetPage --noDirectX7 --newScriptText "on alertHook me, err, msg" --newScriptText "return(1)" --do "set the alertHook to script(newScriptMember)"'

[parkerlreed]

It looks like CLIFP is passing all of the arguments as if they are the first argument. If you notice they're not broken up by the same array of double quotes and comma

[parkerlreed]

I'm currently poking at the ./src/task/t-exec_linux.cpp in the dockerless branch but you may beat me to the fix :)

EDIT: Yeah I'm terrible at C++, so did not get anywhere there :D

[oblivioncth]

Nice catch, that's certainly at least part of the issue.

This is a rough overview of how process execution works with the Linux version of CLIFp (and the official launcher):

1) Get executable and arguments of title 2) Swap the executable path if a corresponding entry for it exists in the appPathOverrides of preferences.json 3) Swap the executable path if a corresponding entry for it exists in execs.json 4) Check executable type a. If the executable is native (Linux), start it with its arguments. b. If the executable is a shell script, escape/quote its arguments and run it with /bin/sh c. If the executable is a windows batch file, swap the extension to .sh and see if there happens to be a corresponding shell script as a last resort (likely to fail) d. If the executable is a windows PE (.exe), try to run the executable with WINE as a last resort (mixed bag).

Under normal circumstances appPathOverrides and execs.json handle everything that's supported on Linux cleanly and 'd' almost never comes into play; however, I'm guessing that that part of the behavior is what's allowing you to use the Linux version of CLIFp at all. The Windows build of Flashpoint doesn't have execs.json and nearly doesn't use appPathOverrides (let alone for anything Linux related) and so basically every game you'd try to start will end up being handled by that last case since what's passed is an 'exe'.

Because on Linux normally it was such an edge case, I didn't put much effort into testing it out. The documentation for the 'start' command of WINE is a tad basic, and so I was unsure if the arguments for the windows executable specifically should be quoted or not in order to act as one argument, similar to cmd.exe /c and sh -c. As would turn out to be incorrect, I opted to surround them in quotes like you pointed out.

I initially spent some time cleaning up loosely related code and taking care of a few other odds and ends, but this did end up being a fairly straight forward change.

I hesitate to think you're out of the woods yet however. I wanted to see if I could reproduce the issue first just to be thorough, and despite it being relatively straight forward I didn't yet feel like throwing together a Frankensteined build of FP. Instead I just removed the execs.json entry for the executable that's used for Slipstream so that CLIFp would use the WINE fallback as already described. When I did this before implementing the quoting change I didn't receive an error like you did. Instead, the projector simply ran and presented a file dialog as if it hadn't been passed arguments, since it presumably ignored the incorrect ones it was given.

Because of this, there might be more going on here, but I'm keeping my fingers crossed that it can be chalked up to a weird discrepancy between WINE versions or something like that.

Nonetheless, here is the CI run and it's artifacts for the merged fixes https://github.com/oblivioncth/CLIFp/actions/runs/4115254470

[parkerlreed]

Success :)

New Flash and New Shockwave working well (As well as existing)

Java works after a Launcher mapping of FPSoftware\startJava.bat to FPSoftware\startJava.sh

I believe CLIFp is now on par with the Linux Flashpoint minus docker! Extra platforms is the same configuration between the two now.

[parkerlreed]

Of course I try celebrating too soon. Gamezip mounting seems to have issues.

https://gist.github.com/parkerlreed/6a553cf3dd39d3bad42cfbfb66798756

 - <14:25:53> [TMount] QMPI connected to QEMU Version: {"package":"","qemu":{"major":7,"micro":0,"minor":1}} | Capabilities: ["oob"]
 - <14:25:53> [TMount] Creating data pack mount point on QEMU instance...
 - <14:25:53> [TMount] QMPI command blockdev-add {"driver":"raw","file":{"driver":"file","filename":"/home/deck/Games/Flashpoint/Data/Games/887fa483-b9f9-489d-ac5f-7c88335c7946-1664775243535.zip"},"node-name":"ddkacmdpdtybmaie","read-only":true} returned - "{}"
 - <14:25:53> [TMount] QMPI command device_add {"drive":"ddkacmdpdtybmaie","driver":"virtio-blk-pci","id":"ddkacmdpdtybmaie","serial":"Lju4Q\\bSdeXC$@21L,2p"} returned - "{}"
 - <14:25:53> [TMount] Mounting data pack via PHP server...
 - <14:26:05> [TMount] CRITICAL) An error occurred while mounting a data pack. Error transferring http://127.0.0.1:22500/mount.php?file=887fa483-b9f9-489d-ac5f-7c88335c7946-1664775243535.zip - server replied: Bad Request
 - <14:26:05> [driver] Premature end of task 5
 - <14:26:05> [driver] End of task 5

Launcher with the same QEMU server works (albeit not as verbose)

INFO  [14:29:05]        QEMU GameZip: 200 GET /mount.php?file=Lju4Q%5CbSdeXC%24%4021L%2C2p HTTP/1.1

[parkerlreed]

The only difference I see is CLIFp is passing in the zip name to PHP mount.php while Launcher is passing in the serial

[parkerlreed]

Holy crap one I was able to fix myself :D

mounter.cpp I copied the win32 serial usage down to Linux and it works great

    QUrlQuery query;
    QString queryKey = "file";
#if defined _WIN32
    QString queryValue = QUrl::toPercentEncoding(mCurrentMountInfo.driveSerial);
#elif defined __linux__
    // FP Launcher uses "basename" but Node.js basename is actually filename
    QString queryValue = QUrl::toPercentEncoding(mCurrentMountInfo.driveSerial);
#endif
    query.addQueryItem(queryKey, queryValue);
    mountUrl.setQuery(query);

[oblivioncth]

Covered by 5c9ee1b