oblivioncth / FIL

A completely automated importer of Flashpoint Archive for LaunchBox and other frontends
GNU Affero General Public License v3.0
8 stars 0 forks source link

[Feature] Support for ES-DE #54

Open Joelis57 opened 1 month ago

Joelis57 commented 1 month ago

First of all, thank you for all the hard work you have done creating FIL and CLIFp!

I love flash games and would want there to be an option for Steam Deck users to use Flashpoint (even though I myself don't have a Steam Deck).

So your project are perfect for this!

People mainly use EmuDeck for setting up and playing old games on the Steam Deck. EmuDeck supports the EmulationStation-DE fronted and only recently it also added Pegasus frontend support. As most people are using EmulationStation, I think it would be the best option to focus on.

I am a C/C++ developer and would love to help implement this in my free time. Maybe you could give some advices on how to start approaching this? As I currently do not have a Steam Deck, I'm thinking to maybe try adding support for the Windows version of EmulationStation-DE.

Thank you so much for everyything!

oblivioncth commented 1 month ago

Hey!

I kept waiting till I can make a longer response, but just have been busy, so I didn't want to leave you hanging entirely.

I think EmulationStation is a good choice for the next frontend to add, especially for the reasons you've stated. Starting with the Windows version is probably for the best anyway since currently FIL can't run on anything but Windows. Since the main frontend this was made for is LB and that doesn't even exist on Linux, I hadn't bothered trying to make this cross-platform yet; though, that won't take that much effort and I can handle that myself.

Just give me a bit longer and I'll write out a high-level overview of how the frontend part of FIL works and the interfaces you'd need to implement to add ES. It can be a little tricky given that this whole group of projects is making a ton of things work together that were never designed to, but I don't think it's too bad.

oblivioncth commented 1 month ago

@Joelis57 Ok.

I could ramble for a while about the myriad of things I wish were better, or that I need to improve in time, but here is the gist of things.

Your best reference points are:

That import worker function drives the entire import process, mapping the user selections to tasks that include platform import, playlist import, and image import.

You'll end up creating at least these files:

app/src/frontend/emulationstation/
  es-install.h
  es-install.cpp
  es-data.h
  es-data.cpp
  es-items.h
  es-items.cpp

The overall paradigm of how an frontend is implemented is largely based on how LaunchBox is structured, since that was the first frontend, and to me, its facets seemed most adequate to represent any arbitrary frontend (at least I hope).

This is reflected through the various base classes you'd need to inherit from. Here is an overview of how they're organized and their purpose:

es-install.h/.cpp:

es-data.h/.cpp:

es-items.h/.cpp:

Essentially, the main thing you're doing is implementing a derivative of Fe::Install, while everything else is only created in order to support that one class.

As for general contribution "rules":

Of course I'm happy to help provide more detail with any part of the process as needed, or answer questions otherwise.

Joelis57 commented 1 month ago

Thank you so much for the detailed response! I don't have much free time, so it will take a while, but I will definitely try to implenent ES-DE. Thank you once more for all the great work you have done!

Joelis57 commented 1 month ago

@oblivioncth Also, could I maybe add you in some chatting app? So that I could ask small questions directly.

Joelis57 commented 1 month ago

So I had a crack at it today. I haven't been able to build FIL using CMake, but I've figured out what files need to be added/modified, tested it by doing it manually and everything works very well.

What needs to be done:

EmulationStation-DE\ES-DE\gamelists\flashpoint\gamelist.xml

This file contains a list of all games added from Flashpoint together with their metadata. Here is an example where I added Alien Hominid:

<?xml version="1.0"?>
<alternativeEmulator>
    <label>Flashpoint</label>
</alternativeEmulator>
<gameList>
    <game>
        <path>./08143aa7-f3ae-45b0-a1d4-afa4ac44c845.zip</path>
        <name>Alien Hominid</name>
        <desc>Alien Hominid HD is now available on Xbox 360 Live Arcade! Go try it and buy it!

Your UFO has crash landed, and the FBI is out to get you! Time to take them out!

Programming by Tom Fulp of Newgrounds.com!
Art by Dan Paladin!

Controls:
Use the arrows to run around and aim your gun. The 'a' key shoots and the 's' key jumps. When jumping over an enemy, press DOWN and 's' to do a freak attack!

****HINTS****
* You can ride enemy heads past roadblocks. they can run right through while they are freaking out!

* Eat enemy skulls in front of other enemies while on their shoulders -- their friend's reaction will give you a free cheapshot!

* If all else fails, you can try crawling your way to the end like the scum you are! haha

8/20/02 UPDATE:
Fixed grenade / Freak Attack Glitch
Fixed CAPS LOCK issues
Removed first grenade guy (now just 1)
Added first level intro cinema!</desc>
        <releasedate>20020807T000000</releasedate>
        <developer>The Behemoth</developer>
        <publisher>Newgrounds</publisher>
        <players>Single Player</players>
    </game>
</gameList>

I had trouble with the path field as ES-DE requires it to be inside the emulator's roms folder and ES-DE also requires the extension to match emulators list of extensions (which cannot be empty). But it doesn't actually check whether that file actually exists and just forwards it to CLIFp. So this solution I think is good enough.

EmulationStation-DE\ES-DE\collections

This folder contains Collections, so a custom list of games. Here is an example xml with only 1 game:

%ROMPATH%/flashpoint/08143aa7-f3ae-45b0-a1d4.zip

Here we have the same requirements as with the gamelist.xml and the same solution applies. Also the file has to be named custom-X.cfg, where X is the name of the Collection.

EmulationStation-DE\ES-DE\settings\es_settings.xml

From this file we retrieve these two values:

[...]
<string name="ROMDirectory" value="C:\Emulation\roms" />
[...]
<string name="MediaDirectory" value="C:\Emulation/storage/downloaded_media" />
[...]

Then ES-DE requires that we create an empty folder: %ROMDirectory%\flashpoint. No need to put anything inside. After that, we can copy images. Screenshots need to be copied to the %MediaDirectory%\flashpoint\screenshots directory and for logos I think the closest is %MediaDirectory%\flashpoint\covers. I also created a symlink %MediaDirectory%\flashpoint\miximages -> %MediaDirectory%\flashpoint\screenshots as miximages is the most popular option and otherwise it would display no preview.

EmulationStation-DE\resources\systems

There are 3 files that need to be updated for each OS. For now I've only tested on Windows.

EmulationStation-DE\resources\systems\windows\es_find_rules.xml:

[...]
    <emulator name="FLASHPOINT">
        <!-- Adobe Flash emulator Flashpoint -->
        <rule type="staticpath">
            <entry>%ESPATH%\Emulators\flashpoint\CLIFp.exe</entry>
            <entry>%ESPATH%\..\Emulators\flashpoint\CLIFp.exe</entry>
        </rule>
    </emulator>
[...]

EmulationStation-DE\resources\systems\windows\es_find_rules_installer.xml:

[...]
    <emulator name="FLASHPOINT">
        <!-- Adobe Flash emulator Flashpoint -->
        <rule type="systempath">
            <entry>CLIFp.exe</entry>
        </rule>
        <rule type="staticpath">
            <entry>%ESPATH%\Emulators\flashpoint\CLIFp.exe</entry>
            <entry>%ESPATH%\..\Emulators\flashpoint\CLIFp.exe</entry>
        </rule>
    </emulator>
[...]

EmulationStation-DE\resources\systems\windows\es_systems.xml:

[...]
    <system>
            <name>flashpoint</name>
            <fullname>Flashpoint</fullname>
            <path>%ROMPATH%\flashpoint</path>
            <extension>.zip</extension>
            <command label="Flashpoint">%EMULATOR_FLASHPOINT% play -i %BASENAME%</command>
            <platform>flashpoint</platform>
            <theme>flash</theme>
    </system>
[...]

Here I added a CLIFp.exe symlink in the EmulationStation-DE\Emulators\flashpoint folder. Another option would be to not do that and instead hardcode the CLIFp location inside the above 3 files. But that would stand out from all other emulators as it seems that these files should not be constantly updated. For example, in case ES-DE would some day want to officially support Flashpoint via CLIFp.

Flash already supported

Flash as a system is already supported in ES-DE, but only using .swf files with the emulators supported being Ruffle and ArcadeFlashWeb. But I added everything as a "new" system - Flashpoint instead of adding it as a third emulator because the implementation is quiet different from the other two. Also as a nice bonus we can reuse the "flash" theme label and with just that all themes in ES-DE display Flashpoint games with a nice theme.

Questions

  1. Is it possible to launch games fullscreen instead of manually having to enable that? That would greatly increase the user experience.
  2. It's hard for me to understand the "Platforms" part of FIL. As I understand "Playlists" in case of ES-DE map to "Collections", i.e. it's just a list of games. For Platforms at first glance I would think it's the same, just a convenient way to add for example all Flash games. But it looks like it's not that. It also requires to select at least one Platform. Or is it just to add a tag next to a game on what platform it is?

Thank you once more for all the great work you are doing!!!

Edit: Just to put everything in the same place. I removed my changes from the 3 files in EmulationStation-DE\resources\systems folder and instead added the same info in the 2 files (es_systems.xml, es_find_rules.xml) in the EmulationStation-DE\ES-DE\custom_systems folder. I think custom systems that are not officially supported by ES-DE should be put here, the outcome is the same. Also I think the root folder should be considered ES-DE instead of EmulationStation-DE as for example EmuDeck on SteamOS I think only has a ES-DE folder.

oblivioncth commented 1 month ago

So I had a crack at it today. I haven't been able to build FIL using CMake, but I've figured out what files need to be added/modified, tested it by doing it manually and everything works very well.

The latest commit on dev had an issue that prevented it from being built. I've fixed it and pushed a new one. It may be possible you need a static build of Qt as I don't recall how well mixing a dynamic build of Qt with a static build of FIL (which is the only one truly useful at the moment due to how it has to deploy CLIFp) since I don't normally build this project like that. Qt doesn't distribute static builds so if you try it with one of their dynamic ones and have issues I can share a static build with you.

I had trouble with the path field as ES-DE requires it to be inside the emulator's roms folder and ES-DE also requires the extension to match emulators list of extensions (which cannot be empty). But it doesn't actually check whether that file actually exists and just forwards it to CLIFp. So this solution I think is good enough.

Does it forward it as actually .zip ? Cause the zip would have to be stripped off or else it won't work. Finding have each entry map to a call of CLIFp play -i <GAME_ID> is optimal, though as a worst case scenario I could have CLIFp support the id containing an arbitrary extension that it just ignores, though I'd like to avoid that if possible.

Flash already supported

Flash as a system is already supported in ES-DE, but only using .swf files with the emulators supported being Ruffle and ArcadeFlashWeb. But I added everything as a "new" system - Flashpoint instead of adding it as a third emulator because the implementation is quiet different from the other two. Also as a nice bonus we can reuse the "flash" theme label and with just that all themes in ES-DE display Flashpoint games with a nice theme.

Yea, definitely try to keep things handled independently of that so that the use of FIL is as consistent across frontends as it can be, though reusing the icons may not be bad. Flashpoint has some already that are exposed to frontends during the import process, but if there are more/better ones that ES-DE has feel free to use them instead.

Questions

  1. Is it possible to launch games fullscreen instead of manually having to enable that? That would greatly increase the user experience.

At the end of the day CLIFp is in a way just a launcher like the stock Flashpoint launcher, and ultimately runs an underlying executable depending on the game in question. Because of this, whether a given title ends up being fullscreen depends on the launch parameters that are set up for that game in the FP DB. At some point I'll see if there is an elegant way to force fullscreen, though it might be tricky and require a case-by-case basis of certain switches or system calls depending on the underlying application, which, given how many are in FP now would be quite a maintence burden.

If it's realistic to achieve though, I agree it's a good idea, as generally people using CLIFp/FIL over the standard launcher are probably doing so in environments where they'd like the game to be full screen.

  1. It's hard for me to understand the "Platforms" part of FIL. As I understand "Playlists" in case of ES-DE map to "Collections", i.e. it's just a list of games. For Platforms at first glance I would think it's the same, just a convenient way to add for example all Flash games. But it looks like it's not that. It also requires to select at least one Platform. Or is it just to add a tag next to a game on what platform it is?

AFAICT, yes, ES-DE's Collections are what you'd want to turn the playlists into. The Platforms maybe be slightly tricky for this, depending on how anal a given user is. Since flashpoint itself is a collection, it's really comprised of a variety of systems/platforms (i.e. Flash, HTML5, Shokwave, etc), so actually instead of having "Flashpoint" be a system in ES-DE, the actually 1-to-1 mapping would be to have a new system for every Platform that the user has selected in FIL. LaunchBox has a nice way to handle this in that it has Platform Groups. So with LB we create a Platform for each selected Flashpoint Platform and then a Platform Group called "Flashpoint" which is assigned as a parent to all of them. This way you can see all Flashpoint entries in one place, but also have each "system" show up independently.

I only glanced, but my first impression is that ES-DE doesn't have any kind of "third-level" in its hierarchy like this, and only has:

Systems
|- Games

and

Collections
|- Collection

If this is the case, the most basic solution is to simply do each platform separately as a system and have no representation of Flashpoint as a whole, simply as a technical limitation. You could create a Flashpoint system that acts as a conglomeration of all the other systems that a user imported and have it contain all of the same game entries, albeit redundantly (so meh). You could also consider having there be a default Collection "Flashpoint" that contains an entry for ALL imported games, and exists even if the user didn't pick any playlists, which I imagine would better serve this purpose.

However, if you think that they should all be under a Flashpoint system for other complicating factors you come upon, then you can do that, and just have a playlist (Collection in this case) for each platform, so that users can still filter down to each platform if they want to. This is how I handled AttractMode due to its own limitations.

The most important thing is that there should be some kind of representation of the individual Flashpoint platforms that get added to ES-DE. The most idiomatic from an organizational standpoint is to do Flashpoint Platform -> ES-DE System, since that is what they are, but as a touched on above, there are other ways to handle it if you really feel its best.

It definitely seems like some people just want all of flashpoint together and don't care about the breakdown of each platform, like in #45, though I certainly don't want to make that the default. At some point I may get around to being able to do the import both ways, but of course this will add a good amount of complexity so I'd need a lot of time set aside for that.

When I have some spare time I'll also look over the ES file structure so I can better comment on some of your other findings if it makes sense to do so.

Also, could I maybe add you in some chatting app? So that I could ask small questions directly.

It would be good to keep really critical aspects of this discussion here so that this may serve as an example if anyone else comes along and wants to implement another frontend, but yea for real basic stuff you can add me on Discord if you want.

oblivioncth commented 1 month ago

Edit: Just to put everything in the same place. I removed my changes from the 3 files in EmulationStation-DE\resources\systems folder and instead added the same info in the 2 files (es_systems.xml, es_find_rules.xml) in the EmulationStation-DE\ES-DE\custom_systems folder. I think custom systems that are not officially supported by ES-DE should be put here, the outcome is the same. Also I think the root folder should be considered ES-DE instead of EmulationStation-DE as for example EmuDeck on SteamOS I think only has a ES-DE folder.

From what I've seen in the user guide so far, using custom_systems and treating ES-DE as the root appears to make sense.

Joelis57 commented 1 month ago

Does it forward it as actually .zip ?

Yes, where I use %BASENAME% above it only returns the GAME_ID.

AFAICT, yes, ES-DE's Collections are what you'd want to turn the playlists into. The Platforms maybe be slightly tricky for this, depending on how anal a given user is. Since flashpoint itself is a collection, it's really comprised of a variety of systems/platforms (i.e. Flash, HTML5, Shokwave, etc), so actually instead of having "Flashpoint" be a system in ES-DE, the actually 1-to-1 mapping would be to have a new system for every Platform that the user has selected in FIL. LaunchBox has a nice way to handle this in that it has Platform Groups. So with LB we create a Platform for each selected Flashpoint Platform and then a Platform Group called "Flashpoint" which is assigned as a parent to all of them. This way you can see all Flashpoint entries in one place, but also have each "system" show up independently.

Okay, everything makes more sense now. Basically, in the ES-DE\custom_systems\es_find_rules.xml we need to add the Flashpoint emulator. Then after that for each FIL Platform we need to add a system in ES-DE\custom_systems\es_systems.xml, which uses the Flashpoint emulator that we added. And after that for each playlist we add a collection in ES-DE\collections. I agree that a Flashpoint collection which includes all Flashpoint games is not as important and doesn't make sense to add a system. But it could be added as a Collection (playlist) sometime in the future, if you come around to doing that.

oblivioncth commented 1 month ago

Yes, where I use %BASENAME% above it only returns the GAME_ID.

Oh OK, then NP.

Okay, everything makes more sense now. Basically, in the ES-DE\custom_systems\es_find_rules.xml we need to add the Flashpoint emulator. Then after that for each FIL Platform we need to add a system in ES-DE\custom_systems\es_systems.xml, which uses the Flashpoint emulator that we added. And after that for each playlist we add a collection in ES-DE\collections. I agree that a Flashpoint collection which includes all Flashpoint games is not as important and doesn't make sense to add a system. But it could be added as a Collection (playlist) sometime in the future, if you come around to doing that.

It sounds like you have the right idea. Like you said, an overarching collection can always be added later. It would be simple enough to do.

Since you need to work with XML, you can reference how i did that with LaunchBox if you want to keep it fairly consistent.

Joelis57 commented 1 month ago

Since you need to work with XML, you can reference how i did that with LaunchBox if you want to keep it fairly consistent.

I'll try to keep everything as consistent as possible.

Just to be sure, if I select a platform in FIL, it imports all games from that platform with no option to import only a selected few from that platform? (that's not a feature request, just making sure how it's intended to work).

Also, now that we will have a separate system for each platform, they will also use separate rom folders. I think naming them flashpoint-flash, flashpoint-unity, etc. would make sense.

I've also tested the same steps on a Steam Deck. The only thing that's different is this part:

I had trouble with the path field as ES-DE requires it to be inside the emulator's roms folder and ES-DE also requires the extension to match emulators list of extensions (which cannot be empty). But it doesn't actually check whether that file actually exists and just forwards it to CLIFp. So this solution I think is good enough.

On the Steam Deck not only does there need to be an empty folder created in the ROM directory, but the folder needs to be populated with files for each game. So to keep it consistent, I think it makes sense to create empty files for each game (e.g. 08143aa7-f3ae-45b0-a1d4.zip).

And a Steam Deck specific issue: everything was working fine on Desktop Mode, but on Game Mode only a black screen was shown when launching games. To fix this, I went to Flashpoint Configs and in the pulldown menu "Native Platforms" checked "Flash". After that everything worked flawlessly.

oblivioncth commented 1 month ago

Just to be sure, if I select a platform in FIL, it imports all games from that platform with no option to import only a selected few from that platform? (that's not a feature request, just making sure how it's intended to work).

Yes. The intended way to import only a subset is for one to make a playlist with the titles they want from any given platform (or number of platforms), and then set the Playlist Game Mode option to "Force All".

Also, now that we will have a separate system for each platform, they will also use separate rom folders. I think naming them flashpoint-flash, flashpoint-unity, etc. would make sense.

👍

On the Steam Deck not only does there need to be an empty folder created in the ROM directory, but the folder needs to be populated with files for each game. So to keep it consistent, I think it makes sense to create empty files for each game (e.g. 08143aa7-f3ae-45b0-a1d4.zip).

Wow, that's really annoying, and would be nice if we could eventually find a way to do away with it, but at least for now do whatever you must to get it working.

And a Steam Deck specific issue: everything was working fine on Desktop Mode, but on Game Mode only a black screen was shown when launching games. To fix this, I went to Flashpoint Configs and in the pulldown menu "Native Platforms" checked "Flash". After that everything worked flawlessly.

I believe this is a known issue, that obviously is more of a Flashpoint problem than a CLIFp/FIL problem, but I wasn't aware that the native flash player (which I think is ruffle) gets around the issue.

oblivioncth commented 1 month ago

I'm going to move some of the XML implementation out of lb-data and into fe-data.h when I get a chance later tonight, as it should be pretty reuseable for ES-DE.

There's also an issue in that Fe::Install has a virtual function that expects to be implemented with a apth to an executable. Obviously that's not gonna fly given the ES-DE subdir is to be used, so I'm going to tweak that so that the functionality it provides is option and that method is no longer required.

Finally, to go with all this, I'll work on getting FIL to run and have AttractMode working on Linux as I'm able.

Joelis57 commented 1 month ago

There's also an issue in that Fe::Install has a virtual function that expects to be implemented with a apth to an executable. Obviously that's not gonna fly given the ES-DE subdir is to be used, so I'm going to tweak that so that the functionality it provides is option and that method is no longer required.

If you're talking about the path to CLIFp executable, then I think we can hardcode the path to the executable instead of using the symlink inside ES-DE subdir. Because my idea to use symlink occurred when I tried to add the system to the main system xml file. Now that we use the custom systems xml, I don't think it's as important. Also, on the Steam Deck all paths of custom systems in the xml are also hardcoded.

I'm going to move some of the XML implementation out of lb-data and into fe-data.h when I get a chance later tonight, as it should be pretty reuseable for ES-DE.

Finally, to go with all this, I'll work on getting FIL to run and have AttractMode working on Linux as I'm able.

Thank you!

oblivioncth commented 1 month ago

I actually meant the executable of the Launcher in question. All it uses it for right now is to read version info from the file details that only exist on Windows, so I can change that to be opt in and not required.

An issue related to this will be the fact that install are expected to be able to provide a version number to display to the user, but there might not be a great way to do that with ES-DE. The default implementation results in "unknown", so technically its not required.

This isn't the end of the world, but something to keep in mind if you come up with anything.

Joelis57 commented 1 month ago

Oh okay. We could get the ES-DE version by calling ES-DE.exe --version on Windows or ~/Applications/ES-DE.AppImage --version on the Steam Deck.

oblivioncth commented 1 month ago

Oh okay. We could get the ES-DE version by calling ES-DE.exe --version on Windows or ~/Applications/ES-DE.AppImage --version on the Steam Deck.

I just wasn't sure if there was a reliable place on things like a deck where ES could be found, but idk why as such things are usually pretty standard. It's slightly different on Windows as the EXE may or may not be in the parent directory of ES-DE, but of course you can just check for its existence and simply fall back to the base implementation (returns "Unknown Version") if not available.

oblivioncth commented 1 month ago

Alright, the QoL changes I mentioned:

Have been in dev for a bit, and now also rudimentary Linux support is there too. It runs, though I haven't tested that it actually works right yet.

As mentioned on Discord, FIL can be built with MINGW64 now as well.