(WiiU) Update WUT and add RetroArch target for WUHB files (Aroma enviroment)

ShadowOne333 commented 1 year ago

Description

A few weeks ago, the WiiU received a new form of CFW called "Aroma", which runs apps under the WUHB extension. What this is effectively creating a coldboot option for the system, and also adds a bunch of features and other things as plugins directly on boot.

One of said options is the ability to be able to run Homebrew apps straight from the WiiU's home menu, as if it they were official apps on the system. This way, each homebrew app can be threated as their own app within the system.

In order to make RetroArch compatible with this setup, an update of WUT is required, as well as having a TV logo image (tv_splash.png, 1280x720), the gamepad image (drc_splash.png, 854x480) and the home menu image (icon.png, 132x179). With that, creating the WUHB file out of the rpx should be easy enough by following these recommended instructions after updating both wut and wut-tools: https://gbatemp.net/threads/release-aroma-environment-for-wii-u.618474/page-17#post-9940681

Expected behavior

Updating to the Aroma-friendly WUHB format would allow for RetroArch to be recognized and properly loaded on the system, alongside their proper boot logo, icon, and message.

Actual behavior

RetroArch currently cannot run under this environment since; while Aroma does have support for RPX apps, upon loading it simply stays on a black screen.

vgmoose commented 1 year ago

Maschell added the following info in the Aroma discord about rpx-loading:

Retroarch currently loads a core by loading into a special area in the memory and then using some old HBL magic to launch it. With aroma we can't use this region, because it's reserved for aroma and it's plugins. You can load a .rpx via Aroma though, Mocha_LaunchRPX is too "low" level, but librxpload (https://github.com/wiiu-env/librpxloader) could be extended to load a different .rpx which is inside the .wuhb.

grockk commented 1 year ago

I wonder if the vWii retroarch could be updated to support the sign_c2w_patcher in a forwarder or inject. We'd be limited to 480p but for retro games on Wii U that seems like it'd work pretty good enough. And then it would be compatible with Aroma.

ashquarky commented 1 year ago

I think I'm going to have a go at a wut port, at least depending on my availability over the holidays... I wrote up a bit of a game plan that might work but there are a few key questions:

Merging changes to all the cores and the frontend and the docker image at the same time is going to be a pain. Do we try and make a new platform= (wiiu-wut alongside wiiu?) to ease that or do we just wait for a release and then merge immediately after? I would prefer the latter but I don't know how RetroArch's release cycle works
Should wut be added in-tree? Since all the builds run on Docker now I don't see a reason to (especially considering that wut versions are somewhat tied to devkitPPC versions) but since libogc is in-tree it's worth asking

cc @LibretroAdmin @gblues

LibretroAdmin commented 1 year ago

wiiu-wut - as in a new platform/target - seems like a good idea. Overall I'm fine with whatever @gblues thinks we should do.

Should wut be added in-tree? Since all the builds run on Docker now I don't see a reason to (especially considering that wut versions are somewhat tied to devkitPPC versions) but since libogc is in-tree it's worth asking

How big is it ?

nstrong-scw commented 1 year ago

I would prefer that we avoid adding WUT in-tree. We should consume WUT artifacts, and we can control the version we get without resorting to completely duplicating the source tree. Adding it in-tree creates a bunch of toil. If we needed to add functionality to WUT, it should be contributed upstream. Which yes, may take a little longer, but it means that more projects than just RetroArch benefit.

I am in favor of having a separate build target. I propose that we just call it wut (which already implies wiiu). I also propose that once all cores have successful wut builds that the wiiu target builds be left deprecated. This is because the whole point of WUT is to provide a stable dev toolkit and I have no interest in maintaining 2 builds indefinitely.

gblues commented 1 year ago

whoops, I was signed into my work github account. ^ that's me. sorry!

ashquarky commented 1 year ago

I will mention that my current draft of RetroArch (frontend) does not keep both codepaths - it assumes wut.

Also, the differences in the cores might not be that bad - snes9x at least was just some (backwards-compatible) compiler flag changes; thus for that core a new target isn't needed. I assume other cores will be more complicated though.

(specifically -mwup needs to be replaced - I swear I've PRed this to some cores before but I can't find it now..)

ashquarky commented 1 year ago

For those following along:

RetroArch tree: https://github.com/ashquarky/RetroArch/tree/wiiu-wut
toolchain (docker): https://git.libretro.com/quarky/libretro-build-wiiu/-/tree/wut

Right now it mostly works under Tiramisu, aside from some long load times. Toolchain container is not well-tested, and cores still need to be updated.

ashquarky commented 1 year ago

Just submitted the makefile changes for snes9x - with luck, all the cores will be like this. The core now builds fine on both old and new devkitPPC (with and without wut) at the cost of some extra meaningless warnings on the old version.

ShadowOne333 commented 1 year ago

I didn't see the last messages, this all sounds fantastic. Did the recent commit that got merged for the makefile changes add .wuhb cores for RetroArch on WiiU? Or is that still WIP?

ashquarky commented 1 year ago

As an update for everyone:

Basically everything is working except for Salamander, which bootloops for some reason
I've had to disable libfat for the SD card since many Aroma modules mount the SD using the Cafe driver, and it's unsafe to have two drivers mounting the same filesystem at once. This makes startup really slow so I may re-enable it on Tira only. Playlists might be broken due to sd: becoming fs:/vol/external01.

I didn't see the last messages, this all sounds fantastic.

Did the recent commit that got merged for the makefile changes add .wuhb cores for RetroArch on WiiU?

Or is that still WIP?

wuhbs aren't in for this initial porting stage, though it may end up being a solution to the loading time issue.. we'll see! I'll probably try and land initial support first.

ShadowOne333 commented 1 year ago

As an update for everyone:
* Basically everything is working except for Salamander, which bootloops for some reason

* I've had to disable libfat for the SD card since many Aroma modules mount the SD using the Cafe driver, and it's unsafe to have two drivers mounting the same filesystem at once. This makes startup really slow so I may re-enable it on Tira only. Playlists might be broken due to sd: becoming fs:/vol/external01.
I didn't see the last messages, this all sounds fantastic. Did the recent commit that got merged for the makefile changes add .wuhb cores for RetroArch on WiiU? Or is that still WIP?

wuhbs aren't in for this initial porting stage, though it may end up being a solution to the loading time issue.. we'll see! I'll probably try and land initial support first.

That's awesome! Really looking forward to what you come up with for the RA rework for WiiU. Hope it all goes well with the last tidbits of nuances

Clownacy commented 1 year ago

I've had to disable libfat for the SD card since many Aroma modules mount the SD using the Cafe driver, and it's unsafe to have two drivers mounting the same filesystem at once. This makes startup really slow so I may re-enable it on Tira only.

That sounds familiar: devkitPPC's newlib port had its disk IO buffering disabled for the longest time, and it has only been re-enabled in a recent commit. Without buffering, SD card accesses are extremely slow. No update to devkitPPC has been released yet with this fix, so in the meantime IO buffering has to be explicitly enabled with setvbuf(fp, NULL, _IOFBF, BUFSIZ);. I've done this before in a port of mine, and it does greatly improve the SD card access times (including making the homebrew start-up much faster).

I hope this helps.

ashquarky commented 1 year ago

It's a good shout - setvbuf does help a lot, and we're already doing it in the wiiu port. The issues are instead mostly from the hundreds of small files existing in the assets folder, as well as the directory listings RA does. (the dirent functions get slower if there are larger files in the folder? not more, but larger?)

...wait, why are we setting a 128k vbuf and then a 4k one straight after? that's not in a3be192, did we get screwed on a merge conflict?

ashquarky commented 1 year ago

Also hey, had no idea devkitPPC had enabled buffering upstream. Do we know what their buffer sizes are and what release rev that change is in? Very cool if they've fixed one of the biggest pain points for porting things (fgetc)

Clownacy commented 1 year ago

From what I can make of newlib's code, the default buffer size is 1024 bytes (BUFSIZ). There doesn't appear to be any release of devkitPPC that has this change yet: buffering was enabled in August of last year, but the most recent release of devkitPPC is from May.

ashquarky commented 1 year ago

Makes sense. Looks like we'll still have to setvbuf in RetroArch (64k and 128k are the ideal sizes for large files in Cafe) but it'll be a nice help for other projects.

Ploggy commented 1 year ago

Hey all. I've been chipping away at editing the Retroarch Cores Makefiles to add the necessary changes Quarky mentioned for Aroma support. I have changed all the Cores that are listed here: https://github.com/libretro/RetroArch/pull/14925

I manged to compile a handful of modified Cores and have tested them on WiiU to see how they perform. There are a few Cores that fail to compile with the new changes but I guess that's to be expected and will have to be fixed on a per-core basis? Good news is they do boot direct from the WiiU Menu and can load a rom :) Bad news is Core switching seems to be broken and loading a rom from usb will throw an error.

Another problem (as mentioned above) is the long core load/rom load times, it can take up to 2 1/2 mins from loading the Core from WiiU Menu to game boot. https://www.youtube.com/watch?v=jn8YIVMi4EA&feature=youtu.be

But even with current issues is great to see Retroarch Cores working in Aroma! the future is now! :D Thank You Quarky!!!

Sowden commented 1 year ago

Hey @Ploggy, is there anywhere we could download this experimental wuhb to try it out? 😁

Ploggy commented 1 year ago

https://www.mediafire.com/folder/bvrennhlwsw2s/Aroma+Cores It isn't in Wuhb format just single rpx files but sure :) I did create a Wuhb file for testing but Retroarch hasn't got the ability to load cores from within a packaged Whub.

EDIT: fixed link

Sowden commented 1 year ago

Thanks for the speedy response! Editing my first post cause I'm dumb, lol. I'll work with the Atari cores to see how well it will work, but it is nice to see it running in Aroma. Good job!

Ploggy commented 1 year ago

The Atari Cores should be fine, just be warned the Cores do take a while to boot currently and so does rom loading ;)

ashquarky commented 1 year ago

Cores do take a while to boot currently and so does rom loading ;)

I am pretty sure this is an Aroma-specific issue, by the way. I don't have a great fix for it right now though. If you're switching between Aroma and Tira, try deleting retroarch.cfg and see if that speeds it up - it'll remember that libfat was disabled (from Aroma) otherwise ;)

Ploggy commented 1 year ago

(Testing Amstrad CPC Core) On Tira CFW = Core swapping possible.. loading a Aroma Core in Retroarch takes 41 seconds Once that Core is loaded, loading a game takes a further 41 seconds

On Aroma CFW (with fresh Retroarch.cfg) = Core swapping not possible.. When you try to load another Core after 54 seconds it'll return back to the Retroarch UI but the last Core will still be loaded. loading Core from WiiU Menu takes 1 min 18 seconds. Once that Core is loaded, loading a game takes a further 55 seconds.

So its a problem for both CFW but on Aroma Core switching is broken as well.

ashquarky commented 1 year ago

Core switching was definitely working in my tests :( Guess I'll have to check again...

Ploggy commented 1 year ago

Closing content and browsing long romlists with thumbnails is alot slower too :( Its a similar situation we had in Retroarch a while back with slow io?

ashquarky commented 1 year ago

It's really odd that you're still seeing it in Tiramisu. You've tested Tira, with everything up to date off tiramisu.foryour.cafe, and a clean retroarch.cfg; and it's still much slower than the current nightlies?

Ploggy commented 1 year ago

I can recheck tira updates. gimme sec.

Ploggy commented 1 year ago

The Cores are relatively recent like a week old and the Retroarch I'm using is your wiiu-wut-merge.. Even with 100% up to date Tira and fresh Retroarch.cfg it still takes a while to boot core/rom?

ashquarky commented 1 year ago

Alright, thanks for checking. I'll have another look :/

Ploggy commented 1 year ago

Sorry to bring bad news :(

..good news, at least it doesn't effect emulation performance ;)

Sowden commented 1 year ago

https://easyupload.io/m4je0i It isn't in Wuhb format just single rpx files but sure :) I did create a Wuhb file for testing but Retroarch hasn't got the ability to load cores from within a packaged Whub.

Hey guys. So I did some testing on the link you shared and didn't get great results :\ First off, I wasn't able to get anything to boot in Aroma, other than the Retroarch main menu. I could move around the Retroarch menus just fine though. When I tried to load content or even exit the app to return to the Wii U main menu, it would crash. Now I read that it has some long loading times, so each test I gave it about 10 minutes, but each one froze on the screen. And secondly, it also locks when I'm trying to load a core. And then oddly it showed the information for the last rpx core I loaded. So say for example that I tried loading Hatari rpx first. It crashed on me so I restarted the Wii U. I then loaded the DOSBox rpx and I got to the Retroarch menu. If I look at the "Core Information" at the top it would display "Atari - ST (Hatari)" and display all of the information on the ST core. Even though at the bottom corner it shows that "DOSBox - SVN" is loaded. Very odd. However, loading the rpx files you gave me through Tira works perfectly fine. Does this sound right or am I possibly doing something wrong? And in a side note, if the problems are Aroma related, might it be time to get Marshall in on this conversation?

Sowden commented 1 year ago

Man, has progress on this stalled?

Ploggy commented 11 months ago

Updated the link with the Test Aroma Cores since it was dead :) ^^^^

harrysof commented 9 months ago

still nothing on this ? :(

CrossScarDev commented 6 months ago

Any more work?

Nahieluniversal commented 5 months ago

Any news on the port?

Ploggy commented 5 months ago

Kinda, I still don't know what the cause of my slow loading is if it doesn't happen in Quarky's tests.. tho it has been said that the issue with the slow loading could be remedied with a PR that apparently has a 100x performance increase in WiiU native FAT32 driver :D not a bad addition lol https://github.com/devkitPro/wut/pull/340 It's hoped that that will fix any slow loading but it hasn't been implemented yet. I'm hoping it can be added to both Tira Retroarch and Aroma Retroarch. :D

But still strange Quarky doesn't get the slow loading issue. But others here have :/ Perhaps I'm compiling it wrong? Quarky do you still have an Aroma Core lying around that you used for tests?

SMwaterSHLDhelp commented 5 months ago

Has there been any easy to use implementation yet?

Ploggy commented 2 months ago

https://www.youtube.com/watch?v=lI2cNgmiosg&ab_channel=Ploggy

Aroma RA now Loads Cores/Games at a much more acceptable speed, I don't know what has changed between now and last year when I last tried it but I assume it must be Aroma CFW updates.?

Clownacy commented 2 months ago

Beta 17's changelog mentions patching the OS to make readdir and stat faster. Maybe that is what has fixed the speed problem?

Ploggy commented 2 months ago

Beta 17's changelog mentions patching the OS to make readdir and stat faster. Maybe that is what has fixed the speed problem?

I thought it could be that too :) It made a massive difference over what it was before. It's what piqued my interest to try it again.

Sowden commented 2 months ago

Aroma RA now Loads Cores/Games at a much more acceptable speed, I don't know what has changed between now and last year when I last tried it but I assume it must be Aroma CFW updates.?

This looks great. Are the downloads for this still the link you posted above months ago?

Ploggy commented 2 months ago

Aroma RA now Loads Cores/Games at a much more acceptable speed, I don't know what has changed between now and last year when I last tried it but I assume it must be Aroma CFW updates.?

This looks great. Are the downloads for this still the link you posted above months ago?

Yeah technically the Cores are the same but best to use this link.. It has a wuhb retroach file for a cleaner setup 👌

.. And a working Snes9x Core lol https://www.mediafire.com/file/jgnw7qol59h00gp/Aroma_Cores_Extract_to_SD_Card.zip/file

Dont Try to load any Tiramisu Cores with this lol

EDIT: Fixed link :)

vonmillhausen commented 2 months ago

.. And a working Snes9x Core lol https://www.mediafire.com/file/ttevxgvdxddv76h/Aroma_Cores_Extract_to_SD_Card.zip/file

Sorry to bother you @Ploggy, but I get an error from MediaFire with that link indicating that maybe the file has been removed - did you take it down?

Ploggy commented 2 months ago

.. And a working Snes9x Core lol https://www.mediafire.com/file/jgnw7qol59h00gp/Aroma_Cores_Extract_to_SD_Card.zip/file

Sorry to bother you @Ploggy, but I get an error from MediaFire with that link indicating that maybe the file has been removed - did you take it down?

Sorry here's a new link :)

https://www.mediafire.com/file/gcevyd36qgqtc5k/Aroma_Cores_Extract_to_SD_Card.zip/file

vonmillhausen commented 2 months ago

Sorry here's a new link :)

https://www.mediafire.com/file/gcevyd36qgqtc5k/Aroma_Cores_Extract_to_SD_Card.zip/file

You're a legend, thank you kindly! ❤️

Sowden commented 2 months ago

Hmmmmm. It appears that the Atari 5200 (A800) core crashes on loading a Atari atr floppy with a white text error screen. It looks like this is better, but not yet ready for primetime. And @Ploggy, could you please render and include a hitari core please? Those are the only two cores I use retroarch for anyways. Thanks man.

Ploggy commented 2 months ago

Hmmmmm. It appears that the Atari 5200 (A800) core crashes on loading a Atari atr floppy with a white text error screen. It looks like this is better, but not yet ready for primetime. And @Ploggy, could you please render and include a hitari core please? Those are the only two cores I use retroarch for anyways. Thanks man.

Yea, its true this is all WIP right now and Core weirdness is to be expected. I'll try and recompile that Core and the Hatari one and post it here if there's any difference :)

Ploggy commented 2 months ago

@Sowden I compiled Hatari and recompiled Atari800 but both seem the crash the same way? I dont know why? Probably these Cores are particularly sensitive to the Makefile changes that Aroma need. I dont think this is something I can fix.. I'll have to leave these two Cores to the people working on porting Aroma :/ Sorry.

libretro / RetroArch