Modifying Cores to work with LibretroUnityFE

[MiLO83]

Hi. I have experience in modding Libretro Cores and can compile them for Windows, Linux, and Android. I am wanting to talk to you about modding OpenGL cores to just pass the texture without whatever is stopping them from working. Can you explain the differences between the cores that work and which don't? I'm sure I can edit and recompile compatible cores by making the OpenGL ones instead work like the functional ones.

I have over the passed 3 weeks modified the VBA-M core to natively run at times higher resolutions. I multiplied the VRAM and render output, so each tile becomes 16x16, 32x32, 64x64... etc, and it works. The issue is I would like to get the core running in unity so people can edit the tiles "in-game". It doesnt currently work in LibretroUnityFE. I already wrote a tile editor and the core loads and perfectly replaces the tiles with the higher res graphics, but in my Unity Tile Editor its hard to find the tile you want to edit and if I can make it so you edit the screen while you play, the same Tile files will work on all platforms in RetroArch. The core works perfect in RetroArch but it needs a good Tile Editor to improve the resolution by hand. I know there are filters like xBR etc but this is actual smaller pixels whereas filters smooth corners but don't make lines thinner. It's already working and done. Can we get the VBAM core working in LibretroUnityFE together? I will be eagerly awaiting your response,

• MiLO83

[Skurdt]

Hi, vbam isn't an opengl core. It doesn't start because an exception is thrown when setting the core options (the method I use works for other cores so either I need to do something extra, or the core doesn't set these properly).

Basically if you just return true for RETRO_ENVIRONMENT_SET_CORE_OPTIONS then games start (although the input didn't work for me but it might be because the options are not set I'm not sure yet...).

For your tile editing stuff, I've got no idea what's that all about ^^ don't you need access to memory maps or something? Once the core starts, the logs tell you what environments are not implemented (or partially implemented if I half-did it and made a proper log message...)

For the opengl issues there are so many...

• The fe itlsef has to be run using opengl, but then implementing a dx11 core wouldn't work...
• We can only call the core's run function in a unity opengl render thread (it switches between multiple so most of the time, if the core uses libco, threads are messed up and crash the editor and/or the build), it makes it that we more likely want to load the core and call every core functions in a separate thread, create our own gl context, handle input differently (because calling unity's input system outside the main thread isn't allowed) and then pass the texture somehow to unity (maybe using spout or something like slow pixel copies, again creating textures outside of the main thread isn't allowed since it reaches for render threads internally...).

Mupen64plus next works more or less inside of a build though, but still it's far from ideal...

[MiLO83]

While Textures can't be created outside the main thread, you can still manipulate Color[]'s on a thread and set them on thread in Update(); I'm not sure this is what you're talking about, just thought i'd mention. I'm gunna go have lunch and try what you said about setting true;

[MiLO83]

Also, in the VBAM Core : void retro_set_audio_sample(retro_audio_sample_t cb) { } It's empty, could this be the problem?

[Skurdt]

I pushed a commit to catch the exception sooner, the core should run now, but with non-working options. Input doesn't work for me in vba-m but does in vba_next, not sure why yet.

You can only manipulate the texture data if you have access to the actual texture, I can process all the pixels in a job (so in one or more threads) because the core sends pixel data as a basic array when not using hardware acceleration. For OpenGL, it outputs an OpenGL texture ID, which you can't do anything with, unless you are in the corresponding OpenGL context. The problem is that gl contexts in unity are not predictable, it does it's own switching and creation/deletion for different things and when you call the core, even in a render thread using a plugin, you may not be in the correct context, or even in a destroyed one if the core decided to set it's main thread while in this particular gl context but unity feels like it doesn't need it anymore (like for example when running in the editor, many temporary contexts are created and deleted)

[Skurdt]

Also, in the VBAM Core : void retro_set_audio_sample(retro_audio_sample_t cb) { } It's empty, could this be the problem?

most cores don't use this, they use the sample_batch callback

[MiLO83]

Could you make a List<> of contexts and lock it to the one that works?

[Skurdt]

You can't access the contexts on the C# scripting side, only in a native plugin, and only when the current context is current. Caching contexts doesn't work in opengl, you make one, use it and destroy it, sharing resources is very limited.

[MiLO83]

Could this be fixed in the cores?

[Skurdt]

That's very unlikely, creating a shared context is done at context creation, the core makes it's own then generate a texture ID when it feels like it (most often once at startup but it can do whatever, contexts are often recreated when resizing for example). But in unity we don't have context creation access, you can manipulate unity stuff easily but external stuff is more complicated...

I mean, it's probably doable but that's a bit beyond my expertise at this point ^^ When I had my own context in my own window there were no such issues, but then it wasn't integrated in the scene...

[Skurdt]

Not all issues come from OpenGL though, calling retro_init or retro_load_game also sets up threads as main thread for the cores. Calling these in the unity's main thread is fine in this case because it's always alive. Most of the issues are when calling retro_run, it's a single function that basically runs an entire frame, including input, sound, context switching, coroutine switching, etc... and when it's called from a render thread, which is the only way I found to make it work for n64, then it gets really messy... :| Things may or may not be there basically... without too much control.

[MiLO83]

Are you getting the Context of the n64 scene meshes or the Quad its rendered to?

[MiLO83]

isn't it rendered to a Quad?

[Skurdt]

what is?

[MiLO83]

The Framebuffer?

[Skurdt]

The C# Texture2D is sent to the native side, the core renders into it, when you get it back you can apply it to whatever object accepts a texture (like a quad, but not limited to)

[MiLO83]

Ok, for n64 would it be possible to mesh the scene in 3D?

[MiLO83]

That I would be very interested in if you could skip the framebuffer and mesh it out in unity as a 3d scene. https://www.youtube.com/watch?v=_yQCjKuRRl0

[Skurdt]

Honestly I have no idea, the end result is only a bunch of pixels for a 2d image. There's no depth or original mesh info or any 3d at all once everything is shown on a quad, screen or monitor (it's all fake matrix projection stuff).

To recreate the game's environment in real time you most likely want to be extremely clever reading and interpreting pixels (using readpixel) and position you own meshes using maths (but I doubt it's close to feasable...), or fiddle around with the running rom's memory and find the data you want and translate that to unity transforms... which may or may not be possible...

[Skurdt]

The only thing the emulator gives you when it renders is the final image, there's no data in that final render about the things you don't see (stuff outside the game's camera frustum). You'll basically get the same result as in the video you linked, where it looks like the game is rendered on a non flat surface but with still no extra data.

[MiLO83]

I know :) But if you could manipulate the in game camera..

[Skurdt]

More a side discussion than an issue, I don't think anything can really be done on the wrapper side for whatever the goal for this is. Closing the issue for now.