Statically linked multicore RetroArch

[x6fan]

Statically linked RetroArch produces a copy of the front end for every core. This is documented in many places, such as the PlayStation Vita/TV build documentation:

With statically linked RetroArch, each executable is a separate libretro core instead of the core being separately loaded from a single executable. A pre-existing libretro library needs to be present in the root directory in order to link RetroArch Vita. This file needs to be called libretro_vita.a.

After a few seconds/minutes you should be able to find a retroarch_vita.elf and retroarch_vita.self file under that directory.

This is necessary because all cores must provide the exact same symbols defined by the libretro API. Attempting to statically link multiple cores into RetroArch will produce linker errors due to duplicated symbols.

If a simple symbol namespacing convention is adopted, it would make all exported libretro symbols specific to each core, avoiding the link time errors. A simple core_name + "_" prefix should suffice for this purpose.

The objcopy utility can be used to add a prefix to all exported symbols:

objcopy --prefix-symbols=core_ core.o

This can be a part of the build process of a libretro core. It can also be applied to a pre-built object file archive after the fact.

RetroArch can then be taught how to load these namespaced statically linked symbols:

• Maintain a list of cores statically linked into RetroArch
  • Gets populated at build time
  • Used to show the user the cores built into RetroArch

• Load symbols from the current process image and instead of opening external shared objects

  • dlopen can be given NULL instead of a path

    If filename is NULL, then the returned handle is for the main program.

• Statically linked core symbols can be resolved by
  • Forming a prefix
    • Concatenate the chosen statically linked core's name to the _ separator
  • Concatenating the formed prefix to the desired libretro symbol
  • Passing the resulting symbol to dlsym or platform equivalent

This feature is orthogonal to dynamic loading of cores compiled as shared objects. RetroArch can be compiled with any number of cores built into it and still maintain the ability to load and use other shared objects. It should even be possible to use a shared object version of a statically linked core. This will make statically linked RetroArch builds useful even on platforms that support dynamic linking. Perhaps it will even be possible to benefit from link time optimization.

[orbea]

It can also be applied to a pre-built object file archive after the fact.

Should not be hard to add it to the buildbot, but I'm unsure about the other changes personally.

[x6fan]

I'm unsure about the other changes personally.

I do not believe this feature would affect other code. Here are my reasons:

• The statically linked cores would be special cases in RetroArch's dynamic loading logic
  • It would still be possible to use normal shared objects in platforms that support them
• It is possible to build multiple RetroArch executables with one statically linked core each
  • Useful in platforms where it is desirable to minimize RetroArch executable size
  • Should be essentially the same as existing statically linked RetroArch builds
    • The only difference would be the symbol namespacing
• It does not force external libretro core projects to adhere to the namespacing convention

This is the same mechanism used by some dynamic programming language virtual machines in order to load and initialize native extension modules. When loading extension.so the virtual machine will attempt to execute a function named after the extension itself such as init_extension. For example, the reference implementations of Python and Ruby do this.

[orbea]

Sorry, I was not clear. I do not disagree with your suggestion, just I'm not sure how to go about implementing this. Maybe someone else would?

[ghost]

What is the intended use case for this exactly? The only thing that immediately comes to mind would be embedding multiple cores into one binary, which 1. isn't good for memory-constrained systems, and 2. those systems usually don't have dlopen() either.

I also think this does not conform to the libretro API and it shouldn't be made specific to RetroArch either, so it may have to be either something else entirely or part of a later extension/newer version of the libretro API... but maybe if you can clarify the usage, I can get a better understanding of your goal.

[x6fan]

What is the intended use case for this exactly?

By enabling fully static builds, it gives people the option to provide some built in cores for the user.

I talked to an emulator developer who decided to not identify himself. He favors static builds and expressed disappointment in the fact libretro cores usually require position independent code. During our discussion I had this idea and decided to create an issue.

The only thing that immediately comes to mind would be embedding multiple cores into one binary, which 1. isn't good for memory-constrained systems, and 2. those systems usually don't have dlopen() either.

It could be a valuable feature on systems that do not have these constraints. Static linking simplifies dependency management. It would let people ship a fully featured RetroArch frontend with their core built right into it.

I've just begun exploring the source code and I'm not yet familiar with RetroArch's implementation details on platforms that do not have dlopen. How does it bind the libretro symbols in those cases? Is it done at compile time? With macros, perhaps? The same mechanism can probably be used to support many statically linked cores.

I also think this does not conform to the libretro API and it shouldn't be made specific to RetroArch either, so it may have to be either something else entirely or part of a later extension/newer version of the libretro API

In its current form this convention is external to the libretro API. It's like a post-processing stage for conforming libretro cores that would allow many of them to be statically linked into RetroArch. Other frontends will have to add support for the core name prefixes as well if they want to use the modified cores for static linking.

I wasn't able to come up with an alternative design that allowed multiple statically linked cores and maintained backwards compatibility.

[ghost]

disappointment in the fact libretro cores usually require position independent code

Yes this is not technically a requirement we impose but simply a side-effect of using dynamic libraries as you might know.

It would let people ship a fully featured RetroArch frontend with their core built right into it.

This can already be done as you can build static RA even for desktops, today, without changing any code. You basically just use the --disable-dynamic option to configure and then provide a libretro.a file for it to link with.

How does it bind the libretro symbols in those cases? Is it done at compile time?

Yes, at compile time.

[x6fan]

You basically just use the --disable-dynamic option to configure and then provide a libretro.a file for it to link with.

In my understanding, this is limited to one statically linked core per RetroArch executable. It would be nice if people could statically link an arbitrary number of cores.

Turns out RetroArch already contains a mechanism that's similar to what I proposed. It is used for the internal cores, such as the image viewer core:

#ifdef RARCH_INTERNAL
#include "internal_cores.h"
#define IMAGE_CORE_PREFIX(s) libretro_imageviewer_##s
#else
#define IMAGE_CORE_PREFIX(s) s
#endif

void IMAGE_CORE_PREFIX(retro_run)(void) { /* ... */ }

/* internal_cores.h */

void libretro_imageviewer_retro_run(void);

The CORE_PREFIX macro is the compile time equivalent of the objcopy --prefix-symbols invocation. I haven't yet learned how RetroArch references these prefixed symbols defined in internal_cores.h but I suspect the same mechanism can be used to allow it to reference other statically linked cores.

[ghost]

Those are all hardcoded in internal_cores.h, so I don't think it's technically usable for what you want.

[RobLoach]

I've seen a LIBRETRO_STATIC variable get added to a few cores. When compiling with TIC-80, for example, it builds a tic80_libretro.a. This is used for some platforms, but won't be adopted across all RetroArch builds.