Problems with netplay guest

alessandropaggi commented 4 years ago

I've posted many times on the forum about this issue, but I've still got no answer so here I am. I can't connect to netplay rooms with Retropie despite running the same core and the same rom (same CRC) of the host. Every time I try to connect to a netplay room I get the the messages: Compatible content found Couldn't find a suitable core or content file, load manually I've also ran this test: I am running Retropie on a Pi4 and Retroarch on my laptop (same Retroarch version, same core, same rom). I did not perform any rom scan on neither system (see below), I was just launching the same rom on the same emulator. On the host side I went in Retroarch menu Netplay->Host->Start Netplay Host. On the guest side I went in Retroarch menu Netplay->Refresh Room List and selected the host room.

Retropie is able to serve as a netplay host, that is, I was able to join the Retropie netplay host room as a guest from Retroarch running on the laptop.
Retropie cannot join netplay as a guest, that is, when I try to join the host room of Retroarch running on the laptop, I get the messages shown before. This also happen on any netplay host room I see in the room list.

I also tried installing Retropie on an Ubuntu virtual machine, obtaining the same results. I don't know if this related, but I also tried to create a playlist by scanning the rom directory, by going in Retroarch menu Main Menu->Load Content->Playlist->Scan Directory, selecting the rom directory and then Scan This Directory. On Retroarch running on the laptop this correctly created a playlist with the roms found in the directory, while Retropie is not able to find any content.

cmitu commented 4 years ago

I managed to reproduce this, but it's not fully clear why it happens. The content is matched based on CRC (so I think a playlist won't help), and if fails with

[...]
[INFO] [CONTENT LOAD]: CRC32: 0xc3508f7e .
[INFO] [Lobby]: Testing CRC matching for: C3508F7E|crc
[INFO] [Lobby]: Current content CRC: C3508F7E|crc
[INFO] [Lobby]: CRC match C3508F7E|crc with currently loaded content
[INFO] [Lobby]: Couldn't find a suitable core

which is puzzling, since the core is already loaded (!). The error message is from https://github.com/libretro/RetroArch/blob/01bd76405fc1441c200de54f5cfe920c6da44fcd/tasks/task_netplay_find_content.c#L145, but tracing back to see where state->core_path is set didn't show anything in particular.

The error shows up even with a vanilla RetroArch configuration (i.e. no RetroPie specific configs). One thing that seems to work is connecting directly to the remote netplay host from the RetroPie guest, but that has a limited utility, since it requires a keyboard.

@alessandropaggi in your test, the RetroArch host on the PC is from another RetroPie installation or is it an install downloaded from the Libretro site ?

alessandropaggi commented 4 years ago

@cmitu The host on the laptop is running RetroArch downloaded from the Libretro site, which just works for netplay out of the box. I also set up a RetroPie running on a virtual Ubuntu machine, and it fails connecting to the RetroPie on the Pi4, since none of them can act as a guest. The RetroPie running on the virtual Ubuntu machine works as a host for RetroArch running on the laptop but cannot join as a guest, exactly like the RetroPie running on the Pi4.

alessandropaggi commented 4 years ago

I don't know if this is related, but after loading a core on RetroArch running on the laptop if I go in the RetroArch Menu Main Menu->Load Content->Playlists->Manual Scan->Core I can see, besides Unspecified, the core that I just loaded. If I do the same thing on RetroPie after loading any rom I can only see Unspecified. It looks like RetroPie does not know the core it just loaded.

cmitu commented 4 years ago

That's probably because there's no playlist associated - the playlist contains the core info.

   {
      "path": "/home/pi/RetroPie/roms/pc/Quake.conf",
      "label": "",
      "core_path": "/home/pi/src/c1/dosbox_core_libretro.so",
      "core_name": "DOSBox-core",
      "crc32": "",
      "db_name": ""
    },

It may be that RetroArch tries to re-load the core, but since the cores are not installed where RetroArch expects them to be, it errors out.

Did this process ever worked for you or is it something that's been - relatively - changed recently ? I did a test with 1.8.5 (as client, from RetroPie) and I get the same behavior.

alessandropaggi commented 4 years ago

I think that I was able to create playlists with older RetroPie versions, but I was never able to join netplay.

cmitu commented 4 years ago

There is a workaround for this, just by adding the cores to the location where RetroArch expects them:

copy the core .info file(s) from https://github.com/libretro/libretro-core-info to $HOME/.config/retroarch/cores

symlink the core .so file to the same location - for instance fceumm:

ln -s /opt/retropie/libretrocores/lr-fceumm/fceumm_libretro.so $HOME/.config/retroarch/cores

alessandropaggi commented 4 years ago

Ok, this changed something. With the symbolic link to the core in the same directory of the core infos I am able to create playlists. RetroPie is still able to serve as a host. When I try to join netplay RetroPie loads the correct core and content, but it always shows the message Compatible content found Failed to initialize netplay This happens also if I directly load the correct content, and when trying to connect to RetroArch running on the laptop or to every netplay host shown in the netplay room list.

cmitu commented 4 years ago

Haven't tried all the situations, but in my tests I always have the (same) content loaded when starting netplay, both on host and client. I didn't use a playlist at all.

Enable verbose logging on the client and and post the debug messages from the client (not the whole log, just the portion regarding netplay initialization).

alessandropaggi commented 4 years ago

This is the log of RetroArch running on the laptop successfully joining the RetroPie host

[INFO] CRC32: 0xb19ed489 .
[INFO] [Lobby] Testing CRC matching for: B19ED489|crc
[INFO] [Lobby] Current content crc: B19ED489|crc
[INFO] [Lobby] CRC match B19ED489|crc with currently loaded content
[INFO] [Lobby] Loading core /Applications/RetroArch.app/Contents/Resources/cores/snes9x_libretro.dylib with current content
[INFO] [Netplay] connecting to 2.235.172.186:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[libretro INFO] Nonexistent Port (8).
[libretro INFO] Nonexistent Port (9).
[libretro INFO] Nonexistent Port (10).
[libretro INFO] Nonexistent Port (11).
[libretro INFO] Nonexistent Port (12).
[libretro INFO] Nonexistent Port (13).
[libretro INFO] Nonexistent Port (14).
[libretro INFO] Nonexistent Port (15).
[INFO] Connected to: "saccublenda"
[INFO] [netplay] You have joined as player 2

(I don't know if the Nonexistent Port entries have some meaning). This is the log of RetroPie unsuccessfully trying to join the RetroArch host

[INFO] CRC32: 0xb19ed489 .
[INFO] [Lobby] Testing CRC matching for: B19ED489|crc
[INFO] [Lobby] Current content crc: B19ED489|crc
[INFO] [Lobby] CRC match B19ED489|crc with currently loaded content
[INFO] [Lobby] Loading core /home/pi/.config/retroarch/cores/snes9x_libretro.so with current content
[INFO] [Netplay] connecting to 2.235.172.186:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[ERROR] Failed to set up netplay sockets.
[WARN] Failed to initialize netplay.

The same output is obtained when RetroPie tries to connect to a host in the room list

[INFO] CRC32: 0xbb49df09 .
[INFO] [Lobby] Testing CRC matching for: BB49DF09|crc
[INFO] [Lobby] Current content crc: BB49DF09|crc
[INFO] [Lobby] CRC match BB49DF09|crc with currently loaded content
[INFO] [Lobby] Loading core /home/pi/.config/retroarch/cores/fceumm_libretro.so with current content
[INFO] [Netplay] connecting to 24.50.204.144:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[ERROR] Failed to set up netplay sockets.
[WARN] Failed to initialize netplay.

cmitu commented 4 years ago

The failures on the PI site look like network/connection issues, not related to RetroArch. Is the host computer in the same LAN ?

alessandropaggi commented 4 years ago

Yes, but the same happens when the laptop is connected to another network: laptop as a guest

[INFO] CRC32: 0xb19ed489 .
[INFO] [Lobby] Testing CRC matching for: B19ED489|crc
[INFO] [Lobby] Current content crc: B19ED489|crc
[INFO] [Lobby] CRC match B19ED489|crc with currently loaded content
[INFO] [Lobby] Loading core /Applications/RetroArch.app/Contents/Resources/cores/snes9x_libretro.dylib with current content
[INFO] [Netplay] connecting to 2.235.172.186:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[INFO] Connected to: "saccublenda"
[INFO] [netplay] You have joined as player 2

RetroPie as a guest

[INFO] CRC32: 0xb19ed489 .
[INFO] [Lobby] Testing CRC matching for: B19ED489|crc
[INFO] [Lobby] Current content crc: B19ED489|crc
[INFO] [Lobby] CRC match B19ED489|crc with currently loaded content
[INFO] [Lobby] Loading core /home/pi/.config/retroarch/cores/snes9x_libretro.so with current content
[INFO] [Netplay] connecting to 93.56.72.43:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[ERROR] Failed to set up netplay sockets.
[WARN] Failed to initialize netplay.

cmitu commented 4 years ago

If I look at the log, I can see that it's the same (host) IP address used to connect for both cases:

RetroPie as host

[INFO] [Netplay] connecting to 2.235.172.186:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[INFO] Connected to: "saccublenda"

RetroPie as guest

[INFO] [Lobby] Loading core /home/pi/.config/retroarch/cores/snes9x_libretro.so with current content
[INFO] [Netplay] connecting to 2.235.172.186:55435 (direct)
[INFO] [netplay] Connecting to netplay host
[ERROR] Failed to set up netplay sockets.

This is not a LAN address - did you forward the netplay port (55435) on your router to go the RetroPie host ?

alessandropaggi commented 4 years ago

This is not a LAN address - did you forward the netplay port (55435) on your router to go the RetroPie host ?

Yes.

cmitu commented 4 years ago

Then it can't possibly work - at least in this test - since RetroPie client cannot connect to itself. Try using the LAN address to connect. It's a bit difficult to see in RGUI, but using the XMB or Ozone input driver I think you're able to see the IP address.

alessandropaggi commented 4 years ago

I've set the respective LAN addresses and the result is always the same (RetroArch connects as guest, RetroPie does not). I've also tried disabling the port forwarding, and connecting the laptop to another network. The result did not change.

cmitu commented 4 years ago

I can't reproduce this case - for me the LAN connection worked - with the aforementioned workaround.

alessandropaggi commented 4 years ago

Can you please pastebin your retroarch.cfg? Maybe it's some setting that I can't figure out.

cmitu commented 4 years ago

I have a stock RetroArch config - if you'd like to replicate it, just replace yours with the retroarch.cfg.rp-dist file.

alessandropaggi commented 4 years ago

Still the same behavior. I guess I have some connection problems or router configuration to fix. Thanks for your help.

alessandropaggi commented 4 years ago

I did some test on netplay rooms and was finally able to join as a guest. I guess I was having some lan problems.

cmitu commented 4 years ago

Glad you sorted out the network issue. We'll see if we can find a solution to the main problem - core (re)loading from a RetroPie guest.

cmitu commented 2 years ago

@alessandropaggi this has taken more than needed, but it should be solved with #3549 and #3548. I've been meaning to solve it a few times, but always got derailed with a half-hacked implementation and put it on the side.

But now, since RetroArch 1.10.1 will make - almost - mandatory the presence of .info files, I took to opportunity to find a simpler solution than the workaround.

I don't know if you can still test it (make sure to update RetroPie-Setup first and then the RetroArch package), but would be nice to have a 2nd confirmation.

alessandropaggi commented 2 years ago

@cmitu Thanks for looking into it. I'm a little busy with work at the moment, but I will gladly test it next week. Do you suggest to remove the symbolic links before testing it?

cmitu commented 2 years ago

Yes, you should remove them and also clear the .info files. Updating retroarch will create back the .info files, symlinks shouldn't be needed anymore. Thank you.

alessandropaggi commented 2 years ago

After removing the symbolic links and the .info files, I've updated RetroPie-Setup script and then RetroArch. The $HOME/.config/retroarch/cores gets populated with the .info files, however without the symbolic links to the .so files I'm not able to join a netplay room, as I keep getting the Couldn't find a suitable core or content file, load manually message, even if the same rom is loaded and running on the same core. After recreating the symbolic links I am again able to join the netplay rooms. Running as a host always works as before.

cmitu commented 2 years ago

@alessandropaggi hm, can you also re-install runcommand ? After running a game, can you check what are the contents of /dev/shm/retroarch.cfg ?

EDIT: do you have a saved retroarch.cfg or you're using the default provided with the RetroPie installation ? If your retroarch.cfg contains a value for libretro_directory, then the netplay connection will fail, due to RetroArch checking that folder instead of relying on the value passed by runcommand at runtime.

alessandropaggi commented 2 years ago

Ok, I've cleaned again the $HOME/.config/retroarch/cores directory, reinstalled RetroArch and runcommand. The retroarch.cfg indeed contained the line

libretro_directory = "~/.config/retroarch/cores"

which I commented out. Trying to join a netplay room as a guest now just crashes retroarch going back to ES with this log

Executing: /opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-snes9x/snes9x_libretro.so --config /opt/retropie/configs/snes/retroarch.cfg "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc" --verbose --appendconfig /dev/shm/retroarch.cfg
[INFO] [Config]: Loading config from: "/opt/retropie/configs/snes/retroarch.cfg".
[INFO] [Config]: Appending config "/dev/shm/retroarch.cfg".
[WARN] [GameMode]: GameMode cannot be enabled on this system ("dlopen failed - libgamemode.so: cannot open shared object file: No such file or directory.") https://github.com/FeralInteractive/gamemode needs to be installed.
[WARN] [Config]: GameMode unsupported - disabling...
[INFO] RetroArch 1.10.0 (Git e9e85f3)
[INFO] === Build =======================================
[INFO] Capabilities:  NEON VFPv3 VFPv4
[INFO] Built: Jun 15 2022
[INFO] Version: 1.10.0
[INFO] Git: e9e85f3
[INFO] =================================================
[INFO] [Input]: Found input driver: "x".
[INFO] [Core]: Loading dynamic libretro core from: "/opt/retropie/libretrocores/lr-snes9x/snes9x_libretro.so"
[INFO] [Overrides]: Core-specific overrides found at "/home/pi/.config/retroarch/config/Snes9x/Snes9x.cfg".
[INFO] [Overrides]: No content-dir-specific overrides found at "/home/pi/.config/retroarch/config/Snes9x/snes.cfg".
[INFO] [Overrides]: No game-specific overrides found at "/home/pi/.config/retroarch/config/Snes9x/Super Mario World (USA).cfg".
[INFO] [Config]: Appending config "/home/pi/.config/retroarch/config/Snes9x/Snes9x.cfg".
/opt/retropie/supplementary/runcommand/runcommand.sh: line 1313:  6961 Segmentation fault      /opt/retropie/emulators/retroarch/bin/retroarch -L /opt/retropie/libretrocores/lr-snes9x/snes9x_libretro.so --config /opt/retropie/configs/snes/retroarch.cfg "/home/pi/RetroPie/roms/snes/Super Mario World (USA).sfc" --verbose --appendconfig /dev/shm/retroarch.cfg

This is the content of /dev/shm/retroarch.cfg:

video_fullscreen = "true"
video_fullscreen_x = "1920"
video_fullscreen_y = "1080"
libretro_directory = " /opt/retropie/libretrocores/lr-snes9x"

Uncommenting the libretro_directory line and putting back in $HOME/.config/retroarch/cores the *.so symbolic links brings the netplay working again. I don't know if it is related in any way, but I have noticed that retroarch.cfg contains another line

libretro_info_path = "~/.config/retroarch/cores"

cmitu commented 2 years ago

Thanks for testing. I haven't encountered any crash, but I wonder if it's because of an incorrect libretro_directory (the space at the beginning).

alessandropaggi commented 2 years ago

Thanks for testing. I haven't encountered any crash, but I wonder if it's because of an incorrect libretro_directory (the space at the beginning).

Yes, that's possible.

alessandropaggi commented 2 years ago

Ok, after 3558 I don't get the crash anymore. However after cleaning the $HOME/.config/retroarch/cores directory, reinstalling RetroArch and runcommand, and commenting the libretro_directory = "~/.config/retroarch/cores" line in retroarch.cfg I keep getting the Couldn't find a suitable core or content file, load manually message.

cmitu commented 2 years ago

Are the host and client running the same core and ROM ? Check if you haven't got libretro_directory set also in the system's retroarch.cfg.

alessandropaggi commented 2 years ago

Are the host and client running the same core and ROM ?

Yes.

Check if you haven't got libretro_directory set also in the system's retroarch.cfg.

It is not set in the system's retroarch.cfg.

cmitu commented 2 years ago

What game are you using for testing ?

alessandropaggi commented 2 years ago

Super Mario World for SNES. Note that the same rom and core work without cleaning the $HOME/.config/retroarch/cores directory and commenting the libretro_directory = "~/.config/retroarch/cores" line in retroarch.cfg.

cmitu commented 2 years ago

I see. I've tried with Super Mario World (USA) and the netplay works - I can connect to the host without the error (Couldn't find a suitable core or content file, load manually).

One difference may come from the .cfg files you have - can you upload both general /opt/retropie/configs/all/retroarch.cfg and /opt/retropie/configs/snes/retroarch.cfg ? If the snes specific config is the default, then there's no need to upload it. Make sure you remove any user/passwords (netplay) from the .cfg.

alessandropaggi commented 2 years ago

Thanks for taking the time to dig into it. Here is the /opt/retropie/configs/all/retroarch.cfg: https://pastebin.com/vDQZcSGy Here is the /opt/retropie/configs/snes/retroarch.cfg:

# Settings made here will only override settings in the global retroarch.cfg if placed above the #include line

input_remapping_directory = "/opt/retropie/configs/snes/"

#input_player1_joypad_index0
#input_player2_joypad_index1
#include "/opt/retropie/configs/all/retroarch.cfg"

I also use a core override, pasted here: https://pastebin.com/wZ1NF2cs

cmitu commented 2 years ago

I still can't reproduce it - either by a direct LAN connection or a Relay server. Host (Win10) :

# After starting hosting, using Madrid Relay server to announce the host
[INFO] [Netplay] You have joined as player 1
[INFO] [Netplay] Queued tunnel link connection.
[INFO] [Netplay] Tunnel link connection completed.
[WARN] [Netplay] WARNING: A netplay peer is running a different version of RetroArch. If problems occur, use the same version.
[INFO] [Netplay] Connection slot 0
[INFO] [Netplay] Got connection from: "saccublenda"
[INFO] [Netplay] saccublenda has joined as player 2 (ping: 101 ms)

# Stopping host, disable relay 
[INFO] [Netplay] "saccublenda" has disconnected

# ..then start host again.
[INFO] [Netplay] You have joined as player 1
[INFO] [Netplay] Netplay UPnP Port Mapping Failed
[INFO] [Discovery] Query received on LAN interface.
[WARN] [Netplay] WARNING: A netplay peer is running a different version of RetroArch. If problems occur, use the same version.
[INFO] [Netplay] Connection slot 0
[INFO] [Netplay] Got connection from: "saccublenda"
[INFO] [Netplay] saccublenda has joined as player 2 (ping: 25 ms)

Client (RetroPie):

# Using relay server
[INFO] [Lobby]: Testing CRC matching for: B19ED489|crc
[INFO] [Lobby]: Current content CRC: B19ED489|crc
[INFO] [Lobby]: CRC match B19ED489|crc with currently loaded content
Map_LoROMMap
Map_LoROMMap
[INFO] [Lobby]: Loading core /opt/retropie/libretrocores/lr-snes9x/snes9x_libretro.so with current content
[INFO] [Netplay]: Connecting to europe-west1.relay.retroarch.com|55435 (direct)
[WARN] [Netplay] WARNING: A netplay peer is running a different version of RetroArch. If problems occur, use the same version.
[libretro INFO] Nonexistent Port (8).
[libretro INFO] Nonexistent Port (9).
[libretro INFO] Nonexistent Port (10).
[libretro INFO] Nonexistent Port (11).
[libretro INFO] Nonexistent Port (12).
[libretro INFO] Nonexistent Port (13).
[libretro INFO] Nonexistent Port (14).
[libretro INFO] Nonexistent Port (15).
[INFO] [Netplay] Connected to: "mitu1"
[INFO] [Netplay] You have joined as player 2 (ping: 110 ms)

# Host stops netplay, client got disconnected
[INFO] [Netplay] Netplay disconnected

# connecting again via LAN discovery
[INFO] [Lobby]: Testing CRC matching for: B19ED489|crc
[INFO] [Lobby]: Current content CRC: B19ED489|crc
[INFO] [Lobby]: CRC match B19ED489|crc with currently loaded content
[INFO] [Lobby]: Loading core /opt/retropie/libretrocores/lr-snes9x/snes9x_libretro.so with current content
[INFO] [Netplay]: Connecting to 192.168.10.199|55435 (direct)
[WARN] [Netplay] WARNING: A netplay peer is running a different version of RetroArch. If problems occur, use the same version.
[libretro INFO] Nonexistent Port (8).
[libretro INFO] Nonexistent Port (9).
[libretro INFO] Nonexistent Port (10).
[libretro INFO] Nonexistent Port (11).
[libretro INFO] Nonexistent Port (12).
[libretro INFO] Nonexistent Port (13).
[libretro INFO] Nonexistent Port (14).
[libretro INFO] Nonexistent Port (15).
[INFO] [Netplay] Connected to: "mitu1"
[INFO] [Netplay] You have joined as player 2 (ping: 38 ms)

I've loaded your main retroarch.cfg, with the override added, but without the SNES config (since it's the same as mine). The only difference may be the host RetroArch (I'm using ver 10.0.3), but I don't think that downgrading to 1.10.0 (same as RetroPie) would have an effect.

Not sure what to try next :/, looks like a real puzzler.

alessandropaggi commented 2 years ago

Thanks anyway for taking the time to dig into it. I will stick to the workaround for now.

cmitu commented 2 years ago

@alessandropaggi thank you for testing.

Still can't explain why it's not working. I've re-tested with libretro_directory set to default (~/.config/retroarch/cores) in both retroarch.cfg files, but its value is still overridden by /dev/shm/retroarch.cfg (which is generated by runcommand at runtime, based on the core name).

alessandropaggi commented 2 years ago

I'm clueless too. I may retest it on a fresh installation during the weekend.

alessandropaggi commented 2 years ago

Ok, I've tried with a fresh install, updated everything, and IT WORKS! I guess there's something wrong with my current configuration, but I will stick with it anyway until the bullseye image comes out. So long, and thanks for all the testing.

cmitu commented 2 years ago

@alessandropaggi thanks again for testing. I expected a fresh install would work, but I wouldn't call your current configuration 'wrong'. It's normal for users to create overrides, edit their config to some degree or even use the (infamous) Save Configuration menu from the RetroArch's GUI. We should anticipate and accommodate (as much as possible) this situation. The workaround you've been using should cover both cases (default/close to default installation and user modified configs).

I'll leave this open; maybe the symlink approach is a safer solution - I'll do some more tests with my previous approach and see if we can find an all-around working solution to this issue.

RetroPie / RetroPie-Setup

Problems with netplay guest #3161