Runing multiple Linux servers causes crash with default steamclient.so

[Johnanater]

This has been an issue for a while. When I try running multiple servers on Linux, one will start, it will load successfully, but when another one is started, the first one crashes with the following error:

[17:12:33 INF][SDG.Unturned] Loading miscellaneous...
[S_API] SteamAPI_Init(): Loaded local 'steamclient.so' OK.
Caught fatal signal - signo:11 code:1 errno:0 addr:0x2d86
Obtained 14 stack frames.
#0  0x007f80b75223c0 in funlockfile
#1  0x007f80b95b233a in _dl_rtld_di_serinfo
#2  0x007f80b95b288a in _dl_rtld_di_serinfo
#3  0x007f80b95b31e9 in _dl_rtld_di_serinfo
#4  0x007f80b747c283 in _dl_sym
#5  0x007f80b73174a8 in dlclose
#6  0x007f80b747c838 in _dl_catch_exception
#7  0x007f80b747c903 in _dl_catch_error
#8  0x007f80b7317b59 in dlerror
#9  0x007f80b7317525 in dlsym
#10 0x007f80b63ba588 in SteamAPI_GetSteamInstallPath
#11 0x007f80b63ba91b in SteamAPI_GetSteamInstallPath
#12 0x007f80b63bb2b6 in SteamInternal_GameServer_Init
#13 0x00000040fb6d98 in (wrapper managed-to-native) Steamworks.NativeMethods:SteamInternal_GameServer_Init (uint,uint16,uint16,uint16,Steamworks.EServerMode,Steamworks.InteropHelp/UTF8StringHandle)
./ServerHelper.sh: line 34: 2611621 Segmentation fault      (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

I have to fix this by manually copying the steamclient.so from the linux64/ directory to the DS root directory. This needs to happen every time there's an update.

Here's the hashes of the two files: steamclient.so in linux64/: 9cae4a81a8f67f59cda22883ce4115f8362f1954 steamclient.so in DS root (the bad one): a6d5ca470d31434a87482c476ee36ae75ec1f33d

This is easy to fix with a script to copy the files after updating, but could we get an official fix? 😄

[educatalan02]

1622

[SDGNelson]

When you say from linux64 are you referring to the U3DS/linux64 directory? U3DS/steamclient.so is the 32-bit library automatically included by the Steam dedicated server redist depot, and U3DS/linux64/steamclient.so is the equivalent auto-included 64-bit library.

So are you saying that copying steamclient.so from U3DS/linux64 to U3DS fixes running multiple servers? Or from steamcmd/linux64? In the past copying from steamcmd/linux64 has been recommended because the one in Steam dedicated server redist is broken - sometimes Valve forgets to update it. Currently the ServerHelper.sh script copies the one from U3DS/linux64 to the location it gets loaded from, but maybe this needs to be updated for the current state of affairs?

[Johnanater]

When you say from linux64 are you referring to the U3DS/linux64 directory?

Yes, I'm referring to the U3DS/linux64 directory.

So are you saying that copying steamclient.so from U3DS/linux64 to U3DS fixes running multiple servers?

Yes, it also works with the steamclient.so from a SteamCMD installation under it's own linux64 directory.

Currently the ServerHelper.sh script copies the one from U3DS/linux64 to the location it gets loaded from, but maybe this needs to be updated for the current state of affairs?

ServerHelper.sh copies the U3DS/linux64 to /Unturned_Headless_Data/Plugins/x86_64/, which doesn't seem to fix it fully.

I'm experiencing this issue on Ubuntu 20.04 LTS as well as 22.04 LTS (on another machine), although I've been told someone else using Ubuntu 20.04 LTS has no issues with this using LinuxGSM.

[SDGNelson]

In that case it sounds like replacing the 32-bit version of the file with the equivalent 64-bit version is helping something, but I would imagine Steam should already be selecting the correct file by default. I wonder if the copying in ServerHelper.sh is no longer necessary and actually causing the problem. If you replace ServerHelper.sh contents with this and disable your custom copy does it launch multiple successfully?

#!/bin/bash

# To mark this script as executable you may need to run "chmod +x ServerHelper.sh"

# Terminal mode compatible with -logfile 2>&1 IO.
export TERM=xterm

# Run the server binary.
# -batchmode and -nographics are Unity player arguments.
# -logfile 2>&1 can be used to pipe IO to/from the terminal.
# "$@" appends any command-line arguments passed to this script.
./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[TH3AL3X]

Try this script `#!/bin/bash

-NetTransport=SteamNetworking

while true do steamcmd +login anonymous +force_install_dir /root/.steam/steamapps/common/U3DS +app_update 1110390 validate +exit export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:dirname $0/Unturned_Headless_Data/Plugins/x86_64/ cp -f /root/.steam/steamapps/common/U3DS/linux64/steamclient.so /root/.steam/steamapps/common/U3DS/Unturned_Headless_Data/Plugins/x86_64/steamclient.so yes | cp -rf /root/.steam/steamcmd/linux64/steamclient.so /root/.steam/steamapps/common/U3DS/ yes | cp -rf /root/.steam/steamcmd/linux64/steamclient.so /root/.steam/steamapps/common/U3DS/linux64/ yes | cp -rf /root/.steam/steamapps/common/U3DS/Extras/Rocket.Unturned/* /root/.steam/steamapps/common/U3DS/Modules/Rocket.Unturned ./Unturned_Headless.x86_64 +InternetServer/"$@" echo "If you want to completely stop the server process now, press Ctrl+C before the time is up!" echo "Rebooting in:" for i in 5 4 3 2 1 do echo "$i..." sleep 1 done echo "Rebooting now!" done

`

[Johnanater]

@TH3AL3X Thanks, but I wrote my own auto updater, which now copies the file automatically after an update is completed.

@SDGNelson I will try that when I get the chance. 🤞

[Johnanater]

I tried the modified ServerHelper.sh, but got the following error:

Loading miscellaneous...
dlopen failed trying to load:
steamclient.so
with error:
steamclient.so: cannot open shared object file: No such file or directory
dlopen failed trying to load:
/home/server/.steam/sdk64/steamclient.so
with error:
/home/server/.steam/sdk64/steamclient.so: cannot open shared object file: No such file or directory
[S_API] SteamAPI_Init(): Sys_LoadModule failed to load: /home/server/.steam/sdk64/steamclient.so
Setting up 4 worker threads for Enlighten.
  Thread -> id: 7f6828bc1700 -> priority: 1 
  Thread -> id: 7f6823fff700 -> priority: 1 
  Thread -> id: 7f68237fe700 -> priority: 1 
  Thread -> id: 7f6822ffd700 -> priority: 1 

So obviously, we still need to tell the script where steamclient.so is. I downloaded a clean U3DS install and messed around with ServerHelper.sh until I got it working. I made it copy steamclient.so from the U3DS/linux64/ directory to the root of the U3DS directory, then changed the LD_LIBRARY_PATH to just the current directory and it seems to work great. 👍

#!/bin/bash

# To mark this script as executable you may need to run "chmod +x ServerHelper.sh"

# Terminal mode compatible with -logfile 2>&1 IO.
export TERM=xterm

# Copy steamclient.so from the linux64 directory
cp linux64/steamclient.so ./steamclient.so

# Tell system where to find the steamclient.so we just copied. Without this
# command API Init will fail with an unable to find file error.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`dirname $0`

# Run the server binary.
# -batchmode and -nographics are Unity player arguments.
# -logfile 2>&1 can be used to pipe IO to/from the terminal.
# "$@" appends any command-line arguments passed to this script.
./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[SDGNelson]

Thanks for the analysis @Johnanater! Considering that the Unturned server is 64-bit only I do not think there is any harm in your proposal... I will update the vanilla ServerHelper.sh script accordingly. 😄 👍

[Arnex0]

hello nelson how are you? I have the same problem as Johnanater, I have already tried the different solutions that are being proposed in the chat but none of them work for me, I am attaching the images of how I have everything and the errors that I get in the console, I can only have one server open. can you help me with any solution? I have the machine VPS at OVH. https://gyazo.com/004237f6cf02af77fff9331302ceda68 https://gyazo.com/d0def190ad60222b8d3194df527840c0 https://gyazo.com/e0c1919c3149cf6253d5129d7e7fbc18 https://gyazo.com/8a7cd054293255c79326c98210f53e87

[SDGNelson]

@Arnex0 it seems to be looking for steamclient.so in /root/.steam/sdk64. Is that where you have steamcmd installed? Or are you using a different install system? I think there was a similar problem a while ago when people were using maybe "pterodactyl" but I am not familiar with it

[Arnex0]

hi, Nelson. I tell you that they gave me a solution so that only two servers can be executed and it was putting the port of 27015 on the first server and 27018 on the second and that's how it worked but when a third server is opened it no longer works. can only two. I share with you how I have the configuration. https://gyazo.com/75306e27447ba7746310c91779403754 and I also did what Johnanater did of copying the steamclient.so from the /home/steam/Steam/steamapps/common/U3DS/linux64 folder and passing a copy to U3DS. this is my configuration.

https://gyazo.com/315fdbde79340200dc9ce2904f17f22d https://gyazo.com/ba4251356f8984f0fda308bd1d4198f0 this is what comes out in the console when this problem happens: https://gyazo.com/6849ff03fc1887f40467522df3159acd

[Johnanater]

hi, Nelson. I tell you that they gave me a solution so that only two servers can be executed and it was putting the port of 27015 on the first server and 27018 on the second and that's how it worked but when a third server is opened it no longer works. can only two. I share with you how I have the configuration. https://gyazo.com/75306e27447ba7746310c91779403754 and I also did what Johnanater did of copying the steamclient.so from the /home/steam/Steam/steamapps/common/U3DS/linux64 folder and passing a copy to U3DS. this is my configuration.

https://gyazo.com/315fdbde79340200dc9ce2904f17f22d https://gyazo.com/ba4251356f8984f0fda308bd1d4198f0 this is what comes out in the console when this problem happens: https://gyazo.com/6849ff03fc1887f40467522df3159acd

All the servers need to be running on different ports. Is your third one running on a different port, such as 27021?

[Arnex0]

Yes, the three servers are on different ports, the first one is at 27015, the second at 27018, the third at 27021, and the fourth at 27024, but it doesn't work.

[Johnanater]

Oof, it looks like this introduced an unforeseen bug! When the servers are already started, and you try starting a new one, the rest will crash when the steamclient.so file is copied.

I've made some changes to the ServerHelper.sh file so that it will only copy the file if the hashes are different (which should only happen once per update). I'm hoping most Linux distros come with the shasum command, at least Ubuntu does.

Here's my new ServerHelper.sh 😄

#!/bin/bash

# To mark this script as executable you may need to run "chmod +x ServerHelper.sh"

# Replace 32-bit steamclient.so with 64-bit version. Thanks @Johnanater in issue #3243.
# Unturned server is only available as 64-bit currently so this seems like a reasonable precaution.
# There is a long history of steamclient.so issues (e.g. #2616) because sometimes the
# official dedicated server redist copy gets out of date if Valve forgets to update it,
# so in the past we have sometimes used the steamcmd version of steamclient.so if available.

# Get the hashes of both steamclient.so files, only copy if they do not match.
LOCAL_SHA=`shasum ./steamclient.so | awk '{print $1}'`
NEW_SHA=`shasum ./linux64/steamclient.so | awk '{print $1}'`

echo "Root steamclient.so SHA: $LOCAL_SHA"
echo "Linux64 steamclient.so SHA: $NEW_SHA"

if [ "$LOCAL_SHA" = "$NEW_SHA" ]; then
    echo "The steamclient.so files match, skip copying..."
else
    echo "The steamclient.so files do not match, copying..."
    cp linux64/steamclient.so ./steamclient.so
fi

# Tell system where to find the steamclient.so we just copied. Without this
# command Steam API Init will fail with an unable to find file error.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`dirname $0`

# Terminal mode compatible with -logfile 2>&1 IO.
export TERM=xterm

# Run the server binary.
# -batchmode and -nographics are Unity player arguments.
# -logfile 2>&1 can be used to pipe IO to/from the terminal.
# "$@" appends any command-line arguments passed to this script.
./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[SDGNelson]

Uh oh! Hmm... Considering that by default it does not seem to find steamclient.so in the game directory, I wonder if the easier workaround would be adding linux64 to LD_LIBRARY path rather than copying at all? Not tested yet but something like: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:`dirname $0`/linux64

[Johnanater]

That sounds a lot more simple.

I just tested it on my servers, and it seems to work 😃

[JustOneMoreBlock]

I had the same bug. Updated steam, changed from ServerHelper.sh to ServerHelper2.sh

[SDGNelson]

Forgot to close but I think 3.22.13.0 or 3.22.14.0 included the change to point at the linux64 directory.