New Linux Crashes - Githubissues

JustOneMoreBlock commented 3 years ago

I've tried searching before posting and unable to find any results with a similar error. We have had numerous crashes for days since the last update and there were no logs being outputted.

I started up two servers running pve and they ran for a little bit.

Unexpected mark stack overflow
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x6c8d
Obtained 19 stack frames.
#0  0x007fb2d7268980 in funlockfile
#1  0x007fb2d6ea3fb7 in gsignal
#2  0x007fb2d6ea5921 in abort
#3  0x007fb2d6513146 in GC_push_all
#4  0x007fb2d650884e in monoeg_g_hash_table_foreach
#5  0x007fb2d64f1f19 in mono_unity_liveness_calculation_end
#6  0x007fb2d651a378 in GC_mark_some
#7  0x007fb2d651b4cb in GC_collect_a_little_inner
#8  0x007fb2d651bfb5 in GC_collect_a_little
#9  0x007fb2d846d901 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#10 0x007fb2d8306ab2 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#11 0x007fb2d8306c46 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#12 0x007fb2d8366ee1 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#13 0x007fb2d836134e in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#14 0x007fb2d8361391 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#15 0x007fb2d83618aa in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#16 0x007fb2d84c9e4e in PlayerMain(int, char**)
#17 0x007fb2d6e86bf7 in __libc_start_main
#18 0x0055715eac8699 in _init
./ServerHelper.sh: line 29: 27789 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

Both have similar errors

root@ut:/home/Games/scripts# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

Startup

screen -dmS ${server} ./ServerHelper.sh +InternetServer/${server}

gegtor commented 3 years ago

Let's not forget that few people above reported it happening on Windows too

TH3AL3X commented 3 years ago

dlopen failed trying to load: /root/.steam/sdk64/steamclient.so with error: /root/.steam/sdk64/steamclient.so: cannot open shared object file: No such file or directory

dlopen failed trying to load: /root/.steam/sdk64/steamclient.so with error: /root/.steam/sdk64/steamclient.so: cannot open shared object file: No such file or directory Look is something related of ur script, :/ are u sure ur steamclient.so is located on "/root/.steam/sdk64/"?, anyways check that....

SDGNelson commented 3 years ago

Unfortunately I am still waiting to hear back from Unity. I will post here when there is any news.

Tomorrow's update is using the latest 2019.4.16f1 of course, but I doubt it will fix this issue yet. For future years I think the lesson has been that "long term support" is not necessarily as stable as we would like it to be, e.g. they keep merging new features, so going forward I will keep Unturned on whichever LTS version is in purely maintenance/fixes mode - Unity refers to this as "legacy LTS" 🙄

In the meantime to clarify a few points:

+SecureServer is an alias for +InternetServer, and +InsecureServer is an alias for +LanServer. These only control whether the Steam gameserver API is initialized in eServerModeAuthenticationAndSecure mode which affects whether authentication, server listing, and VAC are enabled. This may seem like an odd setup, and that would be because it has not changed since the earliest 2014 versions. Maybe a candidate for cleanup in the future, but relatively low priority.
The developer at Valve responsible for SteamNetworkingSockets recently realized that the dedicated server redist (steamclient.so) was being manually updated by developers at Valve which is why it lags behind the client release, and why there are often bugs related to it. Hopefully they will clean up that release process. In the meantime there are no known issues with it, and this crash is unrelated, so copying the client release should not be necessary. If the game is in the default install location the ServerHelper.sh script will copy the client release automatically anyway.

SDGNelson commented 3 years ago

Here is Unity's response:

As you can see the Player crash occurs in mono-2.0-bdwgc.dll on mono_breakpoint_clean_code. Unfortunately, since I do not have your full Player.log I cannot state for sure you are experiencing the same issue (even though error.log shows an almost identical call stack). It is unlikely the other issue is going to backported to 2019.4 as it is difficult to reproduce.

Could you please provide a project on which this issue is consistently reproducible? In order to further investigate and fix the issue, we have to be able to reproduce it on our end. A small project containing a bare minimum of assets required to reproduce the issue would be ideal. In case the project is very large, you can find information on how to strip it down here (step 3): https://unity3d.com/unity/qa/bug-reporting

Unless any of you have noticed any further patterns, I do not think we have a reproducible project for them. I will ask Unity if there is anything else we can do to narrow this down however, and point out that backporting crash fixes to an LTS release would be nice. To clarify, I attached the crash.dmp and error.log files to the report. 😞

gegtor commented 3 years ago

Can they do some of their magic on our servers?

I have 8 servers and a crash every 1-2 hours

I’m willing to give them all access they want to poke around with their tools

SDGNelson commented 3 years ago

@gegtor thanks for offering. In my reply to them I mentioned that it is a frequent crash, and asked if there is anything we can do to help narrow down the issue.

SDGNelson commented 3 years ago

Bad news: Unity closed the case. Regarding narrowing it down, this is what they said:

Unfortunately, without a reproduction project, there is very little we can do to investigate. I have checked similar occurrences and some of them are caused by a bad virus scanner (specifically 360Safe), so a wild-guess would be to turn it off and see if the crash will still occur.

So our only hope is to find something we can stick in a project. Without any pattern this will be finding a needle in a haystack. ☹️

JustOneMoreBlock commented 3 years ago

Welp time to poke it again. Because "specifically 360Safe" is not the answer. Especially on a Linux server.

SDGNelson commented 3 years ago

@JustOneMoreBlock Indeed... Since it seems to be GC-related it seems unlikely to be related to any particular bit of code, but on the other hand if it is exacerbated by a particular plugin then that might narrow it down. I know you already tried disabling uEssentials, but if you try disabling a few of the rest at a time I wonder if it happens less frequently?

The reason I think it is worth experimenting with plugins is that a lot of plugin code is more exotic than the base game. i.e. the base game sticks to an older c# featureset, older .NET functionality, and built-in Unity equivalents, whereas plugins are using tasks/async/who knows what.

gegtor commented 3 years ago

Here is the approximate frequency at which my servers crash^ If anyone is interested here is my system monitor maybe there is something in there http://149.86.225.213:19999

I will start correlating the data and remove all but necessary plugins on different servers

WCPistolPete commented 3 years ago

If it were truly plugin related would not the server crashes occur with equal frequency on both linux and windows systems? The change from Unity2018 to Unity2019 is what introduced the instability with the linux-based servers.

SDGNelson commented 3 years ago

If it were truly plugin related would not the server crashes occur with equal frequency on both linux and windows systems? The change from Unity2018 to Unity2019 is what introduced the instability with the linux-based servers.

I do not dispute that the instability was introduced somewhere between Unity 2018 and Unity 2019. It is not necessarily plugin related, but if there is a plugin that exacerbates the frequency then that would help narrow down the code to include in a reproducible project.

gegtor commented 3 years ago

@TH3AL3X You seem to not have that issue Is it possible that I give you my server files and you run them for a day or two to see if it's environment-related?

I'm also willing to run someone else's files who don't have the issue to see if it happens

TH3AL3X commented 3 years ago

@TH3AL3X You seem to not have that issue Is it possible that I give you my server files and you run them for a day or two to see if it's environment-related?

I'm also willing to run someone else's files who don't have the issue to see if it happens

Add me on discord, TERROR#1337

gegtor commented 3 years ago

I run multiple server instances from one game install

TH3AL3X Runs them from few separate directories

I just moved 4 out of 8 servers to fresh installs of the game and will see if the crash frequency drops or it fixes the issue completely

DronXa commented 3 years ago

I run multiple server instances from one game install

TH3AL3X Runs them from few separate directories

I just moved 4 out of 8 servers to fresh installs of the game and will see if the crash frequency drops or it fixes the issue completely

Any updates ?

gegtor commented 3 years ago

Still crashes :<

JustOneMoreBlock commented 3 years ago

Nothing new with RocketMod and running Plugins:

Unexpected mark stack overflow
Caught fatal signal - signo:6 code:-6 errno:0 addr:0xb70
Obtained 21 stack frames.
#0  0x007f6e137e0980 in funlockfile
#1  0x007f6e1341bfb7 in gsignal
#2  0x007f6e1341d921 in abort
#3  0x007f6e12a8b74e in GC_push_all
#4  0x007f6e12a80e56 in monoeg_g_hash_table_foreach
#5  0x007f6e12a6a521 in mono_unity_liveness_calculation_end
#6  0x007f6e12a92980 in GC_mark_some
#7  0x007f6e12a92bd4 in GC_stopped_mark
#8  0x007f6e12a93a4b in GC_maybe_gc
#9  0x007f6e12a93b53 in GC_collect_a_little_inner
#10 0x007f6e12a945bd in GC_collect_a_little
#11 0x007f6e149a0d61 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#12 0x007f6e148397f2 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#13 0x007f6e14839986 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#14 0x007f6e14899f01 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#15 0x007f6e1489421e in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#16 0x007f6e14894261 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#17 0x007f6e1489479a in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#18 0x007f6e149fd5ae in PlayerMain(int, char**)
#19 0x007f6e133febf7 in __libc_start_main
#20 0x0055fdc9d5f699 in _init
./ServerHelper.sh: line 29:  2928 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

I did an strace -p PID on Unturned with RocketMod and complete vanilla and I'm just spammed with this:

clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803779730}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803816630}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803856532}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803895756}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803927107}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=803958116}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804069476}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804106248}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804138274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804169772}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804217284}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804294942}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804380215}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804449763}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804539039}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804589975}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804638881}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804695388}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804743333}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804787191}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804829419}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804889723}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804937782}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=804999777}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805057019}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805126151}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805168930}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805211985}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805245363}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805277053}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805308976}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805343177}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805380190}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805414432}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805446031}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805477121}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805508398}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805550294}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805589404}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805622234}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805699212}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805731393}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805764258}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805796863}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805849624}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805887924}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805920964}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=805960729}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806020162}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806053545}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806095852}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806130793}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806174440}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806220195}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806263648}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806309227}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806353173}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806396171}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806441957}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806486701}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806531746}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806575204}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806627007}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806677557}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806723031}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806815180}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=806862723}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=807051032}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=807375861}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=807418196}) = 0
nanosleep({tv_sec=0, tv_nsec=9692000}, NULL) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817419206}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817473681}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817512668}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817551073}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817588803}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817620102}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817667004}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817719254}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817769694}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=817834063}) = 0
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 5) = 5
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 5) = 5
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=818222967}) = 0
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
sched_yield()                           = 0
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 1) = 1
sched_yield()                           = 0
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
futex(0x7f6d901a8880, FUTEX_WAKE_PRIVATE, 2) = 2
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819502047}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819542948}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819607427}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819727145}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819758972}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819790249}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819827137}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819866050}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819902664}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819936560}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=819972771}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=820010865}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=820045904}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=820079062}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=820118939}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=820155110}) = 0
recvfrom(7, 0x7ffca8ee0590, 1400, 0, 0x7ffca8ee0580, [16]) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=821741454}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=821798303}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=821850490}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=821901441}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822431162}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822476576}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822513078}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822626288}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822671588}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822708728}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822744664}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822846357}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822887811}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822921997}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822959394}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=822994890}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823036327}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823072911}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823108758}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823202777}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823244191}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823279522}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823313776}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823357034}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823388475}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823419572}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823449799}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823484430}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823516232}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823545224}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823576971}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823609785}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823651318}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823694431}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823736027}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823772748}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823810355}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823847652}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823889125}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823929594}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=823999629}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824060506}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824110878}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824152267}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824190602}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824231959}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824265464}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824297692}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824327962}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824358329}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824393125}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824426184}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824458678}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824489627}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824524380}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824558139}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824589910}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824623997}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824663676}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824715961}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824793106}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824831353}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824866090}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824922362}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=824965037}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825003866}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825040864}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825090273}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825123421}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825160228}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825198796}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825238398}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825303193}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825347657}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825388478}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825428787}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825465560}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825504283}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825622022}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825680152}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825890751}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825936501}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {tv_sec=5049890, tv_nsec=825996477}) = 0

Does the same thing with or without a screen. So, I made sure wasn't that, and I made sure time was updated on the server. Not sure if this means anything. Worth noting.

I'm testing complete vanilla, no rocket mod on a different server for any results.
I'm testing with rocketmod, however, no plugins on my main server to see if it produces the same errors.

Will have more information.

gegtor commented 3 years ago

I tested few servers with different concoctions of plugins and they all behave the same

Few notes

Servers are more stable when they have players on (maybe it's memory related?)
Sometimes I get the same stack trace when for example shutting down a server

I'm now testing vanilla OpenMod instance (without rocketmod)

gegtor commented 3 years ago

I found something interesting

The server sometimes crashes when I launch it or try to close it How can it crash when even the plugins haven't loaded yet? Or when it was just fine but something about the shutdown process caused it to crash

This leads me to believe that this is unrelated to plugins

Signal 11 SIGSEGV means that the program accessed a memory location that was not assigned to it Maybe there is a piece of code that tries to access something relating other running servers or something like that

I will try to run an exact copy of my server on a different machine solo without any other instance

GfxDevice: creating device client; threaded=0
NullGfxDevice:
    Version:  NULL 1.0 [1.0]
    Renderer: Null Device
    Vendor:   Unity Technologies
Begin MonoManager ReloadAssembly
- Completed reload, in  0.057 seconds
WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
UnloadTime: 0.667813 ms
3.21.1.0
Successfully set queue size to 4!
Successfully set name to [EU] S4.RAID.CITY [VANILLA] [SLOW TPA-HOME]!
Failed to find parameters in format [Text]/[R]/[G]/[B].
Successfully set port to 27045!
Successfully set max players to 24!
Successfully hid admins.
Successfully set map to Washington!
Successfully set gamemode to easy!
Successfully set perspective to both!
Caught fatal signal - signo:11 code:1 errno:0 addr:0x7f22c3628eb0
Obtained 36 stack frames.
#0  0x007f2385272980 in funlockfile
#1  0x007f238451fb99 in GC_mark_from
#2  0x007f2384524a21 in GC_mark_some
#3  0x007f2384525ad3 in GC_collect_a_little_inner
#4  0x007f2384525fa2 in GC_allocobj
#5  0x007f2384526dd4 in GC_generic_malloc_inner
#6  0x007f2384529c4c in GC_generic_malloc
#7  0x007f2384529eb4 in GC_malloc_kind_global
#8  0x007f23844fcfff in mono_gc_register_root
#9  0x007f23844b2f07 in mono_object_new_fast
#10 0x007f23844de0a8 in mono_verify_corlib
#11 0x007f23844f7239 in mono_reflection_get_custom_attrs_info
#12 0x00000041a81eb7 in (wrapper managed-to-native) System.MonoCustomAttrs:GetCustomAttributesInternal (System.Reflection.ICustomAttributeProvider,System.Type,bool)
#13 0x00000041b48d94 in SDG.Framework.Debug.TerminalCommandAttributeFinder:initialize ()
#14 0x00000041a86aa4 in SDG.Unturned.Setup:Awake ()
#15 0x007f2384341695 in mono_print_method_from_ip
#16 0x007f23844b1128 in mono_perfcounter_foreach
#17 0x007f23844b1fd5 in mono_runtime_invoke
#18 0x007f2386438d12 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#19 0x007f23864370ea in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#20 0x007f2386437005 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#21 0x007f238646c718 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#22 0x007f238641f299 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#23 0x007f238641f724 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#24 0x007f238641f1d7 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#25 0x007f2386482009 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, int> >*)
#26 0x007f2386337e1f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#27 0x007f2386337a8f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#28 0x007f2386337713 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#29 0x007f2386338afb in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#30 0x007f2386339147 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#31 0x007f238633e2af in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#32 0x007f238632b492 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#33 0x007f238648f4f7 in PlayerMain(int, char**)
#34 0x007f2384e90bf7 in __libc_start_main
#35 0x0055fefe7d5699 in _init
./start.sh: line 31:  3097 Aborted                 (core dumped) ./Unturned_Server/Unturned_Headless.x86_64 -batchmode -nographics +InternetServer/"$@"
senpai@vps:~$

JustOneMoreBlock commented 3 years ago

After @gegtor mentioned his post. I went down the complete vanilla route.

I ran my Vanilla server for days with no crashes at all, however, there was only one server. I copied a Commands.dat config from my main server maybe would provide some form of insight.

name JustPlayHere | Test 1
maxplayers 24
#bind ip_here
PVE
port 27070
map Washington
perspective both
Timeout 10000

Bind was removed a long time ago for some unknown dumb reason. This setting was ignored. I removed it in any case to be sure this wasn't the issue.

I started another server as PvE2 and that loaded fine, and no crashes. I wanted it to crash. So, I just made it the same Port as the same test server.

Set current directory to /home/Games/unturned
Found path: /home/Games/unturned/Unturned_Headless.x86_64
Mono path[0] = '/home/Games/unturned/Unturned_Headless_Data/Managed'
Mono config path = '/home/Games/unturned/Unturned_Headless_Data/MonoBleedingEdge/etc'
Preloaded 'libsteam_api.so'
Initialize engine version: 2019.4.17f1 (667c8606c536)
[Subsystems] Discovering subsystems at path /home/Games/unturned/Unturned_Headless_Data/UnitySubsystems
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0
NullGfxDevice:
    Version:  NULL 1.0 [1.0]
    Renderer: Null Device
    Vendor:   Unity Technologies
Begin MonoManager ReloadAssembly
- Completed reload, in  0.114 seconds
WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
UnloadTime: 1.330760 ms
3.21.1.0
Successfully set name to JustPlayHere | Test 1!
Successfully set max players to 24!
Successfully enabled PvE.
Successfully set port to 27070!
Successfully set map to Washington!
Successfully set perspective to both!
Successfully set timeout to 10000!
Caught fatal signal - signo:11 code:1 errno:0 addr:0x7f064c590e40
Obtained 35 stack frames.
#0  0x007f07090cc980 in funlockfile
#1  0x007f0708379b99 in GC_mark_from
#2  0x007f070837ea21 in GC_mark_some
#3  0x007f070837fad3 in GC_collect_a_little_inner
#4  0x007f070837ffa2 in GC_allocobj
#5  0x007f0708380dd4 in GC_generic_malloc_inner
#6  0x007f0708383c4c in GC_generic_malloc
#7  0x007f0708383eb4 in GC_malloc_kind_global
#8  0x007f0708356fff in mono_gc_register_root
#9  0x007f070830cf07 in mono_object_new_fast
#10 0x007f070830cf6e in mono_array_new_specific
#11 0x0000004040541f in (wrapper managed-to-native) object:__icall_wrapper_ves_icall_array_new_specific (intptr,int)
#12 0x000000404f6a74 in SDG.Framework.Debug.TerminalCommandAttributeFinder:initialize ()
#13 0x00000040437aa4 in SDG.Unturned.Setup:Awake ()
#14 0x007f070819b695 in mono_print_method_from_ip
#15 0x007f070830b128 in mono_perfcounter_foreach
#16 0x007f070830bfd5 in mono_runtime_invoke
#17 0x007f070a292d12 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#18 0x007f070a2910ea in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#19 0x007f070a291005 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#20 0x007f070a2c6718 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#21 0x007f070a279299 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#22 0x007f070a279724 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#23 0x007f070a2791d7 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#24 0x007f070a2dc009 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, int> >*)
#25 0x007f070a191e1f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#26 0x007f070a191a8f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#27 0x007f070a191713 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#28 0x007f070a192afb in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#29 0x007f070a193147 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#30 0x007f070a1982af in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#31 0x007f070a185492 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#32 0x007f070a2e94f7 in PlayerMain(int, char**)
#33 0x007f0708ceabf7 in __libc_start_main
#34 0x0055d66ec9f699 in _init
./ServerHelper.sh: line 29: 25021 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[screen is terminating]

No Processes are running at all and I still get this error regardless when trying to start the server. After waiting a few minutes I'm able to actually run it.

Now I attempt to start Server 2 and it starts loading, however, crashes at the end.

RecordSteamInterfaceCreation (PID 25139): SteamGameServer013 / GameServer
RecordSteamInterfaceCreation (PID 25139): SteamUtils009 / Utils
RecordSteamInterfaceCreation (PID 25139): SteamNetworking006 / Networking
RecordSteamInterfaceCreation (PID 25139): SteamGameServerStats001 / GameServerStats
RecordSteamInterfaceCreation (PID 25139): STEAMHTTP_INTERFACE_VERSION003 / HTTP
RecordSteamInterfaceCreation (PID 25139): STEAMINVENTORY_INTERFACE_V003 / Inventory
RecordSteamInterfaceCreation (PID 25139): STEAMUGC_INTERFACE_VERSION014 / UGC
RecordSteamInterfaceCreation (PID 25139): STEAMAPPS_INTERFACE_VERSION008 / Apps
[S_API FAIL] Tried to access Steam interface SteamNetworkingUtils003 before SteamAPI_Init succeeded.
RecordSteamInterfaceCreation (PID 25139): SteamNetworkingUtils003 / 
RecordSteamInterfaceCreation (PID 25139): SteamNetworkingSockets008 / 
Warning: failed to init SDL thread priority manager: SDL not found
This server will automatically save every 1800 seconds
Setting up 1 worker threads for Enlighten.
  Thread -> id: 7f9c37fff700 -> priority: 1 
src/tier1/fileio.cpp (4914) : Assertion Failed: s_bExit
src/tier1/fileio.cpp (4914) : Assertion Failed: s_bExit
src/tier1/fileio.cpp (4916) : Assertion Failed: m_vecRegisteredWriters.Count() == 0
src/tier1/fileio.cpp (4916) : Assertion Failed: m_vecRegisteredWriters.Count() == 0
Bad thread localsrc/tier1/fileio.cpp (4976) : Assertion Failed: CFileWriterThread already exited
src/tier1/fileio.cpp (4976) : Assertion Failed: CFileWriterThread already exited
Bad thread localBad thread localsrc/tier1/fileio.cpp (4976) : Assertion Failed: CFileWriterThread already exited
src/tier1/fileio.cpp (4976) : Assertion Failed: CFileWriterThread already exited
Bad thread local
[screen is terminating]

This is to be expected as the ip/port is already bound. I try starting it again and do the Mounting Core Files, and then do CTRL+C I `gracefully shutdown kill the server, however, this shows a similar error.

Mounting core: 0%
Mounting core: 1%
Mounting core: 2%
Mounting core: 3%
Mounting core: 4%
Mounting core: 5%
Mounting core: 6%
Mounting core: 7%
Mounting core: 8%
Mounting core: 9%
Mounting core: 10%
Mounting core: 11%
Handling SIGINT or SIGBREAK by requesting a graceful shutdown
Mounting core: 12%
Mounting core: 13%
Mounting core: 14%
Mounting core: 15%
Setting up 1 worker threads for Enlighten.
  Thread -> id: 7fb7d3550700 -> priority: 1 
Caught fatal signal - signo:11 code:1 errno:0 addr:0x250
Obtained 18 stack frames.
#0  0x007fb895d3e980 in funlockfile
#1  0x007fb896eefdf9 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#2  0x007fb896eec36e in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#3  0x007fb896f041eb in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#4  0x007fb89675a643 in _init
#5  0x007fb896759c73 in _init
#6  0x007fb896b6b5ce in std::pair<std::_Rb_tree_iterator<int>, bool> std::_Rb_tree<int, int, std::_Identity<int>, std::less<int>, std::allocator<int> >::_M_insert_unique<int const&>(int const&)
#7  0x007fb896b6b41c in std::pair<std::_Rb_tree_iterator<int>, bool> std::_Rb_tree<int, int, std::_Identity<int>, std::less<int>, std::allocator<int> >::_M_insert_unique<int const&>(int const&)
#8  0x007fb8969c4400 in std::pow(float, float)
#9  0x007fb8969c4569 in std::pow(float, float)
#10 0x007fb896da6309 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#11 0x007fb896da65f2 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#12 0x007fb896da666f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#13 0x007fb896da6828 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#14 0x007fb896da5d20 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#15 0x007fb896e7e2fa in std::_Rb_tree<void const*, void const*, std::_Identity<void const*>, std::less<void const*>, std::allocator<void const*> >::_M_erase(std::_Rb_tree_node<void const*>*)
#16 0x007fb895d336db in start_thread
#17 0x007fb895a5c71f in clone
./ServerHelper.sh: line 29: 25219 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[screen is terminating]

and then again...

cp: cannot create regular file 'Unturned_Headless_Data/Plugins/x86_64/steamclient.so': No such file or directory
Set current directory to /home/Games/unturned
Found path: /home/Games/unturned/Unturned_Headless.x86_64
Mono path[0] = '/home/Games/unturned/Unturned_Headless_Data/Managed'
Mono config path = '/home/Games/unturned/Unturned_Headless_Data/MonoBleedingEdge/etc'
Preloaded 'libsteam_api.so'
Initialize engine version: 2019.4.17f1 (667c8606c536)
[Subsystems] Discovering subsystems at path /home/Games/unturned/Unturned_Headless_Data/UnitySubsystems
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0
NullGfxDevice:
    Version:  NULL 1.0 [1.0]
    Renderer: Null Device
    Vendor:   Unity Technologies
Begin MonoManager ReloadAssembly
- Completed reload, in  0.131 seconds
WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
UnloadTime: 2.011991 ms
3.21.1.0
Successfully set name to JustPlayHere | Test 2!
Successfully set max players to 24!
Successfully enabled PvE.
Successfully set port to 27070!
Successfully set map to Washington!
Successfully set perspective to both!
Successfully set timeout to 10000!
Caught fatal signal - signo:11 code:1 errno:0 addr:0x7fa5cd820670
Obtained 35 stack frames.
#0  0x007fa69e361980 in funlockfile
#1  0x007fa69d60eb99 in GC_mark_from
#2  0x007fa69d613a21 in GC_mark_some
#3  0x007fa69d614ad3 in GC_collect_a_little_inner
#4  0x007fa69d614fa2 in GC_allocobj
#5  0x007fa69d615dd4 in GC_generic_malloc_inner
#6  0x007fa69d618c4c in GC_generic_malloc
#7  0x007fa69d618eb4 in GC_malloc_kind_global
#8  0x007fa69d5ebfb2 in mono_gc_register_root
#9  0x007fa69d5a1f07 in mono_object_new_fast
#10 0x007fa69d5a1f6e in mono_array_new_specific
#11 0x0000004173e41f in (wrapper managed-to-native) object:__icall_wrapper_ves_icall_array_new_specific (intptr,int)
#12 0x0000004182fa74 in SDG.Framework.Debug.TerminalCommandAttributeFinder:initialize ()
#13 0x00000041770aa4 in SDG.Unturned.Setup:Awake ()
#14 0x007fa69d430695 in mono_print_method_from_ip
#15 0x007fa69d5a0128 in mono_perfcounter_foreach
#16 0x007fa69d5a0fd5 in mono_runtime_invoke
#17 0x007fa69f527d12 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#18 0x007fa69f5260ea in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#19 0x007fa69f526005 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#20 0x007fa69f55b718 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#21 0x007fa69f50e299 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#22 0x007fa69f50e724 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#23 0x007fa69f50e1d7 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#24 0x007fa69f571009 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, int> >*)
#25 0x007fa69f426e1f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#26 0x007fa69f426a8f in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#27 0x007fa69f426713 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#28 0x007fa69f427afb in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#29 0x007fa69f428147 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#30 0x007fa69f42d2af in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#31 0x007fa69f41a492 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#32 0x007fa69f57e4f7 in PlayerMain(int, char**)
#33 0x007fa69df7fbf7 in __libc_start_main
#34 0x00560aa7989699 in _init
./ServerHelper.sh: line 29: 25285 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

[screen is terminating]

Almost like the port(s) are not fully dying or something before the next restart? This above is probably all non-sense, however, just me testing.

So, I've tried forcing killing (no luck, really) and using debugging tools like strace and recently valgrind. Trying to identify the issue.

screen -dmS ${server} valgrind --leak-check=full --show-leak-kinds=all --sigill-diagnostics=yes -v ./ServerHelper.sh +InternetServer/${server}

They were a lot of errors and I guess that was to be expected. The one thing I never noticed before was UnityPlayer.so spamming a lot via valgrind and in my original post is says something along the lines of PlayerMain(int, char**)

I've done some Googling and I found two things:

https://forum.unity.com/threads/error-fatal-signal-11-sigsegv.143753/#post-1062013 (I doubt this will be this is related, however, something to look into.)
Just Google UnityPlayer.so and you will see a few bugs from 2020 just from this file alone and might be our issue here.

Like the steamclient.so issue from previous Linux Crashes. This was as simple as copying the steamclient.so to /lib or /lib64/ to be safe as a workaround. Unfortunately, We are still forced to do that. This might be a simular situation with UnityPlayer.so as it's in the root directory of unturned.

Unturned_Headless_Data/Plugins/x86_64/?
/lib64/?

root@unturned_testing:/home/Games/unturned# file steamclient.so 
steamclient.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=8b47cc720d8774629ef6e13a5439a0aa461c3a8d, not stripped

root@unturned_testing:/home/Games/unturned# file UnityPlayer.so 
UnityPlayer.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=3184abede7cc33020c7c82aeb8a271b7f8f48079, stripped

gegtor commented 3 years ago

@SDGNelson https://forum.unity.com/threads/linux-build-stop-runing-on-unity-2019-3.806247/

Did crashes start after update to 2019.3?

SDGNelson commented 3 years ago

@SDGNelson https://forum.unity.com/threads/linux-build-stop-runing-on-unity-2019-3.806247/

Did crashes start after update to 2019.3?

Yes, these crashes are definitely caused by some difference between Unity 2018 LTS and 2019 LTS. The issue you linked however is from a developer trying to run the Linux version of their game without the required UnityPlayer.so.

Regarding UnityPlayer.so: it is being loaded correctly, otherwise the game/server would not be able to run. I think it is safe to assume that it contains the majority of Unity's native code, and that the executable is a wrapper for it. Similarly UnityPlayer.dll is wrapped by Unturned.exe on Windows.

Interesting that you got a similar (mono) crash multiple times during reflection. I will see if I can make something reproducible by generating tons of garbage through reflection.

gegtor commented 3 years ago

🎉 I can confirm crashes are not present Linux Kernel 5.xx 🎉

After upgrading to Ubuntu 20.04 crashes stoped

Ubuntu 18.04 with the upgraded kernel to 5.10 also doesn't crash

This leads me to believe that Unity aims for kernels 5.xx with their newest updates

SDGNelson commented 3 years ago

Hooray! Hopefully that does the trick for everyone else too. 🎉

Forgot to comment, I did experiment with generating garbage through reflection in an empty project, but did not have any luck recreating the crashes. Sounds like that does not matter anymore though.

JustOneMoreBlock commented 3 years ago

I can confirm crashes are not present Linux Kernel 5.xx

After upgrading to Ubuntu 20.04 crashes stoped

Ubuntu 18.04 with the upgraded kernel to 5.10 also doesn't crash

This leads me to believe that Unity aims for kernels 5.xx with their newest updates

Unfortunately, the error still persists for me. Upgraded OS...

root@ut:~# lsb_release
No LSB modules are available.
root@ut:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

root@ut:~# uname -r
5.4.0-65-generic

This copied and pasted. I see a typo below. :) Not sure if it's Unturned or a Plugin.

Data sucessfully saved.
Successfully saved the game.
Unexpected mark stack overflow
Caught fatal signal - signo:6 code:-6 errno:0 addr:0xaee
Obtained 19 stack frames.
#0  0x007f5532dfc3c0 in funlockfile
#1  0x007f5532c3b18b in gsignal
#2  0x007f5532c1a859 in abort
#3  0x007f553269674e in GC_push_all
#4  0x007f553268be56 in monoeg_g_hash_table_foreach
#5  0x007f5532675521 in mono_unity_liveness_calculation_end
#6  0x007f553269d980 in GC_mark_some
#7  0x007f553269ead3 in GC_collect_a_little_inner
#8  0x007f553269f5bd in GC_collect_a_little
#9  0x007f553397f141 in std::vector<unsigned int, std::allocator<unsigned int> >::_M_fill_insert(__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int, std::allocator<unsigned int> > >, unsigned long, unsigned int const&)
#10 0x007f5533817bd2 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#11 0x007f5533817d66 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#12 0x007f55338782e1 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#13 0x007f55338725fe in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#14 0x007f5533872641 in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#15 0x007f5533872b7a in int* std::_V2::__rotate<int*>(int*, int*, int*, std::random_access_iterator_tag)
#16 0x007f55339db98e in PlayerMain(int, char**)
#17 0x007f5532c1c0b3 in __libc_start_main
#18 0x0055b5228bf699 in _init
./ServerHelper.sh: line 29:  2798 Aborted                 (core dumped) ./Unturned_Headless.x86_64 -batchmode -nographics "$@"

gegtor commented 3 years ago

This is weird I moved all my files to 20.04 and crashes magically stoped

Is it a VPS or a dedicated server?

JustOneMoreBlock commented 3 years ago

This is weird I moved all my files to 20.04 and crashes magically stoped

Is it a VPS or a dedicated server?

Dedicated. I'm trying another solution.

If that doesn't work. I'll format and try that. I do have a theory.

You moved your server to a 20.04 install. Possibly re-installing Unturned and getting new files from the steam library?
The error above is a mono error, and I'm wondering maybe something in the upstream isn't downloaded when updating.
Doubt it. If that is TRUE then OS doesn't matter and was a steam issue.

gegtor commented 3 years ago

I moved only the Servers/ folder and did a clean install

I used my scripts and added openmod and it runs okay

I still sometimes get the crash on server start you mentioned

JustOneMoreBlock commented 3 years ago

I moved only the Servers/ folder and did a clean install

I used my scripts and added openmod and it runs okay

I still sometimes get the crash on server start you mentioned

I'm still using rocketmod and I've considered moving to open mod. I get it all the time, so I guess is to break out strace again.

Trojaner commented 3 years ago

The error above is a mono error, and I'm wondering maybe something in the upstream isn't downloaded when updating.

Doubt it. If that is TRUE then OS doesn't matter and was a steam issue.

Unturned does not use the installed mono. It ships its own mono.

JustOneMoreBlock commented 3 years ago

The error above is a mono error, and I'm wondering maybe something in the upstream isn't downloaded when updating.

Doubt it. If that is TRUE then OS doesn't matter and was a steam issue.

Unturned does not use the installed mono. It ships its own mono.

And? I never said installed mono on the server. The point was making is the mono shipped could be bad.

JustOneMoreBlock commented 3 years ago

@SDGNelson I've attached the following:

Excerpt:

munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
read(42, "7f4c6979f000-7f4c697a3000 rw-p 0"..., 4096) = 3333
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=29476392, ...}) = 0
mmap(NULL, 29476392, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a7cfe1000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
read(42, "7f4c6979f000-7f4c697a3000 rw-p 0"..., 4096) = 3333
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=29476392, ...}) = 0
mmap(NULL, 29476392, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a7cfe1000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
read(42, "7f4c6979f000-7f4c697a3000 rw-p 0"..., 4096) = 3333
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=29476392, ...}) = 0
mmap(NULL, 29476392, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a7cfe1000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
read(42, "7f4c6979f000-7f4c697a3000 rw-p 0"..., 4096) = 3333
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=29476392, ...}) = 0
mmap(NULL, 29476392, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a7cfe1000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
read(42, "7f4c6979f000-7f4c697a3000 rw-p 0"..., 4096) = 3333
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=29476392, ...}) = 0
mmap(NULL, 29476392, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a7cfe1000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/UnityPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4a7cfe1000, 29476392)        = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
read(42, "7f4aed32f000-7f4aed330000 rw-p 0"..., 4096) = 4067
read(42, "7f4b34021000-7f4b38000000 ---p 0"..., 4096) = 4078
read(42, "7f4b951bc000-7f4b951bd000 ---p 0"..., 4096) = 4080
read(42, "7f4b96526000-7f4b96536000 rwxp 0"..., 4096) = 4050
read(42, "7f4bf04c0000-7f4bf0500000 ---p 0"..., 4096) = 4004
read(42, "7f4c68008000-7f4c6800c000 r--p 0"..., 4096) = 4061
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc-2.31.so", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=2029224, ...}) = 0
mmap(NULL, 2029224, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4a8c00c000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc-2.31.so", O_RDONLY) = 42
fstat(42, {st_mode=S_IFREG|0755, st_size=2029224, ...}) = 0
mmap(NULL, 2029224, PROT_READ, MAP_PRIVATE, 42, 0) = 0x7f4a7ea0e000
close(42)                               = 0
munmap(0x7f4a8c00c000, 2029224)         = 0
munmap(0x7f4a7ea0e000, 2029224)         = 0
getpid()                                = 88224
openat(AT_FDCWD, "/proc/88224/maps", O_RDONLY) = 42
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c60002000
read(42, "4047b000-4048b000 rwxp 00000000 "..., 4096) = 4047
openat(AT_FDCWD, "/home/Games/unturned/Unturned_Headless.x86_64", O_RDONLY) = 43
fstat(43, {st_mode=S_IFREG|0755, st_size=6344, ...}) = 0
mmap(NULL, 6344, PROT_READ, MAP_PRIVATE, 43, 0) = 0x7f4c60000000
close(43)                               = 0
close(42)                               = 0
munmap(0x7f4c60002000, 4096)            = 0
openat(AT_FDCWD, "/home/Games/unturned/LinuxPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/Games/unturned/.debug/LinuxPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug/home/Games/unturned/LinuxPlayer_s.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
munmap(0x7f4c60000000, 6344)            = 0
write(1, "Obtained 19 stack frames.\n", 26) = 26
write(1, "#0  0x007f4c697b83c0 in funlockf"..., 36) = 36
write(1, "#1  0x007f4c695f718b in gsignal\n", 32) = 32
write(1, "#2  0x007f4c695d6859 in abort\n", 30) = 30
write(1, "#3  0x007f4c6905274e in GC_push_"..., 36) = 36
write(1, "#4  0x007f4c69047e56 in monoeg_g"..., 52) = 52
write(1, "#5  0x007f4c69031521 in mono_uni"..., 60) = 60
write(1, "#6  0x007f4c69059980 in GC_mark_"..., 37) = 37
write(1, "#7  0x007f4c6905aad3 in GC_colle"..., 50) = 50
write(1, "#8  0x007f4c6905b5bd in GC_colle"..., 44) = 44
write(1, "#9  0x007f4c6a33b141 in std::vec"..., 237) = 237
write(1, "#10 0x007f4c6a1d3bd2 in int* std"..., 105) = 105
write(1, "#11 0x007f4c6a1d3d66 in int* std"..., 105) = 105
write(1, "#12 0x007f4c6a2342e1 in int* std"..., 105) = 105
write(1, "#13 0x007f4c6a22e5fe in int* std"..., 105) = 105
write(1, "#14 0x007f4c6a22e641 in int* std"..., 105) = 105
write(1, "#15 0x007f4c6a22eb7a in int* std"..., 105) = 105
write(1, "#16 0x007f4c6a39798e in PlayerMa"..., 48) = 48
write(1, "#17 0x007f4c695d80b3 in __libc_s"..., 42) = 42
write(1, "#18 0x00556c65f30699 in _init\n", 30) = 30
rt_sigaction(SIGABRT, {sa_handler=SIG_DFL, sa_mask=[ABRT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f4c695f7210}, {sa_handler=0x7f4c68f6ca52, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f4c697b83c0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
getpid()                                = 88224
gettid()                                = 88224
tgkill(88224, 88224, SIGABRT)           = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=88224, si_uid=0} ---
+++ killed by SIGABRT (core dumped) +++

The Last 1000 Lines Full Log It's the same error before, however, this crashed naturally without any restarts.

root@ut:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

root@ut:~# uname -r
5.4.0-65-generic

I also tested this same error on CentOS 7. The crashes persist. So, this logically steams from an issue in Unturned or the UnityPlayer.so as you can see in the log.

gegtor commented 3 years ago

How is it possible that i don’t have any crashes and you do now

Maybe you wanna get in touch and I install the servers for you my way and we compare?

JustOneMoreBlock commented 3 years ago

Thanks for the offer, however, I'm a Linux Administrator by trade. I don't help to install things.

The Operating System doesn't matter. The same crash applies to Ubuntu 18.04, Ubuntu 20.04, and CentOS 7.
This seems clearly something wrong either with Unturned, Mono, and/or something new is missing on the installer.
I'm still on RocketMod and really should switch to Open Mod.

Old:

apt install screen htop unzip #Utils
apt install lib32stdc++6 #SteamCMD dependencies
apt install mono-runtime mono-reference-assemblies-2.0 #No longer required
apt install libc6 libgl1-mesa-glx libxcursor1 libxrandr2 # prerequisites for Unity 3D
apt install libc6-dev libgcc-4.8-dev # prequesites for BattlEye #libgcc-4.8-dev didn't exist for 20.04, however, libgcc-10-dev does?

This was originally required.

SDGNelson commented 3 years ago

Since gegtor you mentioned switching to Open Mod when it stopped crashing, I wonder JustOneMoreBlock if you use the Rocket plugin backwards compatibility in Open Mod whether it stops crashing?

Trojaner commented 3 years ago

I would like to mention that OpenMod removes the OpenMod RocketMod Bridge / Rocket plugin support soon when OpenMod 3 releases. We have noticed that this is too complicated to set up for most users. Instead OpenMod 3 can be installed side by side with RocketMod and will no longer try to replace it but integrate with instead.

WCPistolPete commented 3 years ago

Currently 2 servers of 13 running. Russia almost always crashes first, followed by one of the Washington maps. Yukon and the PEI servers soon follow. Arenas crash at random, but they've been bugged for years (ever since the Unity2018 update). https://justplayhere.com/status

TH3AL3X commented 3 years ago

I have managed to keep the two servers open, one in washington and another in pei, but in the PEI absolutely no one lets connect, a Steam network error jumps, this is very rare since it has not happened to me before, and I reinstalled the game nothing, I have the two separate servers in game instance, tried changing steamclient.so

TH3AL3X commented 3 years ago

Due some tests i just removed "-batchmode -nographics" and it's not crashing on startup...

TH3AL3X commented 3 years ago

Now i have the steam network error, it happens when i have 2 servers, like no one can't join...

DronXa commented 3 years ago

Has this issue been resolved for you guys ? I still seem to get the same issue since the update back in November. I am currently using Windows Server 2012 R2

TH3AL3X commented 3 years ago

Has this issue been resolved for you guys ? I still seem to get the same issue since the update back in November. I am currently using Windows Server 2012 R2

Nope

SDGNelson commented 3 years ago

DronXa's crash seems to be slightly different, but it is surprising that there are similar problems even on Windows.

These mono internal crashes are out of the game's control as far as I am aware.

The game is a few LTS releases behind (currently on 20f1 not 28f1) due to a bug with serialized properties in one of the newer releases (around 23f1 I think?), but I have made a note to check if that has been fixed for an LTS upgrade for next Friday.

TH3AL3X commented 3 years ago

For some reason the server turns on, that is, the two servers start up perfectly, but one of them does not connect to the steam servers, and it appears in the list but does not allow it to connect, it kicks you and you get a message about the steam problem networking, I don't know what to do anymore

TH3AL3X commented 3 years ago

It's so weird, cuz i can see the server, but i'm getting this error when i try to connect Server_servidorpei.log

SDGNelson commented 3 years ago

@TH3AL3X it looks like the server port is set to 27016. Is that intentional? Mind you it uses port +1 and +2 as well, so the ports may be overlapping. If you can attach the client log there may be further information.

TH3AL3X commented 3 years ago

Yes, you are right, it was a problem with the ports, it is strange because previously this did not happen to me, ty

SmartlyDressedGames / Unturned-3.x-Community

New Linux Crashes #2088