Has anyone been having issues with arena servers crashing?

[SDGNelson]

Got a report that arena servers have been crashing frequently after the 3.19.16 update, but haven't heard that from anyone else yet. If anyone is having issues with servers crashing please let me know and I'll try to help track it down!

[D-AIRY]

I've got some reports. Last crash log: screenlog.0.1.log

[SDGNelson]

@D-AIRY thanks for chiming in! That's an odd stacktrace (null exception in std::pow?), is there any .dmp file that might have more details? As a precaution I've tidied up most uses of Mathf.Pow in the codebase which typically fell into two scenarios: squaring numbers and measuring the horizontal distance between points, now moved into Math.square and Math.horizontalDistanceSquared.

[D-AIRY]

@SDGNelson I've got two more crashes with similar stacktrace. Now enabled core dump for servers.

[SDGNelson]

Any luck with the dumps? Hopefully we can track this down.

[D-AIRY]

No luck. ulimit -c unlimited didn't do the trick. Do you know how to force unity to dump core on crash?

[SDGNelson]

It usually creates a crash.dmp file next to the error.log and output_log.txt files. Is it creating those?

[D-AIRY]

I never had this files. As I know, *.dmp is windows-only thing.

[JustOneMoreBlock]

Okay, I figure I would take a look as I have been having this issue on our servers.

Set current directory to /home/Games/unturned
Found path: /home/Games/unturned/Unturned_Headless.x86_64
Mono path[0] = '/home/Games/unturned/Unturned_Headless_Data/Managed'
Mono config path = '/home/Games/unturned/Unturned_Headless_Data/MonoBleedingEdge/etc'
Preloaded 'libsteam_api.so'
Preloaded 'steamclient.so'
Initialize engine version: 2018.4.11f1 (7098af2f11ea)
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0
NullGfxDevice:
    Version:  NULL 1.0 [1.0]
    Renderer: Null Device
    Vendor:   Unity Technologies
Begin MonoManager ReloadAssembly
- Completed reload, in  0.049 seconds
WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
ERROR: Shader Shader is not supported on this GPU (none of subshaders/fallbacks are suitable)WARNING: Shader Unsupported: 'Hidden/Nature/Terrain/Utilities' - Setting to default shader.
UnloadTime: 0.546000 ms
3.19.17.0

[World] [Owner] Cory [Cory]: "/checkowner"
An error occured while executing checkowner []: System.Net.WebException: Error: TrustFailure (One or more errors occurred.) ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Security.Interface.TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED
  at Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, Mono.Unity.UnityTls+unitytls_x509verify_result verifyResult, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) [0x00036] in <d2957de1c3fd4781a43d89572183136c>:0 
  at Mono.Unity.UnityTlsContext.ProcessHandshake () [0x00082] in <d2957de1c3fd4781a43d89572183136c>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status) [0x0003e] in <d2957de1c3fd4781a43d89572183136c>:0 
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <d2957de1c3fd4781a43d89572183136c>:0 
  at Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () [0x000ff] in <d2957de1c3fd4781a43d89572183136c>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () [0x0008b] in <d2957de1c3fd4781a43d89572183136c>:0 
   --- End of inner exception stack trace ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at Mono.Net.Security.MobileAuthenticatedStream+<ProcessAuthentication>d__47.MoveNext () [0x00254] in <d2957de1c3fd4781a43d89572183136c>:0 
   --- End of inner exception stack trace ---
  at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00043] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at System.Threading.Tasks.Task.Wait () [0x00000] in <7ba07f088431485bb722f3b3373e87ee>:0 
  at Mono.Net.Security.MobileAuthenticatedStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000d] in <d2957de1c3fd4781a43d89572183136c>:0 
  at Mono.Net.Security.MonoTlsStream.CreateStream (System.Byte[] buffer) [0x0007b] in <d2957de1c3fd4781a43d89572183136c>:0 
  at System.Net.WebConnection.CreateStream (System.Net.HttpWebRequest request) [0x00073] in <d2957de1c3fd4781a43d89572183136c>:0 
   --- End of inner exception stack trace ---
  at System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) [0x00072] in <d2957de1c3fd4781a43d89572183136c>:0 
  at System.Net.WebClient.DownloadString (System.Uri address) [0x00020] in <d2957de1c3fd4781a43d89572183136c>:0 
  at System.Net.WebClient.DownloadString (System.String address) [0x00016] in <d2957de1c3fd4781a43d89572183136c>:0 
  at (wrapper remoting-invoke-with-check) System.Net.WebClient.DownloadString(string)
  at SDPlugins.Library.WebClientRequest (System.String url) [0x00007] in <0bb0be7364d94b76bf8c0f4e61bba887>:0 
  at SDPlugins.Library.SteamRequest (System.String input) [0x00011] in <0bb0be7364d94b76bf8c0f4e61bba887>:0 
  at SDPlugins.Library.TellInfo (Rocket.API.IRocketPlayer caller, Steamworks.CSteamID ownerid, Steamworks.CSteamID group) [0x0010d] in <0bb0be7364d94b76bf8c0f4e61bba887>:0 
  at SDPlugins.CheckOwner.Execute (Rocket.API.IRocketPlayer caller, System.String[] command) [0x00119] in <0bb0be7364d94b76bf8c0f4e61bba887>:0 
  at Rocket.Core.Commands.RocketCommandManager+RegisteredRocketCommand.Execute (Rocket.API.IRocketPlayer caller, System.String[] command) [0x00000] in <466fa14d466749559a64a11ae8acbc60>:0 
  at Rocket.Core.Commands.RocketCommandManager.Execute (Rocket.API.IRocketPlayer player, System.String command) [0x001b2] in <466fa14d466749559a64a11ae8acbc60>:0 
Updating WebXMLFileAsset RocketPermissions from http://justplayhere.com/unturned/permissions/unturned.php?instance=pve1
Successfully updated WebXMLFileAsset RocketPermissions from http://justplayhere.com/unturned/permissions/unturned.php?instance=pve1

Possibly a Mono error

Mono.Security.Interface.TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

Usually means something SSL.

I still had made patch enabled to make Unturned work with the steamclient.so files from the steamcmd directory and not the one that Unturned provides. I disabled my patch and got this error.

(Filename:  Line: 0)

BoxColliders does not support negative scale or size.
The effective box size has been forced positive and is likely to give unexpected collision geometry.
If you absolutely need to use negative scaling you can use the convex MeshCollider. Scene hierarchy path "Russia/Objects/359/Nav"

(Filename:  Line: 0)

BoxColliders does not support negative scale or size.
The effective box size has been forced positive and is likely to give unexpected collision geometry.
If you absolutely need to use negative scaling you can use the convex MeshCollider. Scene hierarchy path "Russia/Objects/626/Nav"

(Filename:  Line: 0)

BoxColliders does not support negative scale or size.
The effective box size has been forced positive and is likely to give unexpected collision geometry.
If you absolutely need to use negative scaling you can use the convex MeshCollider. Scene hierarchy path "Russia/Objects/623/Nav"

(Filename:  Line: 0)

BoxColliders does not support negative scale or size.
The effective box size has been forced positive and is likely to give unexpected collision geometry.
If you absolutely need to use negative scaling you can use the convex MeshCollider. Scene hierarchy path "Russia/Objects/624/Nav"

I know it's not Arena related, however, only the server came online and crashed the others and could be something about the BoxColliders causing a crash in general and in my patch I don't see this error, but could still be persistent.

[D-AIRY]

Mono.Security.Interface.TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

By default mono has empty ssl trust store.

Generic Debian / Ubuntu fix:

Installing the ca-certificates-mono package will add certificates. If possible, use the Mono supplied Debian packages instead of the Mono packages in APT as they are really outdated. If you install the packages from the Mono repository, they generally fix the certificate issues automatically.

If that is not an option, the release notes from Mono suggest running cert-sync:

sudo cert-sync /etc/ssl/certs/ca-certificates.crt

If things still does not work, you can experiment with the environment variable:

MONO_TLS_PROVIDER=legacy

You need at least the mono-runtime package and in some cases libmono-system-net-http4.0-cil to get SSL/TLS working.

[Trojaner]

Unity does not use the installed mono instance. It provides its own mono. Installing mono-runtime will have no effects.

Issue is a Unity bug that is still not fixed. https://forum.unity.com/threads/unity-2017-1-tls-1-2-still-not-working-with-net-4-6.487415/page-2

[D-AIRY]

I know that. But it somehow using ssl storage from the system mono installation.

[SDGNelson]

Re .dmp: Sorry, you are right. Searching online it looks like the log file is the only crash file we can use.

SSL validation: One alternative might be replacing usage with UnityWebRequest. Unturned makes trusted HTTPS requests using it, and you can override its certificate behavior to trust whatever you like: https://docs.unity3d.com/ScriptReference/Networking.UnityWebRequest-certificateHandler.html

Box colliders: That error is annoying, but harmless. They added it a few years ago, and I even asked them they would consider allowing it to be disabled. Unfortunately because the player uses box collider hitboxes and has negative scale for left-handed characters it pops up, and some map objects cause it too. Everything still works properly regardless.

[D-AIRY]

I've managed to make crush dump working. There are two dumps from different servers. What should I do next?

[SDGNelson]

@D-AIRY Oh awesome! If you can send them to nelson@smartlydressedgames.com I'll download them and see if we can get any helpful insights.

[D-AIRY]

@SDGNelson Sent

[SDGNelson]

Thanks to @D-AIRY for sending over those crash dumps! Sadly they seem to be some lower system-level dump information. Looking around online I didn't find anything about debug files for Unity on Linux, but if anyone else has any ideas I'd be happy to hear them.

I'm also keeping an eye out for the next LTS release to see if there are any relevant fixes.

[D-AIRY]

These core files are intended to use with linux debugger, e.g. gdb. Basic syntax like gdb /path/to/Unturned_Headless.x86_64 /path/to/corefile

Is there any way we can get development mono build with debug info enabled?

[SDGNelson]

Ah, thank you. It looks like the most straightforward way to do that is on Linux, so I'll see about setting up a Linux partition on my desktop this week. (in the past have used remote dedicated servers)

The source for current 2018.4 "bleeding edge" mono version is available, so compiling debug mode may be an option: https://github.com/Unity-Technologies/mono/tree/unity-2018.4-mbe

[SDGNelson]

Not particularly exciting news, but since my drives are almost out of space as-is I ordered an SSD which will be dedicated to Linux maintenance that should arrive tomorrow.

[SDGNelson]

@D-AIRY one thing I've been experimenting with today is using the 2019.2 version of Unity mono, and it seems to work okay. I'm considering adjusting the build process to copy in the latest stable versions. Here's a download link of the "development" 64-bit 2019.2 headless binaries:

https://drive.google.com/file/d/10gOQGhCGQIqDz7ixWAFmAe3ppGZw6GmE/view Swapping out the MonoBleedingEdge directory will use the 2019.2 mono.

Debugging the core dumps with gdb didn't get far without full symbols, but if we continue running into issues with these newer versions of mono I'll see if we can build mono independently to get debug symbols. I've also been getting reports of a few mono internal crashes on Windows, so I'm hopeful for the first option.

Now that I've got a Linux desktop setup I'm also going to write a simple Xlib OpenGL renderer to become more familiar with the platform for the future.

[D-AIRY]

The server won't start with this mono, crashes immediately crash.txt

[D-AIRY]

Returned old "unity default resources", got new errors, still crashes on startup. crash.txt

[SDGNelson]

Thanks for giving it a try! I was only swapping out the MonoBleedingEdge directory. If you use the old resources and DLLs with the new mono does it run without crashing? If it still crashes I wonder why, as it seemed okay on Ubuntu. What distro are you on?

[D-AIRY]

That works, I'll add this to servers that crashes, will see if that helps.

distro: Ubuntu server 16.04

[D-AIRY]

It's much more stable! Only few crashes during shutdown

[SDGNelson]

Thanks for the update! It might be worth making that the default then, I'll open a separate to-do issue for it.