search

Pryaxis / TShock

☕️⚡️TShock provides Terraria servers with server-side characters, anti-cheat, and community management tools.
GNU General Public License v3.0
2.41k stars 377 forks source link

Mono behavior is weird in 4.0. #935

Closed moozaad closed 8 years ago

moozaad commented 9 years ago

V4.2.10 I appears within a few seconds of the authcode yellow text.

UpdateManager Exception: System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00065] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:937
  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:949
  at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) [0x00000] in <filename unknown>:0

mono --version

Mono JIT compiler version 4.0.1 (tarball Fri May 29 00:26:38 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

After joining the game, no data is being sent from the server. Ports are open and TDSM runs fine.

moozaad commented 9 years ago

mono --debug --trace=N:nothing TerrariaServer.exe

output:

This token will display until disabled by verification. (/auth-verify)
: [0x7f74a33e4700:] EXCEPTION handling: System.Net.Sockets.SocketException: interrupted
[0x7f74a3bfe700:] EXCEPTION handling: System.Net.WebException: The request timed out
[0x7f74a3bfe700:] EXCEPTION handling: System.Net.WebException: The request timed out
UpdateManager Exception: System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00065] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:937
  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:949
  at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) [0x00000] in <filename unknown>:0
[0x7f74a3bfe700:] EXCEPTION handling: System.Net.WebException: The request timed out
hakusaro commented 9 years ago

This is an issue with Mono and our socket system timing out during the check. It's kinda normal behavior under mono.

moozaad commented 9 years ago

I closed it because there were other issues going on with mono 4 which I'm not sure where the issue resides.

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <IL 0x0000f, 0xffffffff>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <IL 0x00088, 0xffffffff>
  at Terraria.Player.UpdateBuffs (int) <0x00053>
  at Terraria.Player.UpdatePlayer (int) <IL 0x00a45, 0x02433>
  at Terraria.Main.Update () <IL 0x00194, 0x004c3>
  at Terraria.Main.DedServ () <IL 0x00779, 0x01217>
  at Terraria.ProgramServer.Main (string[]) <IL 0x000c0, 0x001c7>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00060, 0xffffffff>

Native stacktrace:

        mono() [0x4ba2d8]
        mono() [0x51046e]
        mono() [0x42b2e8]
        /lib64/libpthread.so.0(+0xf890) [0x7f2e10df1890]
        mono() [0x5d0830]
        mono() [0x5d16f6]
        mono() [0x5d6a2b]
        mono() [0x5d7449]
        mono() [0x5ed529]
        mono() [0x5ed7f3]
        [0x41c3559e]
hakusaro commented 9 years ago

screen shot 2015-05-31 at 10 14 54 am

I've had a lot of luck with this version of mono.

moozaad commented 9 years ago

The default version of mono 3.8 on opensuse 13.2 gave me different errors :dancers: 

Stacktrace:

  at <unknown> <0xffffffff>
  at Terraria.NPC.UpdateNPC (int) <IL 0x00bbe, 0x021af>
  at Terraria.Main.Update () <IL 0x002d6, 0x007a3>
  at Terraria.Main.DedServ () <IL 0x00779, 0x011ff>
  at Terraria.ProgramServer.Main (string[]) <IL 0x000c0, 0x001c3>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00050, 0xffffffff>

Native stacktrace:

        mono() [0x4bcaf8]
        /lib64/libpthread.so.0(+0xf890) [0x7fe89f598890]
        /lib64/libc.so.6(gsignal+0x37) [0x7fe89f217187]
        /lib64/libc.so.6(abort+0x118) [0x7fe89f218538]
        mono() [0x6465d5]
        mono() [0x646716]
        mono() [0x424161]
        mono() [0x4249bc]
        mono() [0x427d2d]
        mono() [0x4287eb]
        mono() [0x4bdb67]
        [0x41a83166]
hakusaro commented 9 years ago

3.12 masterrace.

moozaad commented 9 years ago

heh. I'll see if I can find a repos with that exact version. I'm not in the mood to compile it.

hakusaro commented 9 years ago

To be honest, it's definitely worth looking into why it doesn't work properly with 4.x, because that's the latest release.

hakusaro commented 9 years ago

@moozaad can you do me a favor, and test with this build? According to the Mono changelog, one of the major changes is the removal of support for binaries compiled against .NET 4.0. While we haven't updated the server version in quite some time, it's possible that now that we're building against 4.5, it will work.

moozaad commented 9 years ago

from the debug build in a fresh new directory.

starbound@metis:~/tshock-debug> mono --debug --trace=N:nothing TerrariaServer.exe
TerrariaAPI Version: 1.17.0.0 (Protocol v1.2.4.1)
[0x7fb2ae508780:] EXCEPTION handling: System.Reflection.ReflectionTypeLoadException: The classes in the module cannot be loaded.
[0x7fb2ae508780:] EXCEPTION handling: System.InvalidOperationException: Failed to load assembly "MySql.Web.dll".
[Server API] Error Startup aborted due to an exception in the Server API initialization:
System.InvalidOperationException: Failed to load assembly "MySql.Web.dll". ---> System.Reflection.ReflectionTypeLoadException: The classes in the module cannot be loaded.
  at (wrapper managed-to-native) System.Reflection.Assembly:GetTypes (System.Reflection.Assembly,bool)
  at System.Reflection.Assembly.GetExportedTypes () [0x00000] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System.Reflection/Assembly.cs:365
  at TerrariaApi.Server.ServerApi.LoadPlugins () [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at TerrariaApi.Server.ServerApi.LoadPlugins () [0x00000] in <filename unknown>:0
  at TerrariaApi.Server.ServerApi.Initialize (System.String[] commandLineArgs, Terraria.Main game) [0x00000] in <filename unknown>:0
  at Terraria.ProgramServer.Main (System.String[] args) [0x00000] in <filename unknown>:0
starbound@metis:~/tshock-debug> mono -V
Mono JIT compiler version 4.0.1 (tarball Fri May 29 00:26:38 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen
tylerjwatson commented 9 years ago

If I may chime in here. The ReflectionTypeLoadException is because of a version of the MySQL driver that doesn't like loading under mono, possibly because it may contain mixed-mode code but I'm not sure, and it doesn't warrant investigating when a legacy version works fine.

If you download the artefact from the server, also download a copy of TShock release, and replace MySql.Web.dll from that installation into the ServerPlugins/ directory, replacing the file that's there.

Tested on Fedora 22 with Mono JIT compiler version 4.0.1 (tarball Thu May 28 04:56:48 EDT 2015)

tylerjwatson commented 9 years ago

As a footnote, I have had some pthread-related weirdness in mono whilst running some code on production at work in all Mono series from 3.0 to 3.10 - I would highly recommend not using the distro versions of mono.

Use Xamarin's apt and RPM repos. They have proper release management.

moozaad commented 9 years ago

Replacing MySql.Web.dll as per @tylerjwatson suggestion worked but still errors further in. The webexception is still there and a fatal buff exception

Web timeouts

: [0x7f515e1fe700:] EXCEPTION handling: System.Net.WebException: The request timed out
[0x7f515e1fe700:] EXCEPTION handling: System.Net.WebException: The request timed out
UpdateManager Exception: System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00065] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpW                                                                                      ebRequest.cs:937
  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:949
  at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) [0x00000] in <filename unknown>:0
[0x7f515e1fe700:] EXCEPTION handling: System.Net.WebException: The request timed out

User exception. I presume this is throw on purpose

86.22.118.125:49608 is connecting...
[0x7f516e847700:] EXCEPTION handling: TShockAPI.DB.UserNotExistException: User 'testafro' does not exist
testafro has joined.

This one kills the server

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <IL 0x0000f, 0xffffffff>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <IL 0x00088, 0xffffffff>
  at Terraria.Player.UpdateBuffs (int) <0x00053>
  at Terraria.Player.UpdatePlayer (int) <IL 0x00ac7, 0x0241b>
  at Terraria.Main.Update () <IL 0x001b1, 0x004c3>
  at Terraria.Main.DedServ () <IL 0x007e1, 0x0125b>
  at Terraria.ProgramServer.Main (string[]) <IL 0x000c5, 0x001e7>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00060, 0xffffffff>

Native stacktrace:

        mono() [0x4ba2d8]
        mono() [0x51046e]
        mono() [0x42b2e8]
        /lib64/libpthread.so.0(+0xf890) [0x7f51927fc890]
        mono() [0x5d16ea]
        mono() [0x5d6a2b]
        mono() [0x5d7449]
        mono() [0x5ed529]
        mono() [0x5ed7f3]
        [0x40e6254e]
tylerjwatson commented 9 years ago

What's your:

How many players are on the server when this happens?

I might need you to tarball up your installation, so I can try and replicate a SuSE environment with your issues.

Cheers

moozaad commented 9 years ago 
#:~/tshock-debug> uname -a
Linux metis 3.16.7-21-desktop #1 SMP PREEMPT Tue Apr 14 07:11:37 UTC 2015 (93c1539) x86_64 x86_64 x86_64 GNU/Linux

#:~/tshock-debug> free -h
             total       used       free     shared    buffers     cached
Mem:           23G        22G       1.1G       515M       790M        19G
-/+ buffers/cache:       2.1G        21G
Swap:         2.0G         0B       2.0G

#:~/tshock-debug> mono --version
Mono JIT compiler version 4.0.1 (tarball Fri May 29 00:26:38 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

Doesn't matter on what world size or players (just me testing). There's also an issue where it stops sending data from the server after a minutes - like a keepalive is not registering or something but then unable to reconnect without restarting the tshock.

Stock 13.2 installation but Mono is not the default (see above) it is the Factory build http://download.opensuse.org/repositories/Mono:/Factory/openSUSE_13.2/x86_64/mono-complete-4.0.1.44-1.1.x86_64.rpm . Add this repo http://download.opensuse.org/repositories/Mono:/Factory/openSUSE_13.2/Mono:Factory.repo

tylerjwatson commented 9 years ago

Would you mind tarballing up your installation as-is, so I have something to run on the target SuSE box?

From conjecture, in mono_gc_alloc_vector in mono/metadata/sgen-mono.c there is a malloc that happens on line 1719. I think this is pointing to an out of memory condition. Your free output indicates that something is using most of the memory on your system, but I can't be sure as yet; mono is actually supposed to throw System.OutOfMemoryException objects up the stack in this case, but if the GC can't allocate one it will throw all the toys out and start whinging.

Will see if I can replicate your problem on this end sans the memory issues.

I can use tools like valgrind to detect invalid reads and malloc fails, but it's all useless until I can replicate your problem in the flesh.

moozaad commented 9 years ago

-link deleted-

The original debug build that I based it off before moving the server plugins for mysql.*.dll from the current release (tshock_4.2.10.zip) is also zipped in the root.

It includes a small world with nothing much touched that I used.

The used memory is just file cache. Nothing odd there.

moozaad commented 9 years ago 
[0x7f7917afe700:] EXCEPTION handling: System.Net.WebException: The request timed out
[0x7f7917afe700:] EXCEPTION handling: System.Net.WebException: The request timed out
UpdateManager Exception: System.Net.WebException: The request timed out
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00065] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:937
  at System.Net.HttpWebRequest.GetResponse () [0x0000e] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/System/System.Net/HttpWebRequest.cs:949
  at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) [0x00000] in <filename unknown>:0
[0x7f7917afe700:] EXCEPTION handling: System.Net.WebException: The request timed out
86.22.118.125:50954 is connecting...
testafro has joined.
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_string (intptr,intptr,int) <IL 0x0000f, 0xffffffff>
  at (wrapper alloc) object.AllocString (intptr,int) <IL 0x0006b, 0xffffffff>
  at string.JoinUnchecked (string,string[],int,int) [0x00045] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System/String.cs:2429
  at string.Join (string,string[],int,int) [0x00076] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System/String.cs:2408
  at TShockAPI.Group.HasPermission (string) <IL 0x00068, 0x0010b>
  at TShockAPI.TShock.OnSecondUpdate () <IL 0x003bc, 0x00674>
  at TShockAPI.TShock.OnUpdate (System.EventArgs) <IL 0x00042, 0x000bb>
  at TerrariaApi.Server.HandlerCollection`1.Invoke (ArgsType) <IL 0x00062, 0x00197>
  at TerrariaApi.Server.HookManager.InvokeGameUpdate () <IL 0x00051, 0x0008f>
  at Terraria.Main.DedServ () <IL 0x007db, 0x0124f>
  at Terraria.ProgramServer.Main (string[]) <IL 0x000c5, 0x001e7>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00060, 0xffffffff>

Native stacktrace:

        mono() [0x4ba2d8]
        mono() [0x51046e]
        mono() [0x42b2e8]
        /lib64/libpthread.so.0(+0xf890) [0x7f7949e9d890]
        mono() [0x5d16ea]
        mono() [0x5d6a2b]
        mono() [0x5d7449]
        mono() [0x5ed5ee]
        mono() [0x5ed8b3]
        [0x407f0eae]
             total       used       free     shared    buffers     cached
Mem:           23G       2.5G        21G       515M        16M       682M
-/+ buffers/cache:       1.8G        21G
Swap:         2.0G         0B       2.0G
2015-06-06T13:15:08.167087+01:00 metis kernel: [774056.790825] SysRq : Show Memory
2015-06-06T13:15:08.167104+01:00 metis kernel: [774056.814444] Mem-Info:
2015-06-06T13:15:08.167112+01:00 metis kernel: [774056.814450] Node 0 DMA per-cpu:
2015-06-06T13:15:08.167114+01:00 metis kernel: [774056.814453] CPU    0: hi:    0, btch:   1 usd:   0
2015-06-06T13:15:08.167122+01:00 metis kernel: [774056.814459] CPU    1: hi:    0, btch:   1 usd:   0
2015-06-06T13:15:08.167124+01:00 metis kernel: [774056.814461] CPU    2: hi:    0, btch:   1 usd:   0
2015-06-06T13:15:08.167126+01:00 metis kernel: [774056.814464] CPU    3: hi:    0, btch:   1 usd:   0
2015-06-06T13:15:08.167135+01:00 metis kernel: [774056.814469] Node 0 DMA32 per-cpu:
2015-06-06T13:15:08.167137+01:00 metis kernel: [774056.814472] CPU    0: hi:  186, btch:  31 usd:  93
2015-06-06T13:15:08.167139+01:00 metis kernel: [774056.814474] CPU    1: hi:  186, btch:  31 usd: 155
2015-06-06T13:15:08.167140+01:00 metis kernel: [774056.814479] CPU    2: hi:  186, btch:  31 usd: 162
2015-06-06T13:15:08.167150+01:00 metis kernel: [774056.814483] CPU    3: hi:  186, btch:  31 usd:  78
2015-06-06T13:15:08.167152+01:00 metis kernel: [774056.814484] Node 0 Normal per-cpu:
2015-06-06T13:15:08.167153+01:00 metis kernel: [774056.814487] CPU    0: hi:  186, btch:  31 usd: 182
2015-06-06T13:15:08.167155+01:00 metis kernel: [774056.814489] CPU    1: hi:  186, btch:  31 usd: 126
2015-06-06T13:15:08.167170+01:00 metis kernel: [774056.814497] CPU    2: hi:  186, btch:  31 usd: 138
2015-06-06T13:15:08.167172+01:00 metis kernel: [774056.814498] CPU    3: hi:  186, btch:  31 usd: 171
2015-06-06T13:15:08.167173+01:00 metis kernel: [774056.814504] active_anon:350594 inactive_anon:215229 isolated_anon:0
2015-06-06T13:15:08.167175+01:00 metis kernel: [774056.814504]  active_file:28668 inactive_file:4723 isolated_file:0
2015-06-06T13:15:08.167177+01:00 metis kernel: [774056.814504]  unevictable:20 dirty:0 writeback:0 unstable:0
2015-06-06T13:15:08.167179+01:00 metis kernel: [774056.814504]  free:5524830 slab_reclaimable:4824 slab_unreclaimable:6434
2015-06-06T13:15:08.167180+01:00 metis kernel: [774056.814504]  mapped:34691 shmem:132055 pagetables:4670 bounce:0
2015-06-06T13:15:08.167182+01:00 metis kernel: [774056.814504]  free_cma:0
2015-06-06T13:15:08.167193+01:00 metis kernel: [774056.814516] Node 0 DMA free:15904kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
2015-06-06T13:15:08.167195+01:00 metis kernel: [774056.814523] lowmem_reserve[]: 0 1944 24135 24135
2015-06-06T13:15:08.167197+01:00 metis kernel: [774056.814527] Node 0 DMA32 free:1710284kB min:5440kB low:6800kB high:8160kB active_anon:148392kB inactive_anon:116972kB active_file:12400kB inactive_file:956kB unevictable:16kB isolated(anon):0kB isolated(file):0kB present:2075264kB managed:1996092kB mlocked:16kB dirty:0kB writeback:0kB mapped:11964kB shmem:17400kB slab_reclaimable:1256kB slab_unreclaimable:1648kB kernel_stack:416kB pagetables:1088kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2015-06-06T13:15:08.167206+01:00 metis kernel: [774056.814538] lowmem_reserve[]: 0 0 22190 22190
2015-06-06T13:15:08.167208+01:00 metis kernel: [774056.814542] Node 0 Normal free:20373132kB min:62096kB low:77620kB high:93144kB active_anon:1253984kB inactive_anon:743944kB active_file:102272kB inactive_file:17936kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:23068672kB managed:22723272kB mlocked:64kB dirty:0kB writeback:0kB mapped:126800kB shmem:510820kB slab_reclaimable:18040kB slab_unreclaimable:24088kB kernel_stack:4272kB pagetables:17592kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
2015-06-06T13:15:08.167212+01:00 metis kernel: [774056.814552] lowmem_reserve[]: 0 0 0 0
2015-06-06T13:15:08.169243+01:00 metis kernel: [774056.814556] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB
2015-06-06T13:15:08.169247+01:00 metis kernel: [774056.814572] Node 0 DMA32: 5087*4kB (UEMR) 4704*8kB (UEMR) 3875*16kB (UEMR) 2899*32kB (UEMR) 2103*64kB (UEMR) 1418*128kB (UEMR) 851*256kB (UEM) 464*512kB (UEMR) 205*1024kB (EMR) 94*2048kB (EMR) 79*4096kB (EM) = 1710284kB
2015-06-06T13:15:08.169249+01:00 metis kernel: [774056.814588] Node 0 Normal: 81203*4kB (UEM) 74502*8kB (UEM) 51313*16kB (UEM) 36064*32kB (UEM) 27984*64kB (UEM) 18905*128kB (UEM) 11252*256kB (UEM) 6145*512kB (EM) 2560*1024kB (UEM) 877*2048kB (EMR) 689*4096kB (UEM) = 20373132kB
2015-06-06T13:15:08.169251+01:00 metis kernel: [774056.814605] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2015-06-06T13:15:08.169253+01:00 metis kernel: [774056.814607] 165444 total pagecache pages
2015-06-06T13:15:08.169254+01:00 metis kernel: [774056.814609] 0 pages in swap cache
2015-06-06T13:15:08.169256+01:00 metis kernel: [774056.814611] Swap cache stats: add 0, delete 0, find 0/0
2015-06-06T13:15:08.169258+01:00 metis kernel: [774056.814613] Free swap  = 2102268kB
2015-06-06T13:15:08.169260+01:00 metis kernel: [774056.814614] Total swap = 2102268kB
2015-06-06T13:15:08.169261+01:00 metis kernel: [774056.814616] 6289983 pages RAM
2015-06-06T13:15:08.169263+01:00 metis kernel: [774056.814617] 0 pages HighMem/MovableOnly
2015-06-06T13:15:08.169264+01:00 metis kernel: [774056.814618] 86350 pages reserved
2015-06-06T13:15:08.169266+01:00 metis kernel: [774056.814620] 0 pages hwpoisoned
moozaad commented 9 years ago 
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <IL 0x0000f, 0xffffffff>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <IL 0x00088, 0xffffffff>
  at string.SplitByCharacters (char[],int,bool) [0x00103] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System/String.cs:361
  at string.Split (char[],int,System.StringSplitOptions) [0x0007a] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System/String.cs:242
  at string.Split (char[]) [0x00000] in /home/abuild/rpmbuild/BUILD/mono-4.0.1/mcs/class/corlib/System/String.cs:211
  at TShockAPI.Group.HasPermission (string) <IL 0x00042, 0x000c3>
  at TShockAPI.TShock.OnSecondUpdate () <IL 0x003bc, 0x00674>
  at TShockAPI.TShock.OnUpdate (System.EventArgs) <IL 0x00042, 0x000bb>
  at TerrariaApi.Server.HandlerCollection`1.Invoke (ArgsType) <IL 0x00062, 0x00197>
  at TerrariaApi.Server.HookManager.InvokeGameUpdate () <IL 0x00051, 0x0008f>
  at Terraria.Main.DedServ () <IL 0x007db, 0x0124f>
  at Terraria.ProgramServer.Main (string[]) <IL 0x000c5, 0x001e7>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00060, 0xffffffff>

Native stacktrace:

        mono() [0x4ba2d8]
        mono() [0x51046e]
        mono() [0x42b2e8]
        /lib64/libpthread.so.0(+0xf890) [0x7f0754dc7890]
        mono() [0x5d0830]
        mono() [0x5d16f6]
        mono() [0x5d6a2b]
        mono() [0x5d7449]
        mono() [0x5ed5ee]
        mono() [0x5ed7f3]
        [0x40d6e54e]
tylerjwatson commented 9 years ago

Talking with the OP it seems we have narrowed it down to the sgen GC used by default on Mono. Using the boehm GC completely rectified all the problems, but boehm is much slower and less aggressive than sgen is, albeit older.

This is only a symptom, however, and I am now standing up a SuSE box to see how the mono packages on SuSE affect the running of the server. Mono is crusty/non-existent in Fedora and RHEL respectively, so we (in RedHat land) have to use Xamarin's official repos which could have been built slightly different from the specfiles than SuSEs. If I can reproduce the problems locally then I can point fingers at the SuSE package builds, and recommend people use the boehm GC on SuSE.

More info to come.

Edit: it's worth noting that I don't have problems on either GC on any of my systems; none of them are SuSE based.

tylerjwatson commented 9 years ago

Attaching https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584 as potential cross-reference

tylerjwatson commented 9 years ago

I cannot get the process to crash, but I can, however repliate the locking issue under mono-sgen on OpenSuSE 13.2.

Mono runs at 100% of CPU under this condition.

tylerjwatson commented 9 years ago

Edit: Completely spoke too soon. I can replicate the crash using mono --debug --trace=N:nothing TerrariaServer.exe on OpenSuSE 13.2.

tylerjwatson commented 9 years ago

A bug has been sumitted to OpenSuSE: https://bugzilla.opensuse.org/show_bug.cgi?id=933838

Linking in for reference

fennec2000 commented 8 years ago

I know this is nearly half a year old but I just tried on the latest mono and latest tshock I can't get it to work either, is there a current fix or working version? For reference if needed:

$ uname -a
Linux {removed web address} 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ free
         total       used       free     shared    buffers     cached
Mem:       8057568    6780684    1276884        920     375484    3722588
-/+ buffers/cache:    2682612    5374956
Swap:      2456568     269624    2186944
$ mono --version
Mono JIT compiler version 4.3.0 (master/526c802 Fri Nov 13 11:36:24 GMT 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

What tshock says:

Terraria Server v1.3.0.8

Listening on port 7777
Type 'help' for a list of commands.

: UpdateManager Exception: System.Net.WebException: The request timed out
at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x41f1ede0 + 0x00197> in <filename unknown>:0
at System.Net.HttpWebRequest.GetResponse () <0x420bac00 + 0x0005a> in <filename unknown>:0
at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) <0x420ba960 + 0x0004e> in <filename unknown>:0
hakusaro commented 8 years ago

The stack trace you posted is just the update checker not working. The server itself works fine. On Sat, Nov 14, 2015 at 4:12 AM Stuart Hayes notifications@github.com wrote:

I know this is nearly half a year old but I just tried on the latest mono and latest tshock I can't get it to work either, is there a current fix or working version? For reference if needed:

$ uname -a Linux {removed web address} 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux $ free total used free shared buffers cached Mem: 8057568 6780684 1276884 920 375484 3722588 -/+ buffers/cache: 2682612 5374956 Swap: 2456568 269624 2186944 $ mono --version Mono JIT compiler version 4.3.0 (master/526c802 Fri Nov 13 11:36:24 GMT 2015) Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com

    TLS:           __thread
    SIGSEGV:       altstack
    Notifications: epoll
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug
    LLVM:          supported, not enabled.
    GC:            sgen

What tshock says:

Terraria Server v1.3.0.8

Listening on port 7777 Type 'help' for a list of commands.

: UpdateManager Exception: System.Net.WebException: The request timed out at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x41f1ede0 + 0x00197> in :0 at System.Net.HttpWebRequest.GetResponse () <0x420bac00 + 0x0005a> in :0 at TShockAPI.Utils.GetResponseNoException (System.Net.HttpWebRequest req) <0x420ba960 + 0x0004e> in :0

— Reply to this email directly or view it on GitHub https://github.com/NyxStudios/TShock/issues/935#issuecomment-156687253.

fennec2000 commented 8 years ago

My main problem is: The server says:

ipaddress:port is connecting to slot 0...

and the client get:

Found server

and nothing else happens

moozaad commented 8 years ago

Use the boehm GC - sgen causes deadlocks on some setups.

fennec2000 commented 8 years ago

I have tried mono-boehm and mono --gc=boehm still cant enter the server

tylerjwatson commented 8 years ago

Hi guys, please try bamboo build 583 featured here and see if it aleviates the problem.

Khitiara commented 8 years ago

Having the same issue with newest tshock builds and mono 4.2.2, mono 3 isnt available on my distro (arch)

tylerjwatson commented 8 years ago

Hi gus, we believe this is a duplicate of #1172. Please check there for updates.