gegtor commented 5 years ago

https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/538

v3.29.1.0 Update causes the same old Stacktrace error with Russia map

Full stacktrace: https://pastebin.com/nNHry1ks

SDGNelson commented 5 years ago

This crash appears to be caused by setting the terrain SplatPrototypes on the dedicated server, whereas previously it was caused by setting Alphamaps. Interestingly the splat prototypes are set before the alphamaps, so I'm not sure why we wouldn't have had this problem before.

Right now the game sets some placeholder empty Texture2D.blackTexture splat prototypes, so I'll double-check that nothing else depends on those and disable them on the dedicated server.

SDGNelson commented 5 years ago

Patch will most likely be later today

gegtor commented 5 years ago

Awesome! Thanks

gegtor commented 5 years ago

@SDGNelson I still get SIGABRT in V3.29.2.0

https://pastebin.com/U7vNj6g1

SDGNelson commented 5 years ago

Seems to have been a few posts about linux mono native crashes lately, I'll re-open and link

SDGNelson commented 5 years ago

612

gegtor commented 5 years ago

v3.39.2.1 Update didn't fix it (Duno if it had to)

But still here is 3.39.2.1 stacktrace if it useful to you https://pastebin.com/n50jndZ3

aman7 commented 5 years ago

it did fix some issue but didnt fixed it fully, servers are still getting frozen, and also the same errors as above.

D-AIRY commented 5 years ago

The same problem. Sometimes it gives additional info about out-of-memory (having about 30gb ram free), attached below: oom.txt

SDGNelson commented 5 years ago

Preview branch now has the .2 patch which reverts to the previous version of .NET/mono. If you could opt-into it to verify that it's working again for you that would be super helpful, you can specify the branch on the SteamCMD command-line with the "-beta" flag e.g. "-beta preview" 🤞

gegtor commented 5 years ago

@SDGNelson 🎉🎉🎉v3.29.2.2🎉🎉🎉 🎉🎉🎉 Russia fires up 🎉🎉🎉

Tested Washington / Russia on 3 different servers

Server starts correctly and players can join

Not sure about stability because I had only 20 min to test it but I'm quite confident that everything is okay now

Roll out the update 😁

Trojaner commented 5 years ago

Please do not roll out the update. This would be disastrous for us plugin developers. A lot of work done recently would just vanish into nothing.

MeedicBear commented 5 years ago

Please don't roll back, You player base is gonna more than it already is.

rube200 commented 5 years ago

I'm from Saturday to sleep about 5h a day to make a plugin from scratch, because of the last update and to optimize it, I'm at the end where I only need to correct the small details, in case you reverse the update, I'm going to lose one week. In addition to the other developers who will also lose their work.

aviadmini commented 5 years ago

Instead of figuring out what actually happens Instead of checking whether that happens to vanilla game/modded Despite saying that linux is second class citizen Instead of looking into why everyone is meddling with 32-bit on linux etc etc etc

Oh well, please roll back to Early Access as well then, as it would represent state of it the best

ghost commented 5 years ago

Please do not roll back and try looking into other ways to fix this issue. As Trojaner has said, a lot of work has already been done to support .NET 4.6.1. Around this time of the year Unturned already slows down as well (Look at past year's stats around February - March https://steamcharts.com/app/304930#All) and we really don't need more downtime that would most likely further hurt the player count. Surely we could look for a better solution which would fix the root cause of this instead of just reverting back.

Gsechs commented 5 years ago

I agree with the previous comment.

I also noticed that this is a rough time for Unturned's concurrent player count. I think there are definitely other solutions such as using Unity 2018. Not to mention that this would likely cause quite a lot of disturbance for a lot of plugin developers. 😎

thereis commented 5 years ago

Please, do not roll back! The newest update let us developers to bring alot of features and customizations with the newest .net version.

Also, enable to display the servers with more than 24 people for everyone ;)

SDGNelson commented 5 years ago

Well, the issue is without a doubt the newer .NET/mono (crashes on linux) vs the old one (fine on all platforms). It's "experimental" in Unity 2017.4, so it's not like they'll say "oh ok, we'll put the stable 2019.x version into the next 2017 LTS release". I'm very sorry for any lost progress if we do have to roll back, but 2018.4 is not an option either. Trying out the experimental version was purely by plugin developers request, but the needs of the player base as a whole outweigh that.

For the moment I've reached out to a few of the server networks asking if they use the Linux server, and if so if it's been stable for them. If they're all using the stable Windows server, or the Linux server has been working fine, then maybe it is okay to keep this version for those brave enough to use the Linux server?

Trojaner commented 5 years ago

(...) but 2018.4 is not an option either.

Is there any specific reason for this? E.g. assetbundles not compatible again? If its just because you don't want to go through update hell again, I understand that.

There are 3 options which I can see:

Letting Unity fix it.

Since you are using an LTS release which still gets updates (I believe until 2019 Q4), I see Unity responsible to fix this issue. Opening a bug report is free 😁

Enforcing Wine / Proton / Steam Play on Linux and dropping native Linux builds completely.

Some games are known to run better on Wine / Proton / Steam Play than their native ports. I will check this option out and test stuff when I find some time.

Revert back to .NET 3.5

...and making all developers cry

joeymisfit commented 5 years ago

Hello @SDGNelson ,

I understand that Linux has issues with the new .net update and the majority of us would rather you find a fix to get Linux working with the newest updates rather than reverting what all we've done thus far in progress within the community. A lot of developers and essentially programmers have spent time updating plugins, workshops, discord bots, the connections for them, and many other aspects of their servers to comply and work properly with what was needed to be fixed, even though some if not most of them are rollback compatible, it just would be sad to see what all we've been through in upgrading to have reverted back in feature progress over such a small bug that can be fixed with help of several developers teaming up with yourself to brainstorm on the issues at hand to solve essentially the causes as to why Unturned is broken with the newest .net updates.

I hope you'd take into consideration what all we've spoken here within the thread before pursuing an actual rollback as it would essentially cause most of us to have to change several aspects of our servers with downtime while several modifications will need to take place, which overall affects most communities on all spectrums.

Thank you.

JustOneMoreBlock commented 5 years ago

THIS IS NOT A FIX

So, I've been in Linux all day reading a bunch of strace's, and few things keep popping out at me.

open("/usr/lib/mono/gac/policy.2.0.System/0.0.0.0__b77a5c561934e089/policy.2.0.System.dll", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/Games/unturned/System.dll", 0x7ffe18f7e2d0) = -1 ENOENT (No such file or directory)
stat("/home/Games/unturned/System.exe", 0x7ffe18f7e2d0) = -1 ENOENT (No such file or directory)
stat("/home/Games/unturned/System/System.dll", 0x7ffe18f7e2d0) = -1 ENOENT (No such file or directory)
stat("/home/Games/unturned/System/System.exe", 0x7ffe18f7e2d0) = -1 ENOENT (No such file or directory)
open("/usr/lib/mono/gac/System/2.0.0.0__b77a5c561934e089/System.dll", O_RDONLY) = 3

None of these exist, and should be using System.dll from the Managed/ folder anyway. Anyway, I attempted to symlink System.dll and even the entire folder from Managed and provided the following error.

Can't find custom attr constructor image: /home/Games/unturned/System.dll mtoken: 0x0a000001

Unhandled Exception:
System.TypeLoadException: Could not load type 'System.Runtime.CompilerServices.ExtensionAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
  at Rocket.Unturned.Launcher.RocketLauncher.Main (System.String[] args) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeLoadException: Could not load type 'System.Runtime.CompilerServices.ExtensionAttribute' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.
  at Rocket.Unturned.Launcher.RocketLauncher.Main (System.String[] args) [0x00000] in <filename unknown>:0

Since I'm running on CentOS 7 x64, and running Mono JIT compiler version 3.12.1 I figured I'll do a search of System.dll on the system.

[root@pvp unturned]# locate System.dll
/home/Games/unturned/Unturned_Data/Managed/System.dll
/home/Games/unturned/Unturned_Headless_Data/Managed/System.dll
/usr/lib/mono/2.0/System.dll
/usr/lib/mono/4.0/System.dll
/usr/lib/mono/4.5/System.IO.Compression.FileSystem.dll
/usr/lib/mono/4.5/System.dll
/usr/lib/mono/gac/System/2.0.0.0__b77a5c561934e089/System.dll
/usr/lib/mono/gac/System/2.0.0.0__b77a5c561934e089/System.dll.mdb
/usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
/usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll.mdb
/usr/lib/mono/gac/System.IO.Compression.FileSystem/4.0.0.0__b77a5c561934e089/System.IO.Compression.FileSystem.dll
/usr/lib/mono/gac/System.IO.Compression.FileSystem/4.0.0.0__b77a5c561934e089/System.IO.Compression.FileSystem.dll.mdb

cd /home/Games/unturned/
ln -s /usr/lib/mono/2.0/System.dll .

Well, it didn't crash, responded to saving the world, and responded in the client. Wasn't able to join. :(

This provides SO MANY ERRORS. But, it doesn't crash.

Current Error:

DivideByZeroException: Attempted to divide by zero. - SDG.Unturned.LightingManager.updateLighting () (at <b6f53656616b4fdd9d9d28953c558538>:0)
SDG.Unturned.LightingManager.onLevelLoaded (System.Int32 level) (at <b6f53656616b4fdd9d9d28953c558538>:0)
SDG.Unturned.Level+<init>c__Iterator0.MoveNext () (at <b6f53656616b4fdd9d9d28953c558538>:0)
UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <102dc60ee5974486865758606d7385d4>:0)

Normally I'd assume this is from the Lightining.dat file. Tried removing it, and didn't work. :(

morelogs.log strace_1.log strace_2.log strace_3.log

¯_(ツ)_/¯

SDGNelson commented 5 years ago

Released .2 which fixes a few misc issues, but keeping the .NET upgrade 👍

Bradler26 commented 5 years ago

Thanks for keeping the current version of .net but please if possible don't change it. As many plugin devs mentioned they have worked endlessly to get their plugins migrated over. Some have even began work on new plugins that utilizes the features of the new .net

I personally have been very happy with your latest work and sure, there have been bugs and hardships but overall it will boost the game in the long term. Any extra changes will affect the already low playerbase (due to the time of year plus recent updates causing server downtime)

Doing another update will ruin all servers for another few days. 90% of the playerbase plays multiplayer and if servers go down the playerbase goes down as well...

Trojaner commented 5 years ago

https://github.com/Trojaner/unturned-wine

I have managed to run Unturned Windows Build on Linux using Wine.

However there is a bug that prevents it from working correctly:

EntryPointNotFoundException: AllocConsole
  at (wrapper managed-to-native) SDG.Unturned.ConsoleOutput:AllocConsole ()
  at SDG.Unturned.ConsoleOutput..ctor () [0x00006] in <491015b431a743d9bf5a9bd7f52497bb>:0
  at SDG.Unturned.CommandWindow..ctor () [0x00037] in <fa52c3fc37c54e189f49994f187e446b>:0
  at SDG.Unturned.Dedicator.awake () [0x0003f] in <fa52c3fc37c54e189f49994f187e446b>:0
  at SDG.Unturned.Setup.Awake () [0x00011] in <fa52c3fc37c54e189f49994f187e446b>:0

ArgumentNullException: Value cannot be null.
Parameter name: type
  at System.Activator.CreateInstance (System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, System.Object[] args, System.Globalization.CultureInfo culture, System.Object[] activationAttributes) [0x00003] in <cfc149f8218b496788d84$  at System.Activator.CreateInstance (System.Type type, System.Object[] args) [0x00000] in <cfc149f8218b496788d8493c87de777a>:0
  at SDG.Unturned.Assets.loadFile (SDG.Unturned.Assets+ScannedFileInfo file) [0x000cc] in <fa52c3fc37c54e189f49994f187e446b>:0
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
SDG.Unturned.Assets:loadFile(ScannedFileInfo)
SDG.Unturned.<loadAllFilesScanned>c__Iterator0:MoveNext()
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

Failed to find reader for: System.Guid
Failed to find reader for: System.Type

DirectoryNotFoundException: Could not find a part of the path 'C:\users\root\DedicatedServer\Servers\server\Bundles'.
  at System.IO.__Error.WinIOError (System.Int32 errorCode, System.String maybeFullPath) [0x000f7] in <cfc149f8218b496788d8493c87de777a>:0
  at System.IO.FileSystemEnumerableIterator`1[TSource].HandleError (System.Int32 hr, System.String path) [0x00006] in <cfc149f8218b496788d8493c87de777a>:0
  at System.IO.FileSystemEnumerableIterator`1[TSource].CommonInit () [0x00054] in <cfc149f8218b496788d8493c87de777a>:0
  at System.IO.FileSystemEnumerableIterator`1[TSource]..ctor (System.String path, System.String originalUserPath, System.String searchPattern, System.IO.SearchOption searchOption, System.IO.SearchResultHandler`1[TSource] resultHandler, System.Boolean checkHost) [0x000d6$  at System.IO.FileSystemEnumerableFactory.CreateFileNameIterator (System.String path, System.String originalUserPath, System.String searchPattern, System.Boolean includeFiles, System.Boolean includeDirs, System.IO.SearchOption searchOption, System.Boolean checkHost) [0$  at System.IO.Directory.InternalGetFileDirectoryNames (System.String path, System.String userPathOriginal, System.String searchPattern, System.Boolean includeFiles, System.Boolean includeDirs, System.IO.SearchOption searchOption, System.Boolean checkHost) [0x00000] in $  at System.IO.Directory.InternalGetFiles (System.String path, System.String searchPattern, System.IO.SearchOption searchOption) [0x00000] in <cfc149f8218b496788d8493c87de777a>:0
  at System.IO.Directory.GetFiles (System.String path, System.String searchPattern) [0x0001c] in <cfc149f8218b496788d8493c87de777a>:0
  at SDG.Unturned.Assets.scanFolder (System.String path, System.Boolean usePath, System.Boolean loadFromResources, System.Boolean canUse, SDG.Unturned.AssetDirectory directory, SDG.Unturned.EAssetOrigin origin, System.Boolean overrideExistingIDs, SDG.Unturned.MasterBund$  at SDG.Unturned.Assets+<dedicatedServerUgcLoadingStep>c__Iterator2.MoveNext () [0x0026d] in <fa52c3fc37c54e189f49994f187e446b>:0
  at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00027] in <dec06e59a8644684a220ee8e93af2a2a>:0

game.log

gegtor commented 5 years ago

I'm very sceptical about using Wine

I'm running my server network for about 7 months now and unturned native server worked really well with rocket and everything else it's also quite efficient and runs Russia at 50 TPS on 1 core 2GB ram server

Introducing Wine would mean more complexity and lower performance (And is just another workaround, not a fix)

The ideal solution is to work hard on Linux version and resolve many issues that pilled up over the years (get 64bit build working, resolve steam api error, etc)

Trojaner commented 5 years ago

Introducing Wine would mean more complexity and lower performance (And is just another workaround, not a fix)

I already made some scripts, its basically 2 commands to install and set it up. Also people hosting on Linux should know how to use such scripts.

Regarding performance, as said earlier, some games even have better performance on Wine than their native ports.

gegtor commented 5 years ago

What I mean is not that it would be harder to manage such server

It's just every time something brakes on Linux we find (i'm sorry) a shitty workaround

Trojaner commented 5 years ago

It's just every time something brakes on Linux we find (i'm sorry) a shitty workaround

Thats exactly why Wine is the solution. No more Linux breaking, no more Linux workarounds. Why? Because it will run exactly the same as on Windows.

Johnanater commented 5 years ago

If you do plan on removing native Linux support, be sure BattlEye works with it on the client. 😉

gegtor commented 5 years ago

This is exactly what I was saying ^

+1 rep

Johnanater commented 5 years ago

Also, can confirm it works on Ubuntu 18.04.2 LTS. Just install mono-runtime instead of mono-devel.

EDIT: When loading the map on Russia, the server finishes loading the map and just crashes due to "System out of Memory" despite having 8gb allocated to my vm (Server_Test.log). I allocated more memory to mono using "export MONO_GC_PARAMS=max-heap-size=6000000", but this just causes an even bigger issue (crash.log). :(

Server_Test.log crash.log

strongunturned commented 5 years ago

Will there be a fix for this problem, or will we go to windows?

Trojaner commented 5 years ago

There are distributions and set ups which are working. Just some bigger maps do not work (e.g. Russia). You should experiment a bit.

joeymisfit commented 5 years ago

Windows for server solutions is indeed pricey that I'll admit however there are plenty of work arounds, If you are tech savvy. As for Linux I don't recall Nelson ever offering much support towards it because not too many individuals use it as intended, nor do many folks use Linux as a standalone or vm o.s.

gegtor commented 5 years ago

My whole server network is based on Russia map getting 24/24 constantly

Also OVH Linux VPS: 3.56$ OVH basic Windows VPS: 15.57$

aman7 commented 5 years ago

yes please tell use when you going to fix this issue, its really hard checking all server if they are up or frozen. and please also decide if you going to support linux or not, because it is start of the month, i got invoices and will decide to continue with or not.

Trojaner commented 5 years ago

At the moment this is all about the plugin developers lobby against the server owners/hosters lobby. Both have good arguments on their respective side.

To be honest, I would not want to be in Nelsons place right now. Regardless of the choice he makes, someone will be upset and angry.

CozmicHosting commented 5 years ago

Yeah but if you think about it if theres no servers to play on where will the servers come from some of the popular networks are having to pay more since windows machines cost more than linux.

On Mon, Mar 4, 2019, 12:39 PM Enes Sadık Özbek notifications@github.com wrote:

At this moment it is the plugin developers lobby against the server owners/hosters lobby. Both have good arguments on their respective side.

To be honest, I would not want to be in Nelsons place right now. Regardless of the choice he makes, someone will be upset and angry.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/611#issuecomment-469344792, or mute the thread https://github.com/notifications/unsubscribe-auth/Atxezh1c405guw7pBhzOWp7p_B5p00xdks5vTVpFgaJpZM4bKWCw .

gegtor commented 5 years ago

Is there really no way to fix Russia on new Mono / .NET ?

Or is this issue purely Unity fault and has to be resolved with them

CozmicHosting commented 5 years ago

I cant get any unturned server to work on my linux machine just crashes.

On Mon, Mar 4, 2019, 12:50 PM Gegtor notifications@github.com wrote:

Is there really no way to fix Russia on new Mono / .NET ?

Or is this issue purely Unity fault and has to be resolved with them

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/611#issuecomment-469348905, or mute the thread https://github.com/notifications/unsubscribe-auth/Atxezi02g97vCvFO9vxMCvI0UXLQ5wFVks5vTVzggaJpZM4bKWCw .

Trojaner commented 5 years ago

Is there really no way to fix Russia on new Mono / .NET ?

There are ways, but question is if they are feasible. For example, upgrading Unity again might fix it, but is not feasible.

At the moment I can not think of anything but "Linux vs new .NET".

Well i think Nelson could fix the issue for all by not downgrading the Mono and instead searching for the root cause of the issue and bringing it up with Mono or unity and he could also make linux more stable and better supported for all. If its a Mono issue bring it up with Mono

Problem is that the "root cause" is in Unity code, not in Unturned code. Nelson himself can not fix this, he only could find some workaround at max.

gegtor commented 5 years ago

@CozmicHosting Current state looks something like this

Washington etc Works but crashes every few hours Russia France etc Crashes on server start

If you can't start Washington and join issue lies in your server config or packages and even fixed unturned won't work

SDGNelson commented 5 years ago

Yeah this is stressful :( One option I'm looking into is having an old-mono version for Linux, I wonder are you already downloading a specific Linux depot i.e. the Unturned_Headless Linux32 depot?

strongunturned commented 5 years ago

@SDGNelson Yes, we did this, also tried to increase and decrease the version of mono, the error remained

Trojaner commented 5 years ago

@strongunturned Unity uses its own mono implementation. This is not bound to the mono you install on your Linux system. Only Nelson can set the mono version used by Unturned.

strongunturned commented 5 years ago

@Trojaner Thanks I'll know. My whole flood is connected with the fact that we do not know what to do, wait for a fix or switch to windows.

@SDGNelson I wish you patience and success in correction. I apologize for my flood

SDGNelson commented 5 years ago

3.29.2.4 patch is using the previous mono version for the Headless server! (among other fixes) If need be we can also have previous mono versions for the non-headless and 64-bit versions.

gegtor commented 5 years ago

🎉 It's working 🎉

Thanks for everything

justosius commented 5 years ago

Lets hope 4.0 will support only single os. Less is more.

SmartlyDressedGames / Unturned-3.x-Community

v3.29.1.0 Update broke Russia map on Linux servers (Again) #611

612