Bugs I've found - Githubissues

Bradler26 commented 4 years ago

Okay so let me preface this by saying some of these bugs may be server related but I am going to put them in order of certainty from how sure I am that they are game related to how unsure I am.

~~Free Cam Slow Mode Holding Control key no longer works for slowing down the camera when in free cam mode.~~

Player freezing This one is quite tricky as I have had another server owner report it but my server seems especially bad. Pretty much a person will continue to play on the server perfectly fine. Everything on the server works for them as it should and they can even hear players voices but all players are totally frozen on the server and they can only see them move when they access their inventory. In order to fix this, the player has to relog. Didn't have this issue since the last main update.

~~Mods Broken? A lot of server owners reported mods being broken but sadly I was away during that time so I couldn't really test myself. I have had 1 player report getting this bug https://cdn.discordapp.com/attachments/583931996695298049/592268301791526922/SPOILER_unknown.png The ID displayed is related to more farming mod which is a mod other server owners reported as buggy. Is this a mod issue or is it still an issue from the last update that needs to be patched?~~

Those are all the main bugs I found that could be game related. As I stated above, some of these may just be from my server but I have seen other servers/players reporting it so I thought I ought to post it.

Wanted98 commented 4 years ago

I have found a bug / glitch in which if you collide 2 apcthe character bugs and leave the vehicle partially or totally,the other players cant see him in that position it, if you are a copilot in an apc you can take a gun and kill everything that is outside(like in the screenshot)

20190703005505_1 20190703005406_1

ghost commented 4 years ago

We have also encountered the players freezing issue. I believe it is related to Vanish, as when Admins don't use vanish it seemed to stop happening. Could be happening because vanish manually adds a PlayerMovement update and the methods it uses to do so are marked obsolete but nelson, but that is an uneducated guess.

Bradler26 commented 4 years ago

Yeh I have also had reports that it was caused by vanish so it could be. I am going to go and test it when I get home. Do you use uessentials for vanish or aviadmin. I dont know if the type of plugin would change anything but it might so I'll just double check.

On Sat, 6 Jul 2019, 7:57 a.m. Miku, notifications@github.com wrote:

We have also encountered the players freezing issue. I believe it is related to Vanish, as when Admins don't use vanish it seemed to stop happening. Could be happening because vanish manually adds a PlayerMovement update and the methods it uses to do so are marked obsolete but nelson, but that is an uneducated guess.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/882?email_source=notifications&email_token=ALHFDJYPC2S72QKZAR6NYNDP6AX5JA5CNFSM4H3V2IV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZKTE7I#issuecomment-508899965, or mute the thread https://github.com/notifications/unsubscribe-auth/ALHFDJ5HH722P4PZOIDCQGLP6AX5JANCNFSM4H3V2IVQ .

ghost commented 4 years ago

We use rocket's vanish, I didn't even think aviadmin had vanish. @Bradler26

SDGNelson commented 4 years ago

Freecam slow mode: ctrl to slow down was removed in favor of the scroll-wheel adjustable speed. (much greater range of options)

Vanish: If the custom vanish plugins are manually patching player movement updates or the obsolete methods, that does seem plausible. When removing the need for rocket core to patch the game DLL I added a property to PlayerMovement which pauses sending movement updates to other players which is the recommended way to vanish at the moment. Odd if the built-in vanish command isn't using that? I'll double-check if there's any potential odd behavior with that flag as well.

ghost commented 4 years ago

                Player.GetComponent<UnturnedPlayerMovement>().VanishMode = value;
                PlayerMovement pMovement = Player.GetComponent<PlayerMovement>();
                pMovement.canAddSimulationResultsToUpdates = !value;
                if (vanishMode && !value)
                {
                    pMovement.updates.Add(new PlayerStateUpdate(pMovement.real, Player.Player.look.angle, Player.Player.look.rot));
                    pMovement.isUpdated = true;
                    PlayerManager.updates++;
                }
                vanishMode = value;
            }

It's using the new way to pause sending movement updates as well as manually adding a single movement update it seems.

SDGNelson commented 4 years ago

isUpdated and PlayerManager.updates are not referenced by anything (left behind but marked obsolete to prevent breaking plugins), but it looks like that code should be safe.

If it is only happening after vanish is used perhaps something is injecting code into PlayerManager? It seems bizarre that accessing the inventory would affect it; none of this code references that. 😕

Bradler26 commented 4 years ago

One of my staff has confirmed it is NOT vanish as he was the only staff online with many players suffering from the freeze issue and he doesn't have vanish. I still don't really have any idea what it could be but one point that was noted is that when the "bugged" player sees everyone as frozen and they shoot at one of the frozen players he can still do damage and even kill the player no matter where the other play is on the map.

educatalan02 commented 4 years ago

I already reported this bug #693

Bradler26 commented 4 years ago

Yeh I have got reports that players see multiple versions of the same person as well but it's much more rar3

On Sun, 7 Jul 2019, 10:29 p.m. educatalan02, notifications@github.com wrote:

I already reported this bug #693 https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/693

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SmartlyDressedGames/Unturned-3.x-Community/issues/882?email_source=notifications&email_token=ALHFDJ33UGSNC5O2TN6JRA3P6JG3FA5CNFSM4H3V2IV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZLSQ2Y#issuecomment-509028459, or mute the thread https://github.com/notifications/unsubscribe-auth/ALHFDJZOWCOLC5YUDN3BVP3P6JG3FANCNFSM4H3V2IVQ .

educatalan02 commented 4 years ago

Yeah because you can deal damage to both copys

Johnanater commented 4 years ago

I've had the invisible players bug happen when I was modified SaveLoadPlugin to use async to give players items. Players were fine at first, but after a while players would just be floating where they spawned.

Bradler26 commented 4 years ago

Did anyone have a fix or way of debugging this as it's getting quite annoying for players.

Bradler26 commented 4 years ago

Some players are reporting that it could be related to the "More Farming Mod" If they stand around animals in the mod long enough they will glitch out. Is this even possible?

Bradler26 commented 4 years ago

Sorry for the wall of posts but I got a players logs if it helps someone debug this.

Im sure its caused by a mod but i have no clue how to really read these logs so im hoping someone who has the expertise could help out. output_log.txt

Trojaner commented 4 years ago

If it's from a mod, it would be probably impossible to figure it out from the logs. Just remove all workshop one by one, Brad.

Bradler26 commented 4 years ago

I would but then my player base would complain constantly lol. If only there was a trustworthy way of faking players just so the server can handle a load.

SDGNelson commented 4 years ago

Looking in that log file I do see two exceptions, but I doubt they are related. Opened issues for them (askBrowserRequest and tellZombies).

Bradler26 commented 4 years ago

Does this issue seem mod related or could it be a plugin?

SDGNelson commented 4 years ago

More likely to be plugin-related than custom content. Looking at it with fresh eyes still nothing jumps out as suspicious; even if distance is wonky they should still get an update at least every 8 ticks.

Bradler26 commented 4 years ago

It seems very much like a clientsided bug (a relog fixes it and it only affects 1 player while the rest remain fine) which from what I've seen is most of the time caused by mods being fault y but if you think it's a plugin I will start removing plugins in bunches and try and work it out.

Pigeon-cmd commented 4 years ago

@SDGNelson Hate to take up your time with our plugin issues, but do you think it could have something to do with using Harmony to conditionally cancel PlayAnimator's sendGesture, PlayerStance's checkStance? (Most likely used by popular taser plugin) or VehicleManager's askEnterVehicle and askExitVehicle. (Blacklist Vehicle Plugin).

If you do think this could cause issues, it would be nice if we had a way to tell clients they cannot change their movement or gesture, currently using Harmony is the best solution it seems because it stops other clients from being informed of the action but it isn't perfect - clients themselves can sometimes see themselves standing up/"escape" being locked in prone.

SDGNelson commented 4 years ago

Having an event for enter/exit vehicle prevention makes a lot of sense, opened here: #956

Messing with gesture and stance might break things occasionally because in vanilla stance is mostly deterministic, but it shouldn't affect this issue. I'll investigate the client side of things just in case.

SDGNelson commented 4 years ago

After double-checking the client movement code:

One indicator that it's clientside would be if the other plays are stuck, but are playing the walking animation. That would mean they are receiving the latest state, but that they are not being moved toward it.

Thankfully the network interpolation code is shared between players, zombies, vehicles, etc, so if we don't have the same bug with the other ones that's at least something as well.

Re-reading the OP reminded me of your comment that it's somehow related to them being in the inventory or not, which would be related with the gestures brought up by Pigeon-cmd. Are you using the tazer plugin?

Bradler26 commented 4 years ago

Sorry for closing the issue. I'm on my phone and hit the wrong button lol.

Yes I do use taser plugin but I unloaded it and still got the glitch so I think it's unrelated.

Bradler26 commented 4 years ago

Another thing I should mention is that I have "misc.dont_enable_this true"

In my execute.config

I believe this cancels server side movement checking which helps a lot with rubberbanding which I've always liked. I'm aware it exposes my servers to hackers but I've never had any serious problems with it for the last few months. Could this by any chance change how the glitch happens or maybe why it occurs?

SDGNelson commented 4 years ago

Yep misc.dont_enable_this trusts client movements, but it looks like that shouldn't have any impact on replication.

ghost commented 4 years ago

Hey,

To give an update on this, I believe this is 100% plugin induced now, maybe also caused by mods, but it happened on a vanilla/no mods server.

We had no issue on this before with a completely vanilla server until today when we updated a few plugins along with the unturned update. The plugin we updated don't do anything special - no Harmony but it's the only difference between other servers we have without this issue.

Going to give it some time to make sure it wasn't your update (unlikely though - as I don't think you messed with anything that would cause this more).

I have a unity log though from when it happened on me,

It was spamming Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak

A fair amount, no idea if it was related or if there's anything useful in there, nothing stands out to me in there besides that one error spammed a million times.

To add a bit more context, it happens to both Vehicles and players and they just freeze where they last teleported. (i.e spawn a vehicle 10 feet above you with /v and it will stay there, floating only for that client.)

Thanks for your help and care on this issue

UnityDebug_Client.log

Bradler26 commented 4 years ago

The vehicle glitch also occurs to me as well so I'm sure it's all related

SDGNelson commented 4 years ago

If it's happening with vehicles and players, have you seen it happen with animals or zombies as well?

Reading online it appears that JobTempAlloc warning is related to the newer Unity job system, but vanilla Unturned doesn't use that. (Vast majority of game code from versions prior to its introduction.)

Bradler26 commented 4 years ago

It does happen with animals aswell. Haven't checked zombies but definitely with animals

ghost commented 4 years ago

A Player mentioned it froze animals, but unfortunately our effected servers are our RP Servers - with no zombies and very rare animals.

Thank you for your help on this, hate you bother you about something which just might be a plugin issue on our end but this has bothered us (and other server owners) to no end and nothing obvious causing it.. That unity log I sent you was from it happened to me to JUST Vehicles and not players, which just makes it more confusing.

SDGNelson commented 4 years ago

What I'll do for the next update is add logging to a few places to try and track down where the problem is occurring e.g. does the client have animals but hasn't received an update for a while, are they receiving empty updates, is the server sending updates, etc. Map update is planned for Friday so if everything goes to schedule we should have it sorted out soon. 🤞

SDGNelson commented 4 years ago

In Friday's update if you can enable the "-NsbLogging" (network snapshot buffer) command-line option we should hopefully get an idea of where in the system the problem is occurring. All of this code is wrapped in a compiler switch so we can get rid of it easily once it's fixed. If the main server warning isn't getting triggered then we'll need a UnityDebug_Client.log file from the Logs folder from a player who experienced it on the server.

The key potential problems that should be logged are:

Serverside:

Has any player gone a long time (10s+) without being sent a packet of each active type?

Clientside:

Are we consistently receiving snapshots out of sequence?
Are we receiving empty updates?
Are we receiving updates for entities we don't know about?
Is the buffer null when an entity receives a snapshot?
Have we gone a long time without receiving any player snapshots? (Players-only for this one because there are valid cases to not receive any animal/vehicle/zombie updates.)

SDGNelson commented 4 years ago

Any luck with the logging so far? 🤞

ghost commented 4 years ago

Not quite yet, we're still working on finding the exact issue. Reports of this bug happening have dropped significantly after we made some changes changes to a few of our own custom plugins. We did not track every change we made, as we were just trying to fix it as quickly as possible. Although I believe Bradler26 is still affected. I will try to corner the issue on a test server in a controlled environment and get back to you.

ghost commented 4 years ago

Here's logs when it happened to almost straight after joining:

UnityDebug_Client.zip

SDGNelson commented 4 years ago

Thank you! There are several interesting points from that log:

Quite a few updates are received prior to creating the buffers on the client. This does not appear to be the cause of the final problem, but I should look into it as well.
Multiple times the client goes more then 10 seconds without receiving an update.
Somehow the client sequence number gets set to a huge value (in this case 452,102,803) and starts ignoring new updates from the server (~58,000). This is what causes the players (and presumably animals / zombies / vehicles) to appear frozen. Immediately before that happens we get a bad channel number (1752432912), so I think the packet is getting corrupted.

Do you know if any plugin is bind Provider.onServerWritingPacket? That's the most likely cause plugin-wise, otherwise there's a bug in the game networking code.

Trojaner commented 4 years ago

Such corruptions could be caused by plugins when using multithreading too. Since Unturned doesn't lock anywhere (for example when sending packets) other packets get corrupted. I have noticed this issure when I tried to implement an async chat feature which resulted in all kind of weird behaviours.

Bradler26 commented 4 years ago

I haven't had time to test any of the new logging but I know for a fact that many of my plugins use multithreading. I'm confused at why u didn't have this issue until that one Unturned update.

Trojaner commented 4 years ago

The issue isn't plugins using multithread Brad. I am just mentioning that such issues can happen when plugins try to use multithreaded game networking. Usually plugins don't do this but maybe Pandahut ones do?

SDGNelson commented 4 years ago

What I'm doing as a "temporary" medium-term fix for today's .1 patch is write the sequence number to both the start and end of the message, so that if they don't match the client knows something went wrong. Ideally the underlying problem will be tracked down, and I'll leave this issue open to see if anything comes up with onServerWritingPacket or another cause is discovered.

Bradler26 commented 4 years ago

I'm currently in the process of swapping dedicated machines but I will be sure to send some logs and info once the swap is complete. Thanks for the help so far.

ghost commented 4 years ago

@Trojaner None of our plugins manually send or receive packets - I was thinking this could maybe be Avi related as his plugins do have an async chat configuration option and they let players join past the default max players cap. However, his plugins alone appear fine on our servers, something on RP specifically which causes this that both us and brad have in common. Thank you for trying to help us with this.

What's stranger here is that this issue is completely unreplicable on a server with just two people with the exact same plugins which cause it.

Trojaner commented 4 years ago

I definitly do not recommend enabling any async chat functionality. I know Avi uses the same method as me and this results in corrupted packets exactly like mentioned here. It can also lead to weird chat messages popping up, random players dying, players getting kicked etc since all kind of malformed packets can occur.

ghost commented 4 years ago

I definitly do not recommend enabling any async chat functionality. I know Avi uses the same method as me and this results in corrupted packets exactly like mentioned here. It can also lead to weird chat messages popping up, random players dying, players getting kicked etc since all kind of malformed packets can occur.

It couldn't hurt to try disabling it to see if that helps resolve it, but it is worth noting that we have Avi on 4-5 of our servers with that option enabled (it's enabled by default) and have not had a single issue on those servers with that. No Chat bugging out, random deaths, anything. It could still be the interaction of another plugin with Avi though.

Trojaner commented 4 years ago

Avi could have disabled the feature. Without any doubt async chat and other multithreaded networking code causes these issue since Unturned does not use any locks or queues in it's networking code.

ghost commented 4 years ago

Just to confirm here - Avi does patch the Provider Methods will try to look in further if it is truly him causing this issue as somehow a result of interacting with other plugins - Avi alone is fine.


[08/05/2019 16:50:32.941] [INFO] Patching steam calls 
[08/05/2019 16:50:32.946] [INFO] Successfully patched steam call receive ⬇️
[08/05/2019 16:50:32.948] [INFO] Successfully patched steam call send(NMTA) ⬆️
[08/05/2019 16:50:32.952] [INFO] Successfully patched provider send ⬆️
[08/05/2019 16:50:32.954] [INFO] Successfully patched steam call closeWrite(NST)
[08/05/2019 16:50:32.956] [INFO] Successfully patched steam call closeWrite(NMT)

joeymisfit commented 4 years ago

Hey @SDGNelson

Thank you a ton for being patient with @Bradler26 as both him and I have been working tirelessly to get his server back to normal or semi-normal working standards. From what I've read above I believe you are already aware his server is a full-on rp server with several workshops and several plugins that make a players experience interactive and less is required from staff members.

From the amount of time I've spent on the back-end of his dedicated server reviewing logs, testing plugins, and fixing memory leaks along the way I've also realized while being in-game that when people freeze zombies and animals freeze as well, that's when you have to restart your client to be able to see anyone moving for a short period of time before it happens again. When you don't re-log it eventually ends up timing you out of the server. Speaking of threads, most of Brad's plugins have been custom made to use their own threads so that it doesn't just use the one Unturned thread, however it seems that the one thread tends to get unstable and simply freeze console allowing other threads to respond in the window and not Unturned; meaning people can type via DiscordHook and it will show up but that's about it, until a hard exit of the instance is made.

I'd also like to take a moment and thank you for adding the network snapshot buffer as it helps alot of us server owners and developers alike with diagnosing server problems and getting those fixed rather fast. I did however have a small question that I'm sort of curious about; back in the old days we used to be able to go into Unturned properties on the Steam Library and set advanced Options to give the game more threads, memory, and force certain dx levels, I'm wondering if that is still possible and if it would work server side as well?

Thanks, and I'll be emailing you shortly with a few logs I've been documenting as of late.

SDGNelson commented 4 years ago

@PandahutMiku: Thanks for looking into that! To provide some more context on the potential threading / async issue: Unturned's network-related code mostly uses a single buffer to read and write from, so if a plugin from another thread is using the same Block, SteamPacker or SteamChannel functions it will corrupt packets. Note that if the plugin is on the game thread then that is fine.

In general: Have you guys run into any more reports of this following the .1 patch early Monday? (It should not be happening anymore.)

@joemisfit: The everything freezing at once sounds like potentially a separate issue, and I can imagine I might need to make a similar change to how player inputs are acknowledged. For the most part any code that interacts with unity and game features should not be on a separate thread, so it would not surprise me if that's a major cause of problems on the server. To clarify on the -NsbLogging feature, I've disabled that in .1 now that we know what the problem was, but I will keep an eye out for the logs thanks! Most of those launch options shared by players back in the day were placebo, e.g. players trying to use TF2 or Minecraft launch options, but any launch options from the Unity documentation should work.

SmartlyDressedGames / Unturned-3.x-Community

Bugs I've found #882