Jockey teleport bug on Ubuntu versions newer than 18

LuckyServ commented 2 years ago

With Ubuntu 18 hardware and maintenance updates ending in April 2023, we're approaching time to really consider upgrading the OS on which we host servers on.

Description

A jockeyed survivor sometimes gets teleported in another location, often outside the players space on servers running later versions of Ubuntu.

Reproduction steps

No reproduction steps so far

Related fixes / issues

https://github.com/SirPlease/L4D2-Competitive-Rework/pull/221 (detailed explanation on blood splatter fix by ProdigySim)
https://github.com/J-Tanzanite/Little-Anti-Cheat/issues/48

Potential candidates - plugins or extensions that mess with jockey movement

WaterSlowdown.sp changes some movement flag (m_flLaggedMovementValue) on the jockey when riding a survivor on water, but afaik the bug has happened outside of water
z_jockey_control_variance removes the ability of survivors to counter movement and I know the dev team had a hard time implementing the control logic so perhaps it can't handle a value of 0 as well as it should

It would be helpful if anyone (perhaps @Target5150) who I heard has servers on Ubuntu 20 can provide patterns / repro steps, as well as other sources of modifications to the jockey movement in this repo.

lunatixxx commented 2 years ago

For me it was happening even with the default value of z_jockey_control_variance so it is not that

This plugin was fixing 90% of the teleport i will say. https://forums.alliedmods.net/showpost.php?p=2741956&postcount=42

And in addition of the lilac plugin, i did not see one teleport in months.

Krevik commented 2 years ago

Here is my version of that plugin cause I am encountering this problem on my servers. https://github.com/Krevik/Kether.pl-L4D2-Server/blob/kether_2.0/addons/sourcemod/scripting/l4d2_jockey_unteleport.sp

It doesn't have larry's features like map excluding, but I am running the version above since few months on my servers and it worked in 100% cases of jockey teleport. Even if the teleport origin (0, 0, 0) is inside the map. However (I did not check it) it may create some conflicts for e.g. if you use !teleport command on jockeyed player, it will probably force him back to previous position. Anyway I prefer this solution of the problem, cause there's no need to teleport jockey/jockeyed survivor manually with a command, if this antiteleport works for literally each case of jockeyed survivor teleport.

Only once in my lifetime the teleport bug was caused by charger (maybe charger and jockey jumped at the same moment?), so if such situation happens again, I will expand it's functionality to cover charger also. Also there was some fix in this repo... jockeyed charger fix? I think that mentioned fix might prevent charger induced teleportation bug.

Also I was trying to find any pattern but it looked totally random - sometimes there is no teleport for weeks, sometimes there are few teleports during one map. My guess is that it's mechanism is connected with the well-known problem where infected see survivors not moving and remained at their starting positions <-- happens often if infected minimizes the game or infected player has ping spikes. And as lunatixxx mentioned, for me it happens even with default value of z_jockey_control_variance.

Another problem (I am aware that's not the topic of this issue, however it is related to jockey and probably to operating system of the server) of jockey that is present PROBABLY due to newer ubuntu version is that although l4d2_unsilent_jockey is present and loaded, some of the jockeys remain silent BUT only for some clients. As I see that topic would be especially investigated by people using or testing newer ubuntu versions, could anybody confirm the problem?

My servers' OS: Debian 10 My servers' tickrate: 108

ProjectSky commented 2 years ago

teleport is random, here is a same teleport issues. https://github.com/J-Tanzanite/Little-Anti-Cheat/issues/48#issuecomment-922630124

lunatixxx commented 2 years ago

Pretty sure it is not related to ping or alt tab, this happen between two bots too.

lunatixxx commented 2 years ago

I just discovered something interesting, i was testing things on my server and i had to go spectator so i did !spec command. Guess what it teleported me (the bot i was using) out of bounds and triggered the anti jockey teleportation plugin which does not make sense because i was in ready up and so no infected and without players it gaves me this error wich is correct.

L 07/27/2022 - 10:42:18: [SM] Exception reported: Client index -1 is invalid (arg 5)
L 07/27/2022 - 10:42:18: [SM] Blaming: l4d2_unteleport_jockey_fix.smx
L 07/27/2022 - 10:42:18: [SM] Call stack trace:
L 07/27/2022 - 10:42:18: [SM]   [0] LogAction
L 07/27/2022 - 10:42:18: [SM]   [1] Line 262, plugin.sp::JockeyRideCheck_Timer

Just a guess but the game could have think that there is a jockey on the survivor but when it realise that there is not it brakes and teleport him out of bounds. That will explain why it teleport the survivor when jockeyed sometimes, maybe the game think that there were already one jockey on the survivor ???

Might be something that does not clear from memory or with the jockey event i suppose.

By the way the teleport does not always occur on 0 0 0 position, depend of the map i think.

lunatixxx commented 2 years ago

I can now safely say it is something related to touch links in the game. I installed this plugin that is supposed to block an exploit on team change: https://github.com/shqke/sp_public/tree/master/remove_touch_links

It litterally triggered 4 teleport in less than one minute even when jockey was dead, it even teleported a hunter there were nobody named like that.

"[!] Attempting to fix position of Hunter after jockey teleportation bug."

Removed the plugin and it stopped happening. Just to be clear the occasional jockey teleport does not happen because of this plugin, but there is something related between both on the cause.

lunatixxx commented 2 years ago

These maps are 100% of the time never fixed by jockey_unteleport, only admin can teleport back sadly.

-Sacrifice 2 (near boat) -Dark carnival 4 (near roofs) -Dark carnival 5 (out of bounds) -Hard rain 4 (under a house) -Parish 1 (start area) -Cold stream 3 (inside water)

lunatixxx commented 1 year ago

Well see you in 2023, i suppose that it will get more attention when people got randomly teleported in 25% of their ceda games.

SirPlease commented 1 year ago

I appreciate the effort you've put into testing several plugins and methods over the course of time, I do feel like this is something to be fixed by our bigger brained assembly warriors. A plugin to treat the side effects is not optimal in this case. 😓

jensewe commented 1 year ago

void CTerrorPlayer::OnLeptOnSurvivor(CTerrorPlayer *pVictim)
{
    ...

    this->m_impactTimer.Start(0.4);
    this->m_impactDuration = 0.4;

    ...
}

void CTerrorPlayer::UpdateLeap()
{
    ...

    Vector vecVictimOrigin = pVictim->GetAbsOrigin();
    Vector vecOrigin = this->GetAbsOrigin();

    // m_impactTimer, m_flImpactDuration, m_bFinishedImpact
    // No idea what their actual names are, pure guess.
    if ( this->m_impactTimer.HasStarted() && !this->m_impactTimer.IsElasped() )
    {
        float flFactor = 1.0;

        if ( this->m_flImpactDuration != 0.0 )
            flFactor = 1.0 - (this->m_impactTimer.GetRemainingTime() / this->m_flImpactDuration);

        vecOrigin += (vecVictimOrigin - vecOrigin) * flFactor; // slightly close to victim over time
        this->SetAbsOrigin(vecOrigin);

        if ( this->m_bFinishedImpact )
            this->m_bFinishedImpact = false;
    }
    else if ( !this->m_bFinishedImpact )
    {
        this->m_bFinishedImpact = true;

        this->SetAbsOrigin(vecVictimOrigin);
        this->SetParent(pVictim, -1);

        this->SetMoveType(MOVETYPE_NONE);
    }

    ...
}

Personally no idea what exactly raises the issue, just something I'd like to share. Convinced myself that in demo playback above codes could lead to a visual issue where a riding jockey dies and gets teleported to somewhere like 0,0,0 (i.e. c2m1_highway, 0,0,0 is mid-air above the final hill if I remember). And, I kinda believe all the teleport scenes happened within the 0.4s, from what I've witnessed before.

Krevik commented 1 year ago

I appreciate the effort you've put into testing several plugins and methods over the course of time, I do feel like this is something to be fixed by our bigger brained assembly warriors. A plugin to treat the side effects is not optimal in this case. 😓

I totally agree with you, but imo it is better to treat side effects, than letting the bug to destroy a lot of games

SirPlease commented 1 year ago

I totally agree with you, but imo it is better to treat side effects, than letting the bug to destroy a lot of games

Oh a 100%, I'm just less worried about it in the here and now as I don't have this issue as we're still running out stuff on soon to be very oudated Ubuntu. 😁 Sweat droplets are starting to form though, for sure.

A1mDev commented 1 year ago

This bug does not depend on linux versions in any way, besides it is on all servers, since these are shortcomings in the code.

ProjectSky commented 1 year ago

This bug does not depend on linux versions in any way

i don't think so. random teleport never appeared when my server was not upgraded to 20.04. maybe some newer library triggers this issues?

A1mDev commented 1 year ago

Do not forget that 2 bugs with a jockey, they are in different places.

SirPlease commented 1 year ago

The bug literally appears the moment you upgrade to anything newer than Ubuntu 18.04 (Didn't check 18.10). It is not on Servers that are running 18.04.

It is not a reach to assume this, especially considering the shotgun spread (covered in main post) had the same effect on anything newer than 18.04. However this time, I'm not so sure if it's something on our end that is messing things up.

lunatixxx commented 1 year ago

There are some circumstances where jockey ride event does not unhook propely i think. We called a vote to restart the map while a survivor was jockeyed this person was still considered jockeyed for the whole game and caused teleportation bugs on other players.

Does there is a way to kill an event manually ?
https://sm.alliedmods.net/new-api/events/Event/Cancel

lunatixxx commented 1 year ago

As far i remember (could be wrong) but all teleportations generally happen shortly after round start on first jockey riding, which makes me think this is really related to map change and second round is also considered as a map change by the way. The only thing i don't understand is that even after a server restart it can happen on first map. Maybe if we could cancel jockey ride event on every round start it will help.

SirPlease commented 1 year ago

Does there is a way to kill an event manually ?

Killing an event doesn't mean it doesn't happen, it just doesn't fire its information forward. You'll have to block whatever you're trying to block deeper in the code.

As far i remember (could be wrong) but all teleportations generally happen shortly after round start on first jockey riding, which makes me think this is really related to map change and second round is also considered as a map change by the way. The only thing i don't understand is that even after a server restart it can happen on first map. Maybe if we could cancel jockey ride event on every round start it will help.

It is not tied to round start, Jockeys have been teleporting at the very end of the map just as many times as right away.

ProjectSky commented 1 year ago

Does there is a way to kill an event manually ?

Killing an event doesn't mean it doesn't happen, it just doesn't fire its information forward. You'll have to block whatever you're trying to block deeper in the code.

As far i remember (could be wrong) but all teleportations generally happen shortly after round start on first jockey riding, which makes me think this is really related to map change and second round is also considered as a map change by the way. The only thing i don't understand is that even after a server restart it can happen on first map. Maybe if we could cancel jockey ride event on every round start it will help.

It is not tied to round start, Jockeys have been teleporting at the very end of the map just as many times as right away.

correct, it happens to my server many times

lunatixxx commented 1 year ago

Does there is a way to kill an event manually ?

Killing an event doesn't mean it doesn't happen, it just doesn't fire its information forward. You'll have to block whatever you're trying to block deeper in the code.

As far i remember (could be wrong) but all teleportations generally happen shortly after round start on first jockey riding, which makes me think this is really related to map change and second round is also considered as a map change by the way. The only thing i don't understand is that even after a server restart it can happen on first map. Maybe if we could cancel jockey ride event on every round start it will help.

It is not tied to round start, Jockeys have been teleporting at the very end of the map just as many times as right away.

But it will depend if the jockey managed to do an attack, it could be at the end if it is the first jockey ride of the game.

#pragma semicolon 1
#include <sourcemod>
#include <sdktools>
#include <readyup>

public Plugin myinfo = 
{
    name = "Debug jockey ride event",
    author = "Lunatix",
    description = "Test plugin to see if the jockey ride event always end correctly",
    version = "1.0",
    url = "http"
};

public OnRoundIsLive()
{
    for (int client = 1; client <= MaxClients; client++)
    {
        if (client <= 0 || client > MaxClients) return;

        if (IsClientConnected(client) && L4D2_IsSurvivorBusy(client))
        {
            PrintToChatAll("[Debug] Jockey ride event did not end correctly on %N", client);
        }

        if (IsClientConnected(client) && IsPlayerAlive(client))
        {
            if (GetEntProp(client, Prop_Send, "m_zombieClass") == 5)
            {
                PrintToConsoleAll("[Debug] Old jockey still considered as alive wtf???", client);
            }
        }
    }
}

stock bool L4D2_IsSurvivorBusy(int client)
{
    return GetEntPropEnt(client, Prop_Send, "m_jockeyAttacker") > 0;
}

jensewe commented 1 year ago

There are some circumstances where jockey ride event does not unhook propely i think. We called a vote to restart the map while a survivor was jockeyed this person was still considered jockeyed for the whole game and caused teleportation bugs on other players.

Does there is a way to kill an event manually ? https://sm.alliedmods.net/new-api/events/Event/Cancel

To be clear though, event doesn't do anything but provide information at certain time. Guess you'd like to talk about internal CTerrorPlayer::OnRideEnd(). There are actually some other similar cases like charger's queued pummel, when Survivors get charged right before next half begins they are permanently frozen in place on restart. In this way, it makes sense for jockey teleports with the underlying matter of cleaning state.

lunatixxx commented 1 year ago

Well this is the perfect example. https://streamable.com/86yy6s

I am sure that at this moment the game was still thinking that jockey was riding nick, causing this glitch when it jump on the other survivor. The charger colliding and taking the target too quickly triggering the bug i guess, something happening on the frame probably.

Maybe test in this direction and try to reproduce it.

A1mDev commented 1 year ago

This has nothing to do with the fact that netprop 'm_jockeyVictim' is not reset, otherwise you would hear sound 'JockeyZombie.Ride'. This sound starts when function 'CTerrorPlayer::OnLeptOnSurvivor' is called and ends when function 'CTerrorPlayer::OnRideEnded' is called. If no sound is heard, then this function was called and reset netprop 'm_jockeyVictim'. More like a problem with some code in plugins.

lunatixxx commented 1 year ago

The bug literally appears the moment you upgrade to anything newer than Ubuntu 18.04 (Didn't check 18.10). It is not on Servers that are running 18.04.

I wish official servers stopped running on outdated systems, so more people will start complaining about all these bugs. I'll not be surprised that the invisible player one is because of that.

A1mDev commented 1 year ago

If function 'CTerrorPlayer::OnRideEnded' had not been called you would have run into a lot of problems, you would have noticed it right away. For example, the jockey's ability would not be on cooldown.

A1mDev commented 1 year ago

The bug literally appears the moment you upgrade to anything newer than Ubuntu 18.04 (Didn't check 18.10). It is not on Servers that are running 18.04.

I wish official servers stopped running on outdated systems, so more people will start complaining about all these bugs. I'll not be surprised that the invisible player one is because of that.

Can you share a demo with this bug?

SirPlease commented 1 year ago

Just a little status update, I decided to take a jab at writing a plugin for this and it seems to be resolved. The reason I say "seems", is because there's no easy way for me to reproduce the issue reliably other than forcibly teleporting the player manually while jockeyed.

A few steps I've taken with my approach;

I've set a safeguard of 1000 units difference within a single frame to prevent possible issues, while it's a seemingly wide margin, it's fair to say that these teleports are much further away.
We'll be using the expensive PreThink during the entire Jockey Ride, meaning it'll start once we've confirmed the jockey_ride event and it'll end as soon as the jockey is cleared by relying on netprop m_jockeyAttacker in our PreThink check. Which should minimize the impact this has on the servers while still efficiently preventing the jockey teleport cheat and glitch.
Spectate/bot takeover/disconnect support is added.

I'll push the plugin to the github in a few after this post, I'll go through a few more iterations and cleaning before doing so.

lunatixxx commented 1 year ago

Not fixed, and this was less than 1000 units i guess. Plenty of maps are like that. https://streamable.com/3q4mnd

SirPlease commented 1 year ago

Thanks for reporting, fairly certain we can use a much lower unit difference as we're using OnPreThink rather than a previously used method that relied on repeating sourcemod timers, I'll lower it to 400 and see how it goes. 👍

ProjectSky commented 1 year ago

maybe more reliable to check for out of bounds using TR_PointOutsideWorld

if (GetVectorDistance(safeVector, preVector) > MAX_SINGLE_FRAME_UNITS || TR_PointOutsideWorld(preVector))
{
    TeleportEntity(client, safeVector, NULL_VECTOR, NULL_VECTOR);
    return;
}

jensewe commented 1 year ago

I guess it should be known that sqrt can be really slow and should be avoided. So I'm indeed suggesting:

#define MAX_SINGLE_FRAME_UNITS_SQUARE 160000.0
if (GetVectorDistance(safeVector, preVector, true) > MAX_SINGLE_FRAME_UNITS_SQUARE) {
    ...
}

Hope that I'm not spilting hairs but it's SDKHooks_PreThink, as massive as OnGameFrame calls, why not?

A1mDev commented 1 year ago

Anyway this can't be a bug fix.

SirPlease commented 1 year ago

maybe more reliable to check for out of bounds using TR_PointOutsideWorld

Relying on tracerays doesn't seem like the most cost efficient method. We're talking about prethink here, which fires constantly, a client moving 400 units within that tiny span is practically impossible and a fairly reliable method.

Anyway this can't be a bug fix.

I suppose it depends on your definition of a fix, this is indeed a band-aid as I've said before, but I haven't seen anyone stepping up to actually fix the issue at hand, so all we're left to do is using a bandaid.

It addresses the issue at hand without interfering with the actual game, seems like a good enough fix to me.

Hope that I'm not spilting hairs but it's SDKHooks_PreThink, as massive as OnGameFrame calls, why not?

We only hook on the jockey event and unhook after clear, OnGameFrame would be slightly more expensive as you'd have have that running at all times.

SirPlease / L4D2-Competitive-Rework