Random huge input lag only while cactbot is running

[Jamayyy]

Description

The bug occurs when:

• Fresh ACT + overlayplugin + cactbot install
• Using 1x Default Cactbot Raidboss Alert Only, AND/OR 1x Default Cactbot Raidboss Timeline Only
• While raiding (P5S - P8S)

While Cactbot is working (during the raid encounter), randomly the bug occurs. It happens about 1 time per pull, sometimes multiple times, sometimes not at all. It can also happen before a pull (during countdown or just standing around and discussing strategy). But it seems to never happen outside of raid zones (for example at FC house or limsa).

The bug symptoms:

• ALL my keyboard inputs are being delayed (lagging behind) by about 1 to 5 seconds. For example, I start walking forward (W key down), then the bug occurs, then I stop walking (W key up). Due to the bug, the (W key up) is delayed by about 1 to 5 seconds, which means I cannot stop walking. I watch my character walk off the map or into a mechanic and die.
• Another example: I press enter and start typing a sentence. I look at the chat window and it's empty, because the bug occured. I stop typing. 5 seconds later, a ghost is typing my sentence into the chat window, at exactly the same speed as I wrote it on the keyboard. So the time between keypresses is normal, except they're all delayed by a very large amount.
• After a random time (feels like 3 to 8 seconds), the bug dissappears and all my inputs are instant again.

Additional information

• From what I can tell, all keys on my keyboard are delayed. For example Alt+Tab will also be delayed, and even after the alt+tab completes, the bug still continues.
• Mouse input still works normally, only keyboard is affected.
• I'm using a normal wired mechanical USB keyboard. (nothing bluetooth that could drop connection)
• It only happens in FFXIV with ACT running with the cactbot plugin enabled. If I disable the cactbot plugin, it doesn't happen anymore. I've raided the first 3 weeks of the patch without cactbot and it never happened then. As soon as I enabled it after week 3, it started happening consistently.
• It's a modern computer (Windows 11 21H2, AMD Ryzen 7 2700X, Nvidia RTX 2080), so it's not a performance issue. There are no FPS drops during the bug (always around 100 FPS). I don't have any bad software running in the background, watching task manager shows normal operation during the bug
• It doesn't matter whether I use TTS or not, so it's not TTS related
• I disconnected my controller, so it can't cause it
• I'm using FFXIV QuickLauncher with Dalamud, not sure if that makes a difference
• I have many video recordings of the bug, but you'd just see me running around stupidly in the middle of a fight and cancelling my own casts, so it doesn't help much
• I have a feeling the announcer causes the problem more often than the timeline, but I confirmed that both cause it.
• I've been trying to isolate or fix this bug for the past 2 weeks, I'm at my wit's end, please help. If there's anything I can provide you with, I'll do it.

[Jamayyy]

I attached some recordings of the bug occuring. Some of the recordings show the same mechanic twice, but the timing of the bug is slightly different. When the character keeps running due to the input lag, I spin the camera in an attempt to not run off the map.

You can see when the bug occurs when I stop casting my GCDs (because I'm unable to).

If you turn the sound on, you can hear the cactbot announcer.

Also, something that I noticed, in 4 of the 6 videos the bug occurs a few seconds before an announcement.

https://user-images.githubusercontent.com/52106450/192886453-0ae06cf8-ba70-431f-98f8-7d05fe2d4ad7.mp4

https://user-images.githubusercontent.com/52106450/192886468-cf93bf28-e18a-4585-a41a-6762556cc144.mp4

https://user-images.githubusercontent.com/52106450/192886480-d014ed26-1c0b-4101-896e-f66df9c5de9e.mp4

https://user-images.githubusercontent.com/52106450/192886486-e0d1e514-edd9-4bfb-b5e1-3fbdd753f627.mp4

https://user-images.githubusercontent.com/52106450/192886491-3c51bf6b-01bf-49cd-8f6c-ee681dae4c3b.mp4

https://user-images.githubusercontent.com/52106450/192886501-2f2553ef-c4c8-4017-9a05-39c362c1e08f.mp4

[Jamayyy]

Today it appears that I have accidentally entered a permanent state of this bug! I was just waiting for the hunt train to start, when suddenly the input lag started. I alt-tabbed and noticed that everything was extremely laggy. I check Task Manager and ACT uses 100% of my CPU! Since this time the bug didn't fix itself after a few seconds, I observed it and found the following:

• The huge input lag is a byproduct of 100% CPU usage. I still get high FPS (this apparently only depends on GPU performance), but most ingame actions are extremely janky. All movement is delayed, other players and models don't load in anymore, if I use my mount then I'm invisible for 5 seconds until finally the mount loads in, other players casts are either not happening at all, or then suddenly all happen at once (I saw their castbars in the party list flicker extremely quickly between like 10 different skills)
• After about 2 minutes, ACT gave the error notification: Log Parsing lagging behind. Log line parsing has been behind for 10 seconds. A plugin subscribed to BeforeLogLine/OnLogLine may be processing synchronously and taking too long. I assume this plugin is Cactbot, because that's the only one I have installed, other than the overlayplugin (fresh install, not using any features except the cactbot announce+timeline)
• After another minute, ACT gave the error notification "Failure: Unable to access network data. This may be due to Windows Defender Firewall [...]". (My firewall rules are fine and I don't run any antivirus or firewall software, this is probably another byproduct of the 100% CPU usage)
• when I alt-tab into ACT, even my mouse cursor starts having extreme lag. The mouse cursor updated at like 1 FPS and took half a minute to complete a simple movement.
• I think it's clear that cactbot locks itself in an infinite calculation loop, which may be during BeforeLogLine/OnLogLine, and usually exits the loop after about 3 to 5 seconds. What could this be?

Edit: I've uploaded the log after this occurence: Network_26606_20220929 - Copy.log

The bug started occuring somewhere between 2022-09-29T22:33 and 2022-09-29T22:40, I'm not sure. If you do ctrl+f for "TCPDecoder: >2", you can find some messages that I find suspicious, they're probably a symptom of the lag, not the cause:

251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: >2 sec since last processed packet, resetting stream.|60de99403c3aca7e
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114903161].|0fe7d9dd1466f190
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114902680].|bb5df36aa1129c58
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114902592].|e4538db8849dbb53
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114902353].|3a99b1365679f7ff
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114902131].|66a83bce62d7878f
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114901745].|9149f608f2a2175f
251|2022-09-29T22:39:29.7660000+02:00|TCPDecoder: Missing Sequence # [4114703804], Dropping packet with sequence # [4114901641].|4723d50ab53ca60c
...

Edit 2: I managed to match a log to a video of the bug. This is the log: Network_26606_20220928.log The bug occurs between 2022-09-28T22:14:05.8080000 and 2022-09-28T22:14:10.0890000 Which corresponds to 0:09 to 0:13 in the below video:

https://user-images.githubusercontent.com/52106450/192886491-3c51bf6b-01bf-49cd-8f6c-ee681dae4c3b.mp4

[quisquous]

I'm sorry this is happening. I really am not quite sure what is going on here. Thank you for all the videos and logs, but I am not sure that any of them are going to particularly help get to the bottom of this. The TCPDecoder issues are a symptom of the same issue; because it's fallen behind so much it thinks that it's gotten out of sync. I took a look through the cactbot code and it's definitely not clear to me what might be causing this.

A couple of random thoughts:

You have a lot of p7s in there. One wild thought is that it's stuff cactbot is using from the new OverlayPlugin. Could you try using an older version of OverlayPlugin that doesn't use getCombatants from memory, e.g. https://github.com/OverlayPlugin/OverlayPlugin/releases/tag/v0.19.3? Maybe something is going awry with that call?

What is your DPS update interval seconds and Enmity update interval ms in Plugins -> OverlayPlugin.dll -> Event Settings? Can you turn those way up to like 10 seconds and 10000ms and test with that? Maybe a high update rate is causing C# garbage collection issues which in turn is causing issues with cactbot?

If you just have the cactbot plugin but no overlays using cactbot, does the issue occur?

I think it would also be worth doing a test without Dalamud, and also listing which plugins you're using.

[quisquous]

Is your language set properly in the ffxiv plugin? (ACT -> Plugins -> FFXIV ACT Plugin -> Language.) This is re: https://github.com/OverlayPlugin/OverlayPlugin/pull/84

[Jamayyy]

Hey, thanks a lot for trying to help me out! I've been testing most of the things recently, plus some more I came up with. I just wanted to let you know that I'm still testing and I'll update the issue when I have more conclusive evidence. It hasn't been happening that frequently lately, which is good for my raiding, but bad for my troubleshooting.

[panicstevenson]

Really random thought, but is something happening where now the Chrome overlay is admin, shifting your window focus out of the game and eating your inputs?

[Jamayyy]

Really random thought, but is something happening where now the Chrome overlay is admin, shifting your window focus out of the game and eating your inputs?

Chrome overlay? Do you mean something from the Chrome browser? I'm using Brave Browser so I probably don't have it. I also thought that it might be related to some overlay stealing focus, but I don't use any (disabled discord and disabled steam overlay). I also disabled Discord notification popups to rule that out too. But thanks for the suggestion

[quisquous]

Hi, were you able to try any of my suggestions above? Is this still happening?

[Jamayyy]

Hi, thanks for checking up I've been taking a break from the game since the last time (waiting for new raids / content). However I remember trying everything suggested here and nothing really got rid of it. I'm thinking that it might have a connection to my HDD, but I don't see how. I use an m.2 SSD and the game is installed on it, as well as the OS, so the HDD should be completely irrelevant. But I've read online and heard from some people that it could cause slowdowns even in that case? Not sure I believe it.

I also noticed that I get the same kind of keyboard lag right when I start the computer, while it's still loading the OS. If I instantly open the browser and type into the address bar, the numbers appear very slowly and delayed like during the ffxiv bug. But after a few seconds the computer finishes loading and things stay normal from then on. Apparently the game with cactbot open caused the same symptoms.

In conclusion, I have no idea why this happens and nothing helped so far, maybe it's related to my system somehow if nobody else experiences this

[quisquous]

Thanks for the followup. Sorry that there's no clear answers here. I still don't know why cactbot would be causing these issues specifically (and if it were, I'd expect it to be happening more in cef browser subprocesses rather than in the ACT process itself, cpu%-wise).