Unit orders being lost or canceled at random during multiplayer games

[IceColdCoder]

Describe the bug Orders, both single and queued are being canceled at random. Affects all command types. If a unit is given a move order it will randomly stop. If a unit is given amn attack order (any type), it will randomly cancel or "lose" the order (I suspect). The issue does not cause any other orders to be issued (eg attack is replace with patrol or somesuch). The lost order can affect more or more selected units. All units that were given a specific order will lose the order or queued orders at the same time.

To Reproduce The issue does not appear to be reproducible in single player. It appears to be more frequent the more players are in a game. Mutli-region games may be more susceptible than single region games but I haven't been able to observe a strong correlation.

Expected behavior Not much to say here. I expect the order to continue after it has been given rather than be randomly canceled or lost.

Screenshots None.

Additional context Example replay: https://replay.faforever.com/18537213 In this replay from 41:18-41:25 all of the strategic bombers in the south bombers suddenly stop in mid air then go for then nearest landing position. As if they were given a stop order. It happens again from 41:38-41:42 and then again from 41:48-41:52. In the latter 2 you can tell that an order was issued and then canceled because the strategic bombers do a quick lift off and then land again. No cancel order was given. At this point the units were almost completely unresponsive. Part way through I swapped the attack order from attack ground to attack unit but the result was the same.

The submitted logs are from this replay.

Additional Information: As I recall the issue began around or less than a year ago after one of the network optimization patches. And I think it was one designed to help with desyncing issues. It did not occur before this patch as I have played FAF on and off for some years without the issue occurring.

My thoughts on what is going on: I've had a lot of time to observe the issue and what I think is going on is that An order is not being received in a timely manner (or at all) by one or more of the peer connected clients and the late client reverts the state of the order for all clients. Visible unit movement occurs but is then halted at some point. It is most visible with queued orders where the entire queue is lost. The issue can happen at any point during the game.

It will be difficult to reproduce this bug locally. The only thing I can suggest is more network logging. I also know it is being experienced by other players while others have never experienced the issue. It does not appear to be connection determinate due to these players having known good connections. I have checked for most networking issues including packet drop and ping spiking but this issue will occur when none of that is present (I used pingplotter to determine if I was having any network issues on my end). My connection is hard wired with no wireless hops to my isp.

Please let me know if I can provide more information.

ice-adapter.log irc.log game_18537213.log client.log

Edit: "does" not "dies"

[Garanas]

Actual situation:

Your intel:

Your intel when trying to bomb things:

What was your target, the disruptor?

[IceColdCoder]

First I tried ground firing in the location where both disruptors and 2 power generators meet at the corner. Then I tried ground firing one of the power generators, then I switched to targeting a unit and tried to bomb the disruptor. All orders were canceled. If I had done a misclick then my bombers would have flown off in some other random direction given that I was in an air fight at the same time but I had no reason to accidently press the stop key.

And to clarify: This is a pattern that has been going on for me and others for months with the same group of people complaining consistently while others are having no issue whatsoever. So I can say with confidence that this is an issue affecting some subset of the player base but not the whole thing.

[Garanas]

A reproduction of the issue:

https://user-images.githubusercontent.com/15778155/202398955-373f8614-e44b-4305-957b-80b802bbd25c.mp4

This issue is intel. And the reason some people never experience it and others always experience it is likely because some players scout, and some do not. Scouting clears blips that are attached to destroyed structures.

When a structure is removed the old blip remains. It has a darker color, indicating that it is destroyed. A blip is attached to a unit. When you hover over units, you issue the order on the blip. When the order is processed it tries to find the attached unit. For structures that are destroyed (or: gifted in this example) there is no structure attached to the old blip. As a consequence the order is cancelled.

The reason you see the order is because the UI attempts to feel more responsive by speculating that the order succeeds. But it doesn't, so it shows for less than 0.5 seconds and the order is 'lost'.

This is a long standing issue: the same situation occurs when a structure is destroyed, upgraded or (like in this case: gifted). The fix for the player is to scout more often (which your team did not do, as all the sams that were build in the last ten minutes were not visible). The solution as a developer is to:

• (1) Remove the invalid* blip
• (2) Update the blip, to the new (upgraded) structure
• (3) When you try to ground fire, ignore all units and blips

They all have advantages and disadvantages 😄 , and there has been no conclusion in the past. My personal preference is (1) and (3) combined - ground fire should always ignore units (you're trying to ground fire after all) and invalid blips should be cleared up as it introduces a lot of confusion, like this issue here.

[Garanas]

@Hdt80bro / @Tagada14 what are your thoughts on this?

[Tagada14]

If we were to remove the invalid blips it would make it much harder to see if the opponent upgrades a structure. While this weird mechanic of weird intel is questionable game design wise at this point it's pretty much ingrained into FAF and I would be cautious with removing that. Would the player just immediately see the new updated icon eg. T2 land HQ or how would it work? I definitely agree though that something needs to be done. This especially infuriating when a base is given to an ally and suddenly you can't target anything manually

[Garanas]

If we remove the invalid blip it would appear as if the unit 'is gone' - similar to how blips of mobile units are removed as soon as they are out of radar range.

[Tagada14]

That's not acceptable then. You would be able to make your base completely disappear by gifting it to an ally

[Garanas]

That would be called cheating though, and easy to spot.

[Tagada14]

Upgrading structures would do the same thing. Also when team mate dies in full share it will happen

[IceColdCoder]

My main concern here is that this isn't actually the issue I was attempt to refer to. I try to create another example. Because queued orders for building, reclaiming, assisting etc are also lost in my own base.

[Hdt80bro]

I agree that making updates to enemy strategic icons invalidate a player intelligence blips on them doesn't make sense outside of radar range, and has weird interactions with upgrading & gifting.

I actually have a head canon that the commander has a weak long-range radar built into it that military units (and military-grade civilian units) are invisible to because the enemy commander's quantum signature produces just enough untraceable noise (also, passive radio camouflaging in the armor plating or whatever) to render the long-range scanning a jumble of meaningless information (a more literal "fog of war"). Then once line of sight is established on a structure, it can correlate that information to its physical profile and actively track the distorting through time, but only if it is known that the unit will stay completely still. Thus, if the scanner sees a different instantaneous piece of quantum foam than it predicated, it knows that that structure changed somehow and can inform the player.

Anyway, the problem of invalid blips not having a unit under them to target for all players' simulations making the command auto-fail can be solved by making attack orders on blips a special kind of ground-fire that either dissolves when it is realized that there is no unit under it, or latches on to that unit. This is precisely what happens for jamming blips. I'm not quite sure why the same thing doesn't happen with un-updated blips. Since the bomber would always be able to discover it's a fake blip due to needing to fly right over it, and because they don't really need to use different targeting bones (we already fix the bomb trajectory for most bombers anways), there shouldn't be any behavioral issues with doing that.

Actually, there have been other reasons suggested for making bombers always ground-fire structures instead of targeting them (particularly when said bomber has a large AOE). This might just be the final straw :)

[Garanas]

Bombers that always ground fire introduces a lot of micromanaging 😄

My main concern here is that this isn't actually the issue I was attempt to refer to. I try to create another example. Because queued orders for building, reclaiming, assisting etc are also lost in my own base.

I'm looking forward to the examples that show what you are referring to. But in this particular case, I'm confident your bombers didn't move forward to attack because the blips were invalid.

Upgrading structures would do the same thing. Also, when teammate dies in full share it will happen

Yes, but they also cause this weird situation where you cannot retarget the structure. And that is bad game design / bad game experience

[Tagada14]

IMO it's the lesser of two evils, I would rather not be able to manually target something then have it disappear randomly from my "vision".

[Garanas]

I disagree. One is frustrating because it feels like the game is broken. The other tells you that it is time to scout again.

[Hdt80bro]

I'm not sure what you mean by the game would feel like its broken--I'm envisioning the behavior appearing the same, except for the broken bits being fixed. Can you elaborate?

[Garanas]

What we have now appears to 'work' until you interact with it. The order shows up, and is then removed. As if the order is 'cancelled' by the game. That feels like the game is broken.

[Tagada14]

Perhaps it may be annoying but it doesn't have a big impact on the game overall. Changing that behavior to what you want would actively break parts of the intel that people are used to for over a decade not to mention the possible abuse with sharing bases which btw would not be easy to ban. In a full share, a guy dies and the base is given to Yudi, Yudi doesn't want it so it's given to Farms who doesn't want it either and gives it to someone else. While this is going on enemy tries to attack but the base keeps disappearing since it's being given around. Do you ban those people just for using an inbuilt game mechanic of transferring units that they don't want? No, you can't. That is beside the point since it would make all upgraded mexes/factories disappear making it a nightmare and a giant game regression.

[Hdt80bro]

Oh, I'm just having a completely different conversation, that's what's going on

[relent0r]

Whats the con of making ground fire ignore units and blips? I hate when I try to ground fire and it snaps to the unit.

[Garanas]

I can't think of one

[Garanas]

Ground fire now no longer snaps to units or props, I'm closing this issue