Uveso's long-term bugtracing

[Uveso]

I am using this to track/link errors over longer time periods

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0213d267 #1803 0x023fde58 #1804 0x0047a820 #348 0x00598eec #1900 0x005f39f7 #1574 (Fixed.) 0x0067aa5f #318 0x006cbc63 #1375 (xrb0104) 0x008b94f0 #1559 0x00938ac8 #1117 0x0095854f #1445 0x009d7b75 #1467

GitHub gamebuild stability tests: 11.05.2017 (3676) part I #1969 result=perfect 12.05.2017 FAForever/deploy/faf (3677) #1971 result=stable 15.05.2017 FAForever/deploy/beta (3678) #1991 result=DEADLOOP 18.05.2017 master (3682) #2001 result=stable 29.05.2017 (3676) part II #2044 result=stable 05.06.2017 (3684) + PR AdvancedLeadTarget #2060 result=stable 06.06.2017 deploy/fafdevelop (3684) #2061 result=stable

Unknown/rare errors without error messages:

ERROR1 - count:4 last:06.Jun.2017 Freeze after transition screen while zooming to CDR. (Transition screen is half on the way to fade out. ~50% alpha)

ERROR2 - count:3 last:13.May.2018 Crash to desktop while initiating AI Archetype Last logline: info: AI DEBUG: ARMY \0004\000: Initiating Archetype using SorianMainTurtle Last logline: info: AI DEBUG: ARMY \0001\000: Initiating Archetype using UvesoMainAdaptive (Transition screen is half on the way to fade out. ~50% alpha)

ERROR3 - count:2 last:19.May.2018 Game Frozen after ACU kill. Last logline: warning: ACU kill detected. Rating for ranked games is now enforced.

ERROR4 - count:1 last:05.Jun.2017 Game frozen in midgame, sound and sim still running, but UI/gfx is frozen

Last Update 30.Dec.2019 (no new errors since 06 / 2017)

[Garanas]

Another one to add to the list:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8 attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15187368.log /gpgnet 127.0.0.1:34600 /mean 1333.36 /deviation 222.314 /savereplay gpgnet://127.0.0.1:52527/15187368/Utopian.SCFAreplay /country HU /numgames 28

Callstack: Unknown symbol (address 0x0050dfd8) Unknown symbol (address 0x0068af40) Unknown symbol (address 0x1d1ed9fb) Unknown symbol (address 0x1d1ed9fb)

See also: https://forum.faforever.com/topic/2332/questions-about-performance/18

game_15187368.log

[KionX]

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8 Attempt to read the internal parameter "BlueprintOrdinal" of the blueprint when destroying the unit. No more details can be obtained.

[KionX]

Here's more for the list: https://github.com/FAForever/fa/issues/3235 Does anyone want to fix this?

[Garanas]

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8 Attempt to read the internal parameter "BlueprintOrdinal" of the blueprint when destroying the unit. No more details can be obtained.

Where did you obtain this information from?

Here's more for the list: #3235 Does anyone want to fix this?

I don't know how to approach these type of errors - if you have suggestions

[KionX]

Where did you obtain this information from? I used a debugger. 0x0050dfd8 Address of the instruction where the AV occurred. In normal execution, the register points to the RUnitBlueprint class.(ClassNamePtr -> [[[Self]-4]+C]+8) I looked at the contents at the address and compared with Shift-F6. 0x0068af40 Address of the instruction in the call chain if Ctrl-k. I document almost everything I know about the engine here: https://github.com/FAForever/FA-Binary-Patches/blob/master/sections/include/moho.h https://github.com/FAForever/FA-Binary-Patches/blob/master/sections/include/global.h

[Garanas]

@KionX That is a good find - great work. Here's a few more for this issue:

---

Post: https://forum.faforever.com/topic/2488/constant-game-crashes-need-help-figuring-it-out/17 Replay: 15229180 Branch: FAF Log: 15229180.txt Similar to: (new)

Error provided by game:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x00438168
    attempted to read memory at 0x00000001

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15229180.log /gpgnet 127.0.0.1:20364 /mean 971.12964 /deviation 152.34543 /savereplay gpgnet://127.0.0.1:49336/15229180/Xarlus.SCFAreplay /country DE /clan NUT /numgames 59

Callstack:
    Unknown symbol (address 0x00438168)
    Unknown symbol (address 0x008026ac)
    Unknown symbol (address 0x00801e43)

Error provided by client:

java.lang.RuntimeException: Forged Alliance Crashed with exit code 1. See C:\ProgramData\FAForever\logs\game_15229180.log for more information
    at com.faforever.client.game.GameService.lambda$spawnTerminationListener$43(GameService.java:701)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)

---

Post: https://forum.faforever.com/topic/2488/constant-game-crashes-need-help-figuring-it-out/16 Replay: 15228945 Branch: FAF Log: game_15228945.log Similar to: (new)

Error provided by game:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0042323b
    attempted to write memory at 0x3c003c04

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15228945.log /gpgnet 127.0.0.1:50240 /mean 953.129 /deviation 153.206 /savereplay gpgnet://127.0.0.1:53083/15228945/Xarlus.SCFAreplay /country DE /clan NUT /numgames 58

Callstack:
    Unknown symbol (address 0x0042323b)
    Unknown symbol (address 0x00801e43)

Error provided by client:

java.lang.RuntimeException: Forged Alliance Crashed with exit code 1. See C:\ProgramData\FAForever\logs\game_15228945.log for more information
    at com.faforever.client.game.GameService.lambda$spawnTerminationListener$43(GameService.java:701)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)

Post: https://forum.faforever.com/topic/2488/constant-game-crashes-need-help-figuring-it-out/13 Replay: 15226123 Branch: FAF Log: game_15226123.log Similar to: https://github.com/FAForever/fa/issues/3162

Error provided by game:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x00423190
    attempted to read memory at 0x3e053ea4

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15226123.log /gpgnet 127.0.0.1:28745 /mean 953.129 /deviation 153.206 /savereplay gpgnet://127.0.0.1:58564/15226123/Xarlus.SCFAreplay /country DE /clan NUT /numgames 58

Callstack:
    Unknown symbol (address 0x00423190)
    Unknown symbol (address 0x00801e43)

Error provided by client:

java.lang.RuntimeException: Forged Alliance Crashed with exit code -1073740771. See C:\ProgramData\FAForever\logs\game_15226123.log for more information
    at com.faforever.client.game.GameService.lambda$spawnTerminationListener$43(GameService.java:701)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
    at java.base/java.lang.Thread.run(Thread.java:832)

[KionX]

I haven't studied them deeply, but all three are related to textures.

[Garanas]

Textures in what form? And could you record how you approach this? I'd like to learn about it :)

[KionX]

It looks like the textures were not created. Possible lack of video memory. The same method as described above. I can make a utility for this. If it will be useful.

[Garanas]

If you could - that would be great.

Another exception when playing co-op. The client doesn't always generate a proper dump file, hence - there is none for this particular exception.

[KionX]

Dump file is undesirable, they are uninformative and inconvenient. It seems that this is not a faf error. The replay did not show an error, there is no callstack addresses in the code. Perhaps it was a transition on a corrupted pointer.

[Garanas]

How do you make that conclusion?

And about the dump files - yes, they are inconvenient. But it is better to have one, then not have it at all.

[KionX]

The replay did not show an error, there is no callstack addresses in the code. Perhaps it was a transition on a corrupted pointer. A very strange error.

[Garanas]

He has it quite often - I'll ask him to keep them coming.

[KionX]

If someone plays on Linux, let them indicate this.

[Crotalus]

@KionX @Garanas While looking at the bug concerning OC and killing mobile shield units I stumbled upon the hard crash above at address 0x0050dfd8.

If you do targetEntity: GetBlueprint() when targetEntity is a "shield bubble" and not a unit the game crashes hard or sometimes just hanging the sim

OverchargeProjectile = Class() {
    OnImpact = function(self, targetType, targetEntity)
        -- This will crash the game if targetEntity is the shield bubble and not the actual unit that has the shield
        targetEntity:GetBlueprint()
    end
...
}

Would be good if this is fixed so the game doesn't crash at least. Maybe the engine function GetBlueprint() should just return nil in this case.

[Garanas]

post: https://forum.faforever.com/topic/2569/patch-3721-hotfix-3722-hotfix-3723/33 replay: 15362291 branch: FAF similar to: https://github.com/FAForever/fa/issues/2045#issuecomment-906426363

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8 attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15362291.log /gpgnet 127.0.0.1:18519 /mean 1157.78 /deviation 110.825 /savereplay gpgnet://127.0.0.1:61428/15362291/Gobblerpl.SCFAreplay /country PL /numgames 134

Callstack: Unknown symbol (address 0x0050dfd8) Unknown symbol (address 0x0068af40) Unknown symbol (address 0x3a20736f)

KionX already determined the cause of this - do we have an idea where / when this happens?

[Garanas]

post: https://forum.faforever.com/topic/2569/patch-3721-hotfix-3722-hotfix-3723/35 replay: 15366794 branch: FAF similar to: https://github.com/FAForever/fa/issues/2045#issuecomment-906426363

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8 attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15366794.log /gpgnet 127.0.0.1:2777 /mean 1500.0 /deviation 500.0 /savereplay gpgnet://127.0.0.1:63574/15366794/RomanGiant06.SCFAreplay /country US /numgames 0

Callstack: Unknown symbol (address 0x0050dfd8) Unknown symbol (address 0x0068af40) Unknown symbol (address 0x1c31c5fb)

[KionX]

15362291 https://github.com/FAForever/fa/blob/deploy/fafdevelop/lua/defaultcollisionbeams.lua#L196

15366794 https://github.com/FAForever/fa/blob/deploy/fafdevelop/lua/defaultcollisionbeams.lua#L586

Used this: https://github.com/FAForever/FA-Binary-Patches/blob/master/sections/include/global.h#L6

[Garanas]

I'll try and see if I can understand how to use that properly myself.

[Garanas]

I've looked at it for a bit and I do not understand how to get to the files in question. Could you record how you do it and share that with us through Zulip? Because I'd really like to understand how to analyze these hard-crashes.

[KionX]

It's not all simple there. Enough for a whole lecture. I have a better idea. I will make an automatic analyzer.

[Garanas]

I'm up for a lecture too 😄

But I'll take the automatic analyzer - that on its own would help a lot too.

[KionX]

Persuaded. I'll try to make an auto analyzer first. Then I will write a step-by-step example of manual crash analysis.

[Garanas]

You're a hero, can't wait to read up about it 😄

[Garanas]

And another one:

post: send private to me replay: 15370416 branch: FAF similar to: https://github.com/FAForever/fa/issues/2045#issuecomment-924858053

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8
    attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15370416.log /gpgnet 127.0.0.1:59226 /mean 979.938 /deviation 267.865 /savereplay gpgnet://127.0.0.1:54755/15370416/MAPster.SCFAreplay /country DE /numgames 4

Callstack:
    Unknown symbol (address 0x0050dfd8)
    Unknown symbol (address 0x0068af40)
    Unknown symbol (address 0x11a67dfb)

luaError.txt

[Garanas]

post: https://forum.faforever.com/topic/2586/game-crashing/3 replay: 15366391 branch: FAF similar to: https://github.com/FAForever/fa/issues/2045#issuecomment-924858053

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0050dfd8
attempted to read memory at 0x0000005c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15366391.log /replay C:\ProgramData\FAForever\cache\temp.scfareplay /replayid 15366391

Callstack:
Unknown symbol (address 0x0050dfd8)
Unknown symbol (address 0x0068af40)
Unknown symbol (address 0x1925d9fb)
Unknown symbol (address 0x1925d9fb)

[Garanas]

@KionX To fix this particular issue, can I assume that 0x0000005c is something like a null reference where this line refers to the fact that the C object of self has been destroyed? And that by adding an if statement checking for it, I can prevent these exceptions from happening?

[KionX]

Yes. 0x0000005c is something like a null reference. But I did not find out why it is null and whether it belongs to the unit that self points to. It says here that it can be nil: https://github.com/FAForever/fa/blob/deploy/fafdevelop/engine/Sim.lua#L260

[Garanas]

Maybe it can be nil when it is specifically nil. I think if you specifically give it nil it may be fine, but now it may try to retrieve something and fail doing so.

[Garanas]

Unhandled exception:

bad allocation

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15391838.log /gpgnet 127.0.0.1:52659 /mean 1076.82 /deviation 97.3515 /savereplay gpgnet://127.0.0.1:56356/15391838/UnitySenpai7.SCFAreplay /country US /numgames 116

Callstack:
    Unknown symbol (address 0x008d462c)

[KionX]

At 12:10 minutes, desync. The address refers to the initial initialization of the game. Trash.

[Garanas]

I don't understand? What is trash about it?

[Garanas]

Unhandled exception:

bad allocation

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15391838.log /gpgnet 127.0.0.1:52659 /mean 1076.82 /deviation 97.3515 /savereplay gpgnet://127.0.0.1:56356/15391838/UnitySenpai7.SCFAreplay /country US /numgames 116

Callstack:
    Unknown symbol (address 0x008d462c)

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0092a074
    attempted to read memory at 0xaa99214c

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15265598.log /gpgnet 127.0.0.1:59222 /mean 1155.1655 /deviation 95.35792 /savereplay gpgnet://127.0.0.1:53185/15265598/UnitySenpai7.SCFAreplay /country US /numgames 90

Callstack:
    Unknown symbol (address 0x0092a074)
    Unknown symbol (address 0x009144b0)
    Unknown symbol (address 0x0090d472)
    Unknown symbol (address 0x0090f5d0)
    Unknown symbol (address 0x166ffb7b)

These (and the previous post) are from the same person.

[KionX]

I don't understand? What is trash about it? This does not give any idea of the problem and is not reproducible.

[Garanas]

Ah, I see. Trash indeed 😞

[KionX]

15265598 The replay is strange(and the crash was not reproduced). It looks like the crash occurred in the general Lua function. I think this is a problem with third-party applications.

[Garanas]

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0067a559
    attempted to write memory at 0x00000204

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15412007.log /gpgnet 127.0.0.1:10722 /mean 1075.62 /deviation 97.4277 /savereplay gpgnet://127.0.0.1:51807/15412007/UnitySenpai7.SCFAreplay /country US /numgames 121

Callstack:
    Unknown symbol (address 0x0067a559)

[Garanas]

From: https://forum.faforever.com/topic/2638/game-crashes

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x00915450
    attempted to read memory at 0x00000006

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15270520.log /replay C:\ProgramData\FAForever\cache\temp.scfareplay /replayid 15270520

Callstack:
    Unknown symbol (address 0x00915450)

[KionX]

15412007 and 15270520 are not reproduced. Addresses lead to common functions.

These are their local errors.

[Garanas]

Do you have a suggestion what to tell them in order to fix it? Re-install the game?

[KionX]

Standard advices: reinstall game, reinstall Windows/Linux, update drivers, check for viruses, update Linux & wine, turn off antivirus, check memory, check processor, buy a new PC...

I don't have any original advice.

[Garanas]

I see - I'll keep that list in mind

[Garanas]

Deserialized nil table key.

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\replay.log /replay gpgnet://lobby.faforever.com/15452867/Jip.scfareplay /replayid 15452867

Callstack:
    Unknown symbol (address 0x006e4cd9)
    Unknown symbol (address 0x006e413c)
    Unknown symbol (address 0x1fe9be8f)
    Unknown symbol (address 0x62616e50)
    Unknown symbol (address 0x85105d86)

While watching a live replay - what?

At the end of the log:

info: WLD_GameSpeed -2
debug: Loading module '\000/units/xrl0403/xrl0403_script.lua\000'
debug: Current gametime: 00:27:30
debug: Loading module '\000/units/uaa0304/uaa0304_script.lua\000'
debug: Loading module '\000/units/xes0205/xes0205_script.lua\000'
debug: Loading module '\000/projectiles/caamissilenanite02/caamissilenanite02_script.lua\000'
debug: Current gametime: 00:28:00
debug: Loading module '\000/units/urb2305/urb2305_script.lua\000'
debug: Loading module '\000/units/ual0401/ual0401_script.lua\000'
debug: Loading module '\000/units/zeb9503/zeb9503_script.lua\000'
warning: Attempt to deserialize unknown lua data.

Game did last 30 minutes.

It finished when I tried again.

[KionX]

Deserialized nil table key. It didn't happen.

This happened: log_15452867.txt

[Garanas]

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x0095854f
    attempted to write memory at 0x02b80000

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15455963.log /gpgnet 127.0.0.1:xxxxx /mean 1500.0 /deviation 500.0 /savereplay gpgnet://127.0.0.1:xxxxx/15455963/nomads.SCFAreplay /country XX /numgames 0

Callstack:
    Unknown symbol (address 0x0095854f)

No replay appears to be available.

[KionX]

Similar #1445 (possibility: 32Bit memory full)

[Garanas]

Another crash, happens at every game

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at address 0x025ede58
attempted to read memory at 0x5f5ef079

Program : C:\ProgramData\FAForever\bin\ForgedAlliance.exe
Cmd line arguments : /init init.lua /nobugreport /log C:\ProgramData\FAForever\logs\game_15468792.log /gpgnet 127.0.0.1:54992 /mean 1500.0 /deviation 500.0 /savereplay gpgnet://127.0.0.1:59961/15468792/vmkcom.SCFAreplay /country KZ /numgames 0

Callstack:
Unknown symbol (address 0x025ede58)
Unknown symbol (address 0x025de001)
BaseThreadInitThunk + 20 bytes ((Unknown)(0) + 0 bytes)
RtlGetAppContainerNamedObjectPath + 281 bytes ((Unknown)(0) + 0 bytes)
RtlGetAppContainerNamedObjectPath + 233 bytes ((Unknown)(0) + 0 bytes)

[Garanas]

Issue with the tool that you've made is that I never get hard crashes when I test 😢

[KionX]

The replay did not show an error, there is no callstack addresses in the code.