Stuck cims

[originalfoo]

Got a train station that regularly gets hoards of stuck cims:

Reset stuck cims (via mod options > maintenance tab) sets them free:

Would be nice if we could work out what's causing it.

Will add savegame and logs later.

[originalfoo]

Save: https://steamcommunity.com/sharedfiles/filedetails/?id=2056376792

[ninjamods]

Savegame: https://steamcommunity.com/sharedfiles/filedetails/?id=2057580865

Error: System.NullReferenceException: Object reference not set to an instance of an object at TrafficManager.Manager.Impl.ExtCitizenInstanceManager.StartPathFind (uint16,CitizenInstance+,TrafficManager.API.Traffic.Data.ExtCitizenInstance+,TrafficManager.API.Traffic.Data.ExtCitizen+,UnityEngine.Vector3,UnityEngine.Vector3,VehicleInfo,bool,bool) <0x00c16> at TrafficManager.Custom.AI.CustomCitizenAI.CustomStartPathFind (uint16,CitizenInstance+,UnityEngine.Vector3,UnityEngine.Vector3,VehicleInfo,bool,bool) <0x00146> at ResidentAI.StartPathFind (uint16,CitizenInstance+) <0x007d5> at ResidentAI.SetTarget (uint16,CitizenInstance+,uint16,bool) <0x0096c> at (wrapper dynamic-method) HumanAI.ArriveAtDestination_Patch1 (object,uint16,CitizenInstance+,bool) <0x0020e> at (wrapper dynamic-method) HumanAI.ArriveAtTarget_Patch1 (object,uint16,CitizenInstance+) <0x000b3> at (wrapper dynamic-method) HumanAI.SimulationStep_Patch0 (object,uint16,CitizenInstance+,CitizenInstance/Frame+,bool) <0x02013> at (wrapper dynamic-method) ResidentAI.SimulationStep_Patch1 (object,uint16,CitizenInstance+,CitizenInstance/Frame+,bool) <0x0032a> at CitizenAI.SimulationStep (uint16,CitizenInstance+,UnityEngine.Vector3) <0x001f9> at TrafficManager.Custom.AI.CustomHumanAI.CustomSimulationStep (uint16,CitizenInstance+,UnityEngine.Vector3) <0x00649> at (wrapper dynamic-method) CitizenManager.SimulationStepImpl_Patch0 (object,int) <0x0047b> at SimulationManagerBase`2.SimulationStep (int) <0x0004c> at CitizenManager.ISimulationManager.SimulationStep (int) <0x00024> at SimulationManager.SimulationStep () <0x00693> at SimulationManager.SimulationThread () <0x0018a>

TMPE.log output_log.txt

[originalfoo]

@ninjamods That error isn't in the logs you attached - run the game again until error occurs and attach the log files at that point

[krzychu124]

Hi @ninjamods your logs are "empty". It looks like you attached logs after you shared savegame. You should load savegame, wait/provoke error then close game and collect logs. Log files are always cleaned every time you start the game :/ We will check savegame anyways ;)

Edit: I forgot to click Add comment lol Edit2: Loading savegame might be painfull [>10k assets]

[originalfoo]

Note to self: Check title for typos when creating issues lol

[originalfoo]

Here's my logs btw, for save linked in https://github.com/CitiesSkylinesMods/TMPE/issues/825#issuecomment-612318559:

output_log.txt TMPE.log

Notably they were all waiting for path, but it's only tiny population (about 5k cims in the city) and there was no buildup of PFs that I could see:

Info 564.5636497: UtilityManager.RemoveStuckEntities() called.
Info 564.5655368: UtilityManager.RemoveStuckEntities(): Waiting for all paths.
Info 564.5663157: UtilityManager.RemoveStuckEntities(): Resetting citizen instances that are waiting for a path.
Info 564.5668581: Resetting stuck citizen instance 20 (waiting for path)
Info 564.5674219: Resetting stuck citizen instance 2017 (waiting for path)
Info 564.5679532: Resetting stuck citizen instance 2101 (waiting for path)
Info 564.5684711: Resetting stuck citizen instance 2170 (waiting for path)
Info 564.5689951: Resetting stuck citizen instance 2225 (waiting for path)
etc...

Also, I've tried at least three different train stations in that spot and all had the same issue so it's unlikely to be asset-caused (unless they were all made from same template and that template has some issue?).

[ninjamods]

Reuploaded! This error happens at start for this game every time, so I'm pretty sure it's the same one.

TMPE.log output_log.txt

[ninjamods]

TMPE.log output_log.txt

More detailed logs attached!

[originalfoo]

Narrowed down to this block of code:

            Citizen[] citizensBuffer = Singleton<CitizenManager>.instance.m_citizens.m_buffer;
            ushort parkedVehicleId = citizensBuffer[instanceData.m_citizen].m_parkedVehicle;
            ushort homeId = citizensBuffer[instanceData.m_citizen].m_homeBuilding;
            CarUsagePolicy carUsageMode = CarUsagePolicy.Allowed;
            var startsAtOutsideConnection = false;
            ParkingAI parkingAiConf = GlobalConfig.Instance.ParkingAI;

[krzychu124]

@aubergine10 instanceData can be null there (missing asset?). Look at log:

Info 676.6934940: Loading Data from New Load Routine! Length=4161932
Info 694.3635829: Loading State from Config
Info 694.3647886: Loading 46834 extended citizens
Info 694.4278915: Loading 57528 extended citizen instances

[krzychu124]

Ok, it looks like incorrectly loaded building(missing subscription/removed from workshop)?

Building info is always set when asset is correctly loaded. @aubergine10 I think we can add nullcheck there and log that situation. There is no info about that asset used by this instance of building because info was not set, obviously

[originalfoo]

@krzychu124 What should happen if null though? return false or Reset(ref extInstance)?

[krzychu124]

Good question :D I'll check what is going on there, maybe we could set path as invalid (soft/hard).

[krzychu124]

I tested your savegame @aubergine10 and I noticed similar behaviour when I was debugging broken nodes (AKA Luke, vehicles are despawning). This time m_path of citizen is constantly changing unlike m_path of vehicle like before. Have you tried broken nodes detector?

It's weird because it seems that path can be found (pf success counter rapidly goes up -> x thousands per minute, it's not normal, btw) but it is invalidated for some reason before citizen start moving.

[ninjamods]

Interesting, do I use this: https://steamcommunity.com/sharedfiles/filedetails/?id=1777173984&searchtext=broken+node to test for broken nodes?

[krzychu124]

Yeah that one, but your error was already fixed in Labs and that error seem to be completely unrelated to "stuck cims" issue ;) I'll update Stable today.

[originalfoo]

I've run Broken Nodes detector multiple times, it found nothing. Also ran the broken PT checker and ghost nodes checker to no avail.

Is there somewhere we can determine why the paths are being invalidated?

[krzychu124]

It will be pita but I have to follow call stack :/

[originalfoo]

Does BND scan only transit networks? Could it be some other type of network causing the issue (quays, etc)? Since SH quays can now have ped paths.

[originalfoo]

So, thanks to Krzychu's investigations, it turns out my stuck cims problem is most likely the result of an ancient Viking settlement under my train station

Image from krzychu:

Looks like Move It, Anarchy or some other mod (or asset) is leaving building intestines scattered around the map.

[originalfoo]

fuuukkk....

Like, there's an entire era of human history buried in them thar hills...

[originalfoo]

When zooming in to the overlay, a lot of the text moves over actual (wanted/proper) segments/nodes.

Part of the issue debugging this is the overlay is just splatting stuff everywhere. I think it might be easier to debug if we had better tooling; will create new issue for that. EDIT: #844

[TesseractToby]

@aubergine10 - I can add another save game to the list with this same issue (I believe). My PFs is up to 1.5k and rising, grinding everything to a halt, but I've not been able to find anywhere like your train station which may be causing this specific issue. Included is my output_log and tmpe logs following a search for broken nodes/lines and a reset of stuck sims (which was performed after saving the game, I've left it with the issue growing for review).

Tesseract_TMPE.log Tesseract_output_log.txt

Save game can be found at: https://steamcommunity.com/sharedfiles/filedetails/?id=2066275717

Edit I found the transport building having the issues. In the save attached, it's the Intercity Bus Station at Elm Hills district. Disabling that building causes the PFs to drop to normal levels.

[krzychu124]

@TesseractToby it seems that vanilla also can't handle that situation properly, so it might be game bug 😕 @aubergine10 I've noticed something weird. Why almost all stuck cims have dog??

[originalfoo]

I think dogs just stand out from the crowd so we notice them more.

I've pretty much confirmed the cause of the ancient viking villages... It's due to Move It mod being used on any station that has underground paths/networks.

I'm not sure if it's direct result of moving the building, or side-effet - eg. when building is bulldozed, the bulldozer might not be recognising the viking ruins as belonging to the building.

[krzychu124]

I think that pedestrian path nodes have to be at specific distance to PF sees them as connected. Maybe you moved only visible nodes and segments, but those invisible are still at original position and PF followed internal path without checking connections?

[originalfoo]

Maybe you moved only visible nodes and segments, but those invisible are still at original position and PF followed internal path without checking connections?

Specifically it was the station building I moved. It seems Move It mod isn't properly moving some paths that are internal (or otherwise part of) that station. I'm not sure if the station has sub-buildings that Move It forgets to move or something like that?

I used @kianzarrin network detective mod and all of the problematic segments are ped paths.

I suspect a side-effect of this issue, particularly for detailers, is consumption of the node/segment/lane limits, because they'll have vast numbers of these "viking ruins" lurking under their city considering how much they use Move It mod.

I'm still not quite sure if the Move It bug is limited to specific vs. all train stations. Maybe just those with certain kinds of internal path? Maybe it's something to do with sub-buildings / or path-containing props? While I've only noticed it with train stations, it could potentially be any kind of building containing internal paths.

I think that pedestrian path nodes have to be at specific distance to PF sees them as connected.

For ped paths, this is certainly the case. If within certain proximity to another path/node they seem to auto-connect somehow.

I'm not sure how it would break pathfinder though - if it thinks they're valid it will include them in the path, cims should still walk along them. Maybe it's the AI that's getting confused? Maybe there is some special case for Pedestrian Connection Inside segments?

I'm not sure how we could ever auto-detect the problem segments. While they have some obvious characteristics, there doesn't seem to be any reliable way to determine if they are associated with a building or not. Looking at the structs in ILSpy, I don't see any obvious reference to a 'parent building' (maybe there is one I didn't spot?) - my guess is that once plopped, the game doesn't really care. Although if that were the case, how would bulldozing the building clear up the internal paths? There must be some sort of linkage somewhere? If there is a linkage, maybe that's what's confusing the pathfinder/AI?

[originalfoo]

Just noticed another situation, this time with "internal roads" of a ferry harbor:

Note the road has two sets of nodes/segments:

I replaced the original road (part of the harbor building) with a different road using Klyte45's Touch This mod. It seems that the nodes/segments of the original road are retained, so it's like two roads on top of each other.

Using Alt+Click/Drag (Move It mod) to select the node of the replacement road and drag it:

As you can see, there is a 'ghost segment' that I assume in the original road. Somehow the road mesh isn't showing but the segment is still there.

I suspect this too will cause issues for pathfinder/AI.

[krzychu124]

This issue: https://github.com/CitiesSkylinesMods/TMPE/issues/825#issuecomment-616860983 is caused by ghost nodes/segments there. Probably something happened and intercity bus station was not moved correctly leaving some nods and segments confusing path finder. Basically, path finder created invalid path -> it should not connect disabled to not disabled node if they are not asset internal.

Similar issue (stuck cims) may happen on very wide roads (>64f). It's vanilla code.

Cim will request new path, he will get new one, but because of string max distance value connection not longer than 64f (PF doesn't know about it) is invalidated by cim. This will create infinite loop.

For cyclists situation is a little bit better, because they also invalidate path but released bike is releasing also citizen(cim is not really cycling but bike is used by cim -> citizenAI is superior AI over bikeAI, unlike for other vehicleAIs where vehicle is managing citizen units inside)

I've increased that limit and didn't spot any issues, cims and cyclist can use very wide roads. No despawn or stuck occurred at a time of tests

[krzychu124]

I've been debugging user savegame and I noticed that the user have built metro stop using block metro station.

Cims stuck on one side of segment and block metro stop looked like reachable from the other end of that segment. When I removed all metro line stops, cims immediately unstuck and found different routes, which may suggest that block metro station had broken invisible pedestrian connection - it's pretty easy to break it if you select wrong object with Move It (e.g. tracks alone instead of whole station building).

Invisible connections have limited range of automated connection and once moved too far the connection breaks and since we can't see them, users are completely unaware of the problem.

There must be another hidden variable in cim simulation code which invalidates "valid" paths returned from Pathfinding besides known pedestrian pavement lane distance limit (Node Controller corner adjustment issue) or it's just asset/Move It move building issue.

Worth mentioning is I recently created a little tool that is monitoring correctness of cross thread calls - it checks if vanilla method calls are performed from simulation thread (~120 most used methods are monitored) and notifies the user in case of incorrect calls - returns stack trace, usually very easy to find the source, since in 95% cases cross thread calls are triggered by the user because of bugs in mods UI. With help of that tool I found ~5yo bugs in AVO that could cause rendering invisible vehicles blocking the traffic, some minor MoveIt bugs or CitizenUnits leaks also in AVO - game was returning no longer used CitizenUnit to pool but cleared the unit only partially - some properties was still holding references so game couldn't use them again which at some point in the future could kill the user city -> cims couldn't move in/move/work/'study'/visit places etc.

I haven't had enough free time to find out how to detect broken pedestrian connections but I'm pretty sure that will be the main source of all issues that are different than those caused by the intersection corner adjustments.

[originalfoo]

I think issue with Move It is that it has no way to know what untouchable networks are associated with the building/station being moved (I could be wrong). Or maybe the asset defines them wrong and somehow they become detached from building?

EDIT: It might be issue that paths aren't inside the building bounds box, they are below the building. If move it is grabbing paths based on the 3D bounds box of the building, it would miss those paths underneath it?

The viking village issue I had earlier in this thread is easily reproducible with glass narrow stations (or this old deprecated version - whenever you move those buildings with Move It, they leave behind all their untouchable Pedestrian Connection Inside paths = viking ruins = zombie cims.

Another cause of cims getting stuck is mods that set wait timer (boredom timer) to Zero, as noted by the author of KamineNoDespawn mod. Klyte's Anxiety Reducer mod could temporarily exhibit similar issue, however it does still allow wait timer to increase (just more slowly) so "stuck" effect will only be temporary.

[originalfoo]

BTW, when I was experimenting in-game, I used TM:PE lanes/segments/nodes overlays to work out the id's of the lanes/etc then used Kian's Network Detective mod which has a "search" feature - enter node/lane/segment id and it will highlight it on map and show info about it. I was also able to use Network Detective to then delete the viking paths, and saw that my cims started moving around normally again after the broken paths were removed.

[kianzarrin]

issue with Move It is that it has no way to know what untouchable networks are associated with the building/station being moved

to select building networks using move it, you need to hold alt.

[originalfoo]

to select building networks using move it, you need to hold alt.

Yeah, but that only works for "normal" networks like train tracks, roads, etc. It still doesn't allow select of Pedestrian Connection Inside.

---

Just found a new breed of viking ruin under the ocean in my city - this time some sort of phantom train track ending at precisely 0,0 on the map:

Earlier at that same location there were about 2000 outdoor BBQ stoves stacked on top of each other (luckily it was easy to delete those with Move It).

The other end of that track is the middle of station track in a train station many miles away:

@krzychu124 I wonder if the PathFailed | Untouchable flag mask could help with identifying potential issue networks?

[originalfoo]

Something lurks in the deep ocean:

Something that highlights all Untouchable nodes/segments/lanes would be useful just to discover how much of that stuff is lurking on the map.

[krzychu124]

Something lurks in the deep ocean:

Something that highlights all Untouchable nodes/segments/lanes would be useful just to discover how much of that stuff is lurking on the map.

oh you found Pos. 0,0,0 HELL 🤣 Data corruption -> modifying networks from wrong thread may corrupt node/segment grid which is used to help determining if object should be rendered or not. For vehicles game is rendering everything that is attached to vehicle grid, no matter what flags it has (flags cannot be 0 - None). Basically when network is incompletely released it will be still assigned to grid (very similar problem to mentioned corruption due to cross-thread bugs with AVO) -> game resets all positions and most of properties but leaves grid assignment (that requires looping which is the source of race condition bugs) and at least one of the flags (usually NameVisible or NameVisible2) so objects are still valid for rendering process -> rendered in the center of the map xyz (0,0,0)

Yeah, but that only works for "normal" networks like train tracks, roads, etc. It still doesn't allow select of Pedestrian Connection Inside.

What about moving building but breaking invisible connections because of attaching/detaching with nearby lanes?

[originalfoo]

oh you found Pos. 0,0,0 HELL

Yup, and there's a bunch of cims + some dogs walking on the spot but not going anywhere on the seabed down there lol.

What about moving building but breaking invisible connections because of attaching/detaching with nearby lanes?

That might happen too, but in my testing I could place the station in middle of nowhere (zero other networks or buildings) then use Move It and the underground Pedestrian Connection Inside segments became detacthed.

[krzychu124]

That might happen too, but in my testing I could place the station in middle of nowhere (zero other networks or buildings) then use Move It and the underground Pedestrian Connection Inside segments became detached.

So maybe it's Move It issue/bug? @Quboid any ideas?

[Quboid]

There is an issue with internal pedestrian paths becoming detached in Move It. As far as I know, this is unavoidable due to how pedestrian paths pick where they connect to seemingly at random.