ReorderClusters removes entity if it instantaneously moves further than a cluster size (20km)

[Gwindalmir]

With a limited world size, if I change the position in ModAPI scripting on an entity to near the border more than 20km (but still 10+km away from border), then havok triggers the MyPhysics.HavokWorld_EntityLeftWorld event and deletes the entity.

This causes the entity to disappear, and the player left floating, however there's no HUD warning that you are near the border. You can still move away from the center much further before reaching the real end.

I've attached the world and my mod, which trigger it. World: https://www.dropbox.com/s/i7t96frax2a6obt/Debug%20Test.zip?dl=0 Mod: https://www.dropbox.com/s/ka02dgxr8isluph/FTL.zip?dl=0

To reproduce, simply load the world with the mod, enter the cockpit, and press 1. Once you jump, you can continue moving for another 6km before reaching the red HUD warning. I didn't have GPS markers in the attached world, but if you add them you'll see.

EDIT: This appears to be only reproducible on limited worlds larger than 50km. I tried 50km and smaller with no issues. It appears related to switching between clusters. It seems if I try to move to a location not directly connected to the cluster I came from, it thinks I left the world. I tested by increasing the cluster size from 20k to 50k, and I can move further before it happens.

EDIT2: If I spawn a physics entity every 19k between the source and the destination the movement is successful, which confirms my earlier suspicion. This only seems to work once. The next movement it fails again.

I only have a small understanding of the cluster code (like none), but from what I do understand, here's what I see happening:

1. The ReorderClusters() method handles cases where objects are moving between clusters.
2. This code assumes an object leaving the cluster is also entering a nearby one (ContainmentType.Contains). I think it's more dynamic than that, but that's how I could explain it.
3. It creates a new cluster, large enough to contain all the objects of the two split clusters ('from' cluster, and 'to' cluster).
4. It then checks if this is too large, if so, it splits it. This part I'm not sure about how it happens.
5. The moving object gets moved to a new cluster, with nearby objects of where it's headed.

Trouble I'm hitting is that the object moves to a cluster too far away, forcing a split. However the split itself cannot create a cluster which contains the object that needs to move. The code then sees the object isn't inside the new cluster (ContainmentType.Disjoint) and skips it. Except it gets removed from the old cluster, but doesn't have a new one to be in, hence removal.

KSWH Forum topic: http://forum.keenswh.com/threads/01-083-reorderclusters-removes-entity-if-it-instantaneously-moves-more-than-a-cluster-size-20km.7360507

[joemorin73]

With today's potential addition of Jump Drives. Has this issue been corrected in the latest merge?

[Gwindalmir]

No, this is still reproducible, and with the built-in version now as well. See https://youtu.be/zjqtxBA7EdE Updated world: [https://www.dropbox.com/s/cbvkju3w7oa34is/Debug Test_stock.zip?dl=0](https://www.dropbox.com/s/cbvkju3w7oa34is/Debug Test_stock.zip?dl=0)

I think I better understand what's happening: 1) Object moves to a point where the cluster size is now large enough to split >40km from current cluster center 2) A new cluster is created where the object will end up, with an initial size of 20km on edge 3) All the current objects inside both clusters (intersection of old and new), are tracked. This includes objects not triggering this reorder. 4) The above lists are now added to the union of the new cluster 5) Then the original cluster is split into two, and deleted.

The problem happens in step 3. If the new cluster created in step 1 is more than 20km from the original cluster, there is no intersection between the two (empty set). This means the objects are not moved to this new cluster, and are removed when the original pre-split cluster is removed.

MyClusterTree.cs, Line 655: System.Diagnostics.Debug.Assert(aabb0.Max.GetDim(longestAxis) - minimumCluster.Min.GetDim(longestAxis) > 0, "Invalid minimal cluster");

The length is over 20000. I know the assert does nothing, I'm just pointing out that I think the code is also assuming it's less than 20000.

[joemorin73]

To understand,

Because the grid does not travel to it's destination, but rather appears at it's destination, the system can't split a cluster as the distance is too far?

Would this mean you could jump 100km if cluster was present at the destination? (IE, someone was in the area.)

I haven't looked into the code, but I imagine the fix would require to create a new cluster if there is no cluster in the area or the cluster is too far? This does present problems as cluster splitting is one thing, but creating a cluster and verifying anything in the area is added to the new cluster.

[Gwindalmir]

That's how I understand it. Creating a new cluster is probably the correct fix, or add to an existing overlapping cluster, then possibly split that.

[joemorin73]

Here's hoping it gets in for next Thursday.

[Gwindalmir]

This is still a bug in 01_092_007, contrary to release notes.

[joemorin73]

We need a Github merge to see what they did to address it.

[zrisher]

I was really excited about this being fixed. : (

[devu]

Under what conditions is not working for you? I tested it on 300km limit world and was able to jump to GPS over 100km just fine.

On 24 July 2015 at 00:37, zrisher notifications@github.com wrote:

I was really excited about this being fixed. : (

— Reply to this email directly or view it on GitHub https://github.com/KeenSoftwareHouse/SpaceEngineers/issues/158#issuecomment-124269321 .

[Gwindalmir]

The world provided in the OP shows the problem.

[joemorin73]

Pheonix, you're testing this with your FTL mod as opposed to the Jump Drive. Is your mod still the original code? or is it now overriding the new JumpDrive.

I ask this because I am wondering if the fix is in the JumpDriveSystem and not in the MyCluster.

[Gwindalmir]

Ugh, it does appear they fixed in the jump drive system. I can't reproduce it with the vanilla jump drive now. That's not going to work. Any mod that changes the entity position is still affected by this.

Related to this, I found simply disabling the havok entity removed event handler isn't good enough. Starting about a week ago, on an unlimited world now (which doesn't use this event), the physics is detached, but the visual of the grid remains. This means when you exit the cockpit, you can pass through the ship and can't interact with any terminals. Now using an unlimited world is no longer a workaround.

I wish they'd update github, so I can see what they did.

[joemorin73]

I think there is a summer slowdown. (Vacations and etc.) We may be waiting until next week.

[joemorin73]

Github updated. Time to see the fix.

[Gwindalmir]

It's in Sources/Sandbox.Game/Game/GameSystems/MyGridJumpDriveSystem.cs line 624. See the commit I referenced for the code. Unfortunately I don't think I can do that in my mod.

[joemorin73]

SO they address the issue in the jumpdrive instead of in the world movement?

[Gwindalmir]

Yes, which is exactly what I suspected after the last update. It's possible they are testing the fix in the jump drive before rolling it out wider.

EDIT: I took the github source, applied a custom whitelist, and called that method in my mod, it didn't work. In fact it tripped a debug assertion at MyClusterTree.cs:326. Sadly it wasn't as simple as using their fix (unless I did it wrong).

[zrisher]

Ever since the 7/24 patch that "fixed" jump drive in its class instead of in the entity cluster logic, we've had issues (on our limited-size world) where ships will randomly disappear with no seeming common setup. They can be ships, stations, moving, not moving, piloted, completely empty, being welded, completely untouched, etc. I have a hunch that the common issue is in fact that someone else is using jump drive at the time, because:

• the disappearance (immediately, no messages, kicks you out of cockpit) is so similar to what we saw with broken jump drive
• no one seems to be experiencing this except on high population limited-size worlds
• everyone who's experienced this recently has confirmed they believed someone else was using jump drive at the time (but obviously this is hard to correlate on a survival PVP server)

@Phoenix84 do you think this might make sense given the existing issues with MyClusterTree? I haven't had a chance to look into the code in depth here because there are some new concepts I'm unfamiliar with, but I will get to it soon. Just curious what your thoughts are.

In the meantime, I've disabled the jump drive block in the hopes that will help.

[Gwindalmir]

@zrisher With the way the cluster code works, it can affect any and all objects, even ones not performing the jump. I'm not sure if the fix itself runs on the client, server, or both. If it only runs on on side that triggered the jump, that might explain the disappearance, as the other clients won't get the hint to be careful with the next cluster operation.

Plus, I think this fix is really a test or workaround. Which is why the put it in the jump code instead of the cluster code. If it works, they move it to the cluster code, if not, they figure something else out. It reduces the likelihood for regression, so I understand. Sounds like it's not working though.

As for disabling the jump drive, that should work to fix it, as long as you don't have a high speed limit mod (like several km/s at least). The jump drive issue doesn't cause the problem, it's a symptom.

[zrisher]

That's exactly what I wanted to hear. Thank you so much for the insight!

[midspace]

zrisher said:

Ever since the 7/24 patch that "fixed" jump drive in its class instead of in the entity cluster logic, we've had issues (on our limited-size world) where ships will randomly disappear with no seeming common setup.

Not necessarily. There was an issue I saw a few months ago before Jump drives came long, where a ship would mysteriously move somewhere else. I had it happen on a dedicated test server that I have no admin access to, and I was by myself at the time. It was small ship, with power and dampeners turned on so it didn't drift away if knocked, and ownership set to me. One moment it was there when I went past, perhaps 20 minutes later I noticed it was missing. It certainly wasn't knocked, or hit by anything. No debis. I had to save the game locally to use SEToolbox to find it had moved 100Km away. After flying there in another ship, the ship was still intact and motionless, and still under the correct ownership. There was no explanation as to why it suddenly moved 100Km away.

[zrisher]

In our case, the ships are all actually disappearing. We have a mod that allows players to see the positions of all their owned grids - the ships we've seen disappear are completely and instantly removed from the game.

FYI I disabled jump drives via Enabled="False" on the block definition last night, and I haven't had a single report of the above error since. I was getting about 5 a day before that. So I do believe this is related.

[Gwindalmir]

Since this is a problem in the cluster code (which was added months before), what midspace is saying is it's possible you will still see this bug randomly. However with the jump drive disabled, the conditions to trigger that problem are much less likely to occur. I agree, you might still hit this randomly, but maybe 1 in 10000, as opposed to 1 in 10.

[zrisher]

Sorry I forgot to update this thread - it turned out that even with jump drives off, the disappearance would regularly occur. So basically there is no way to run a Limited-Size world without ships randomly disappearing.

We removed our size limit and have had zero reports of disappearances since. Even after removing jump drives, before I was getting at least 2 a day.

[jpcsupplies]

I can concur on this - I always run my servers on unlimited, as we regularly manually clean up debris on our server; and any missing ship always turn up elsewhere; although the comment above explains how some got 100's of kms away without a player touching it. i've not had a ship simply vanish on me. Exploration mode structures on the other hand.. i've seen these spawn/despawn with no obvious pattern, other than being located where no player has been recently. My own players report ships on other limited servers simply vanishing; one server got so bad all the players moved to mine which is where i found out about it - so there seems to be a major issue with limited worlds.

[Gwindalmir]

This was fixed in the stock jump drive some time ago by calling Sandbox.Engine.Physics.MyPhysics.Clusters.EnsureClusterSpace(). Is there any chance, @OndrejPetrzilka or @LeonidasCZ, that you will be able to make this available to modders? Right now mods still cannot teleport entities very far without hitting the cluster limitation and disappearing.