Drills cause massive sim speed slowdown

[Jake-Rich]

Running 3770k CPU 4.5Ghz and 8GB 1600Mhz Ram, AMD R9 290 GPU.

Tested on both the current steam version (1.91) and the current source code compiled into 64bit mode. Using one drill or more causes the sim speed to go down from 1.00 to 0.50. CPU usage climbs very quickly too. (No drills = 13%, one drill = 44%, two drills = 70%, any more = 99%).

CPU usage climbs -> Sim speed drops -> FPS drops

[mccorkle]

I'm trying to write up a list of use cases that generate lag in SE. Can you help me with some more details on your world setup?

For your world how many floating objects do you have allowed? Does it just kill simspeed when you engage the drills in empty space, or when actually drilling a rock? What is your view distance world setting? Which render engine (dx9 or dx11)? Are you hosting a multiplayer, of just in single player / offline?

Could you also post a screenshot with the debug data (shift-f11 and ctrl-h)? I'd love to match it up to what I'm seeing.

[IhanaMies]

It is caused by this code snippet in MyDrillBase since it gets called every frame by every drill. Disabling it allows very large mining ships to drill without causing any major fps or simulation speed drops. Though disabling it stops drilling "smoke effect".

Line 445-457

foreach (var entry in m_drilledMaterialBuffer)
            {
                somethingDrilled = (!collectOre || TryHarvestOreMaterial(entry.Key, hitPosition, entry.Value, onlyCheck)) || somethingDrilled;

                if (somethingDrilled && !onlyCheck)
                {
                    MyDebris.Static.CreateDirectedDebris(hitPosition,
                            MyUtils.GetRandomVector3Normalized(), 0.1f, 1, 0, MathHelper.Pi, 5, 1, 0.15f, entry.Key);
                }
            }

Getting it called by UpdateAfterSimulation10() would probably fix most of the lag and simulation speed drops and still produces the smoke effect.

[joemorin73]

The "smoke effect" being the debris from what I see. Anyone trying this idea?

[IhanaMies]

I did get some results with a simple +=1 counter when MyDebris.Static.CreateDirectedDebris and CreateParticles in line 343 gets called. I'm experimenting with different counter resetting numbers to see if I could find a good normal for them.

MyDebris.Static.CreateDirectedDebris works like this:

        public void CreateDirectedDebris(Vector3 sourceWorldPosition,
                                         Vector3 offsetDirection,
                                         float minSourceDistance,
                                         float maxSourceDistance,
                                         float minDeviationAngle,
                                         float maxDeviationAngle,
                                         int debrisPieces,
                                         float initialSpeed,
                                         float scale,
                                         MyVoxelMaterialDefinition material)
        {
            ProfilerShort.Begin("Create directed debris");
            MyDebug.AssertDebug(debrisPieces > 0);
            for (int i = 0; i < debrisPieces; ++i)
            {
                var newObj = CreateVoxelDebris();
                if (newObj == null)
                {
                    break; // no point in continuing
                }

                float dist = MyUtils.GetRandomFloat(minSourceDistance, maxSourceDistance);
                float angleX = MyUtils.GetRandomFloat(minDeviationAngle, maxDeviationAngle);
                float angleY = MyUtils.GetRandomFloat(minDeviationAngle, maxDeviationAngle);
                var rotation = Matrix.CreateRotationX(angleX) * Matrix.CreateRotationY(angleY);
                var deviatedDir = Vector3.Transform(offsetDirection, rotation);
                var startPos = sourceWorldPosition + deviatedDir * dist;
                var initialVelocity = deviatedDir * initialSpeed;
                (newObj.Debris as MyDebrisVoxel.MyDebrisVoxelLogic).Start(startPos, initialVelocity, scale, material);
            }
            ProfilerShort.End();
        }

Volume or mass are not defined here. Does this mean these are only visual rock pieces flying off and getting deleted after some time?

[IhanaMies]

With the changes I've made I could drill constantly with a 169 drill ship at 1.0 m/s fps being around 20, frametime 50-100ms and simulation speed around 50 to 60 with no lag spikes.

Normally my fps was around 5, frametime 100-400 and simulation speed 30-45. Lots of lag spikes.

Have to experiment more.

[joemorin73]

It definitely sounds like an improvement.

[mexmer]

can you check how much it affects perfomance it you move it to updateaftersimulation10 (which shlould be every 10th frame)

[Jake-Rich]

First, sorry about 21:9 instead of 16:9 screenshots :)

I am using a 20km view distance. Single player offline. I have a 30 object limit on this world. I tested in both DX9 and DX11

DX11 Drills Off

DX11 Drills On

DX11 Drilling

DX9 Drills On

DX9 Drilling

This is a baseline unmodified.

[joemorin73]

Is that the baseline or with the UpdateSimulation10?

[mccorkle]

@joemorin73 for Aleks976's screen shots, he said it was baseline.

@IhanaMies is it easy for us to get your build of the SE binary to test the changes on some different machines?

[devu]

Interesting find, but I will add into this. On our server with limit of flying objects set to 64 we were able to reproduce consistently the sim speed drop. However what was consistent, it wasn't a process of mining or generating derbies during mining at all, but when miner was leaving drilling area.

How that would apply? From the profiling screen grab send by Joona looks like ReadRange sims to be pretty intensive task in the process.

On 20 July 2015 at 21:28, Mark McCorkle notifications@github.com wrote:

@joemorin73 https://github.com/joemorin73 for Aleks976's screen shots, he said it was baseline.

@IhanaMies https://github.com/IhanaMies is it easy for us to get your build of the SE binary to test the changes on some different machines?

— Reply to this email directly or view it on GitHub https://github.com/KeenSoftwareHouse/SpaceEngineers/issues/362#issuecomment-123023334 .

[IhanaMies]

@mccorkle here: https://www.dropbox.com/s/c9aqrsidcak52a4/MyDrillBase.cs?dl=0 Insert into SpaceEngineers\Sources\Sandbox.Game\Game\Weapons\Guns

It just decreases the spawned amount of particles and debris when mining. They seem to eat a lot cpu

[mexmer]

Ok, i checked code, and seems that function, that call this function is called from UpdateAfterSimulation10(), also from Shoot(). If you have drill powered, it can create debris only once per 10 frames ... disregard my request then.

[Jake-Rich]

I would like to update this thread: this is not a mining issue. It is a issue with voxels being destroyed. I just tested by firing missiles at asteroids (voxels): as more missiles are fired and at the moment they hit the asteroid (voxel based), the cpu usage gets high and higher. At 15+ rocket pods, cpu usage hits 100% and sim speed starts dropping.

[mexmer]

@Aleks976 that would make sense, if debris is created upon voxel destruction, and if debris is culprit.

[ghost]

From the profiling screen grab send by Joona looks like ReadRange sims to be pretty intensive task in the process.

Definitely. The MyOctreeStorage.ReadRange method looks pretty heavy.

[Tyrsis]

If this is the server, debris should not even be spawned. They are mostly effects that don't really need to be tracked. Spawn them on the client to keep the effect, but drop them on the server.

[joemorin73]

@Tyrsis Doesn't this need to be done carefully? The Server not also mean the hosting machine in a non-DS?

Is there a flag that says DS?

[Tyrsis]

You'd be surprised what you can get away with on the server. This really has no place being on the server. Just like the particle manager, this is mostly just an effect that doesn't need to even be run.

You could probably just do a Sync.IsServer check, and if it's true, don't create debris. It'll probably still create the debris on the client that does no damage, but who cares. The debris is not the floating objects, it's just the effect of those little stones flying out then disappearing.

And yes, the dedicated flag is MySandboxGame.IsDedicated

[Tyrsis]

After quickly looking, I don't see anything syncing from server -> client in regards to these entities, so I highly doubt removing it from the server would even cause you to lose the visual, as the visual isn't linked to anything on the server sync wise. The only thing you'd lose is the bit of physics these things have, but I have never seen them do damage to anything, so I don't think it even matters.

[joemorin73]

If it's going through the sync messaging, I can agree. To be safe, it might be best to use the IsDedicated in regards to the rendering on DS.

But this isn't what this issue is about since this discussion can tangent quickly.

[Tyrsis]

Well the issue is that debris causes lag on the client, but it also creates just as much lag on the server, which can be an issue. And while a solution here might optimize one mining ship, it may not fix when you have 10 players mining at once. It's better to just drop this mechanic from the dedicated server completely. I don't think it's too much of a tangent.

[devu]

I would agree on that.

Still the heaviest thing there to investigate is OctreeStorage.ReadRange https://github.com/KeenSoftwareHouse/SpaceEngineers/blob/b969b03f11be41448f7dd85c0bdbd45536d1e420/Sources/Sandbox.Game/Engine/Voxels/MyOctreeStorage.cs#L712 Notice there is a lag and sim speed drop not only during a drilling operation but when you leaving mining area.

On 21 July 2015 at 20:04, Tyrsis notifications@github.com wrote:

Well the issue is that debris causes lag on the client, but it also creates just as much lag on the server, which can be an issue. And while a solution here might optimize one mining ship, it may not fix when you have 10 players mining at once. It's better to just drop this mechanic from the dedicated server completely. I don't think it's too much of a tangent.

— Reply to this email directly or view it on GitHub https://github.com/KeenSoftwareHouse/SpaceEngineers/issues/362#issuecomment-123445177 .

[joemorin73]

@tyrsis, agreed.

[Tyrsis]

ReadRange is how the voxel is read. I'm not entirely certain what you're talking about in terms of "leaving the area", but I believe what you're experiencing is not related to this, and is actually a much different issue.

Where are you seeing ReadRange called?

Oh in the actual cutout. This has to be as it is. Optimizing the voxel deformation is probably not something that should be handled by just anyone.

[ghost]

Tyrsis, we're referring to this Visual Studio performance profiling report:

I'm not sure what the External Code section is about.

[devu]

To clarify, the "leaving area" Is not exactly what I said. I said "when you (are) leaving mining area" / mined asteroid. Because we talking about massive sim speed slowdown due to mining operation here. but I believe it has something to do to mining but the effect is delayed somehow, I am able to reproduce that each time. I believe those 2 things are related.

If that's another issue well... could you point me to the direction?

On 21 July 2015 at 22:23, Joona Heikkilä notifications@github.com wrote:

Tyrsia, we're referring to this Visual Studio performance profiling report: I'm not sure what the External Code section is about. On Jul 22, 2015 00:12, Tyrsis notifications@github.com wrote:ReadRange is how the voxel is read. I'm not entirely certain what you're talking about in terms of "leaving the area", but I believe what you're experience is not related to this, and is actually a much different issue.

Where are you seeing ReadRange called?

—Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/KeenSoftwareHouse/SpaceEngineers/issues/362#issuecomment-123481957 .

[Tyrsis]

Yes look at that profile closely. See the "ParallelTasks.WorkItem" ? That means it's being done in a thread.

That is going to cause CPU usage on another core other than the game thread. It's nothing to worry about (unless you have 1 CPU??). This is actually one of very few parts of the game properly threaded.

I believe you may be experiencing something related to the ore detector or procedural roids. It should be using a thread on a low priority, but maybe for some reason with you it's causing issue? It's definitely a different issue, and will cause no sim speed issues as it should execute on another thread.

[ghost]

I see, that's very interesting. Thank you for clearing it up.

[alex-davidson]

The slowdown when leaving the asteroid's vicinity appears to be caused by meshes being requested on the main thread, possibly because the colliding objects don't get added to the 'nearby' list and therefore never cause precalc work to be queued on other threads. As soon as the number of grids near the asteroid falls to zero, all those meshes get handled on the main thread.

Modifying MyVoxelPhysicsBody to include MyFloatingObjects in m_nearbyEntities (handlers for CollidableRemoved and CollidableAdded) does seem to solve the problem at the expense of increased overall CPU usage. A more efficient solution is doubtless possible but currently beyond my knowledge.

Currently testing a possible workaround.

[alex-davidson]

Workaround: place a single small-ship landing gear near the asteroid (<500m?). This forces a precalc job.

[paradineai]

I have an idea for the devs to try. What about creating a zero collision dummy (like a ship block that you can not collide with) so that when you mine an asteroid once it has been deemed altered (the same way they do for the procedural asteroids to tell the game if it should be saved or not) then it adds the dummy to the game near the asteroid (<500m) so it forces the game to do precalc for the asteroid but the dummy does not show up in the save files or at the least takes up as little data as possible.

[alex-davidson]

That would add complex behaviour to the system to work around a bug, rather than just fixing the buggy behaviour (ie. it's a hack :P). A simpler way of doing the same thing would be just to queue a precalc job as long as floating objects are nearby, instead of adding a dummy object to try to get the rest of the code to do this.

But as it turned out, the lack of precalc doesn't have anything to do with the bug. The problem was that when there are no nearby objects the asteroid's shape information is discarded to save memory (the lack of precalc is just another side-effect). This shape is then recalculated every time the physics engine detects a collision ie. hundreds of times a second.

The fix is therefore to keep the shape in memory while any collidable object exists nearby. This does mean that this memory cannot be freed as aggressively but it does avoid needlessly recalculating that information every sim update. A possible improvement might be to reduce the shape's detail level in these circumstances, so it takes up less memory.

Once the objects all despawn, the shape will be removed as usual and the memory will be freed.