Slow loading times on Operations.Receive

[LukasKlepper]

Hi,

we are currently having issues with long loading times within unity with the latest connector. When we are getting an object with ~10k objects then we need to wait ~4min (at least!) for just the Receiving operation of speckle. The algorithm which generates the game objects from the base object only takes around ~30seconds which is really good for that amount of objects.

The problem is at file Receiver.cs @ line 134.

var @base = await Operations.Receive( objectId, remoteTransport: transport, onErrorAction: OnErrorAction, onProgressAction: OnProgressAction, onTotalChildrenCountKnown: OnTotalChildrenCountKnown, disposeTransports: true );

Nearly the same code which I reproduced on .NET Core 3.1 Console Application with the latest NuGet package of speckle core only needs ~45seconds.

Why is it so slow? Is it a known problem? What can I do to speed it up? We are using the connector within an application for an 3d cave environment with a customer. Later on we want to load even larger models, the customer already told us that he experienced loading times up to 40 minutes.

The test above were taken both from our cave application and the provided unity speckle playground. There was no difference in performance.

I also tried to upgrade the speckle connectors speckle sharp dlls manually to the latest version, which also didn't help. Same issue.

Tested on hardware: 11th Gen Intel Core i7 1165G7 @ 2,8GHz 16GB DDR4 RAM 3200MHz SSD Integrated Graphics

[JR-Morgan]

Hi @LukasKlepper,

Thanks for taking the time to help troubleshoot this issue.

What version of Unity are you using?

I have noted that some of our operations are much slower in a Unity context compared with other .NET contexts. Particularly with Unity 2020, which appears to be disproportional slow. The slowness in 2020 appears to be at least somewhat beyond our control. We are considering dropping support for Unity 2020 for this (and other) reasons.

In Unity 2021, I have not seen slowness anywhere near the numbers you are describing.

---

I've just quickly tested receiving a super large Revit commit (177k objects) in Unity 2021.3. I'd consider this size stream to be quite near the upper limit for what is currently possible with Speckle (and what Unity can render without any fancy optimisation).

This stream took 14 mins to receive using the editor Stream Manager component. And similar at runtime, in our Speckle playground sample project

This level of performance reasonably acceptable to me, for the stream's size, and in comparison to our other connectors, I think it's actually faster than most. However, I'd like to continue performance testing, with some varied streams before concluding anything concrete.

If it's possible, could please add me (jedd@speckle.systems) to one of your streams that's particularly slow, it would help immensely, thanks!

[JR-Morgan]

After some further testing, I am noticing some quite significant slowness even in Unity 2021.

Compared with a .NET project, the Deserialisation appears to be around 10x slower in Unity 😢, not good. Going to do some more rigorous measurements, and continue investigating.

[LukasKlepper]

I've tested it with unity 2019, 2020 and 2021 each with mono and il2cpp scripting background. There was no measureable performance difference.

What's strange: Shouldn't the models be cached? Then the second loading time should be much faster than the first, but it is not.

Unfortunately, I can not share the model, as it is a model from the customer.

[haitheredavid]

hi, I might be able to help clarify what's happening here. I don't think the issue with Operations.Receive but with the conversion routines from a speckle stream to native unity geometry.

The algorithm which generates the game objects from the base object only takes around ~30seconds which is really good for that amount of objects.

This part ends up happening after the receive command has completed and starts dropping all the speckle data into the conversion process, and then I'm assuming that's where the long 20-40 minute delays are coming in. It doesn't matter if the geometry is cached locally on your machine, the conversion process still needs to rebuild all the speckle goodness into the unity scene. The converters are currently all bound to run on the main unity thread, and also have a few steps for serializing a speckle object that are creating the bottleneck.

I have some PR's coming in the near future that will address these issues. In the meantime, @LukasKlepper how are you using the connector? Is it for visualization purposes? If you don't need to worry about the properties of your objects and you're comfortable with jumping around the code you could comment out the part in the converter that uses SpeckleProperties, that would reduce the conversion overhead a bit.

[LukasKlepper]

Hi @haitheredavid ,

thanks for you answer. So correct me if im wrong:

the line Receiver.cs@134 takes part of getting all the data from the speckle server. Thats just the base informations which later on will be converted to game objects which get feeded with the speckle properties. We also use that exactly same method in other applications which build up on speckle, but in asp.net core server applications rather than unity. In other projects this method is very fast (~45 seconds for the same model as in unity)

the line Receiver.cs@142-152 takes those informations from the base object and then converts this base information to unity game objects ands adds the speckle properties to it. It gets queded by the Dispatcher to the main thread as only there you can build game objects or access specific api calls. Of course thats slower than multi-theading but I think thats the way unity currently works on the classical API. I'm not a unity expert so thats a pure idea: Maybe Unity DOTS is a way how those game objects could be built up multi threaded? I've never done it so I dont know if that could solve the problem of conversion process, but as you will later on read, thats (for me) not as much of a problem.

I've debugged the code with visual studio and got to the point where I moved trough it via procedure steps (so for every new call I needed to press it manually). For the timing I grabbed my phone and stopped the watch for the time I've started the method and when it arrived at the next call.

Thats the way I calculated (debugged) that the lines:

134: var @base = await Operations.Receive(...) takes around ~4min

146: var go = rc.ConvertRecursivelyToNative(@base, commitId); takes around ~30 seconds (sometimes its longer, but no longer than one minute)

for a model with ~10k objects in both Unity 2020 and Unity 2021.

Is that a wrong way to calculate it? Did I make an mistake? Or is Operations.Receive already doing a conversion in the background?

---

To answer your question: We are currently visualising a model to walk through it like in a vr application but in a cave. Later on there will be also the need to select objects and look at the speckle properties, so removing those properties is not a solution.

[haitheredavid]

@LukasKlepper I'm a bit confused on the issue now, it seemed like your first post was saying that the game objects were created in the hierarchy relatively fast but there was a slow follow up to loading all of the model data into the scene. With your last point it seems like the heaviest moment is from the 4min Receive call. Am I understanding this right?

Can you give us some info about what is in that stream of yours? I'm mainly curious where the data is coming from (Revit, Rhino). I'm not expert in the technical details of all the different object types used in speckle, but I know @JR-Morgan and I'm sure knowing where the data is coming from would help him help you.

A couple of thoughts I have with the information you tossed out:

• I'm not sure I understand how you are debugging the code and I'm expecting that to be creating some sort of confusion. Sometimes with the debugger (specially with multi-threaded stuff) there can be moments where threads are not finishing or not being held at the same break line. I would recommend modifying the code with some Debug.Log() calls and using the c# native Stopwatch object to report times.
• 10k objects in unity (not using object pooling and all having custom meshes and materials) is no small amount. I've been running test for another project and I usually see a huge performance drop when you move past 8k. It's not impossible to handle it though.
• Operations.Receive is only pulling speckle objects out of the database and loading them into Base objects. The converters handle adapting that data into the native connector.
• You are correct about the Dispatcher structure and that it's pinned to the unity's main thread. In the future PR's this process will be modified so all object and component setup is done prior to the conversion process, making so all main thread stuff is done there and any model / data processing can be awaited. It will also bring along using Unity's new Mesh API that relies more on NativeArrays when building meshes, which will reduce the amount of crap we drop in the GC. This will bring along some of parts of DOTS (Burst and Jobs mainly) but not using entities. DOTS won't be stable till next year and would create a huge change to how the connector works in regards to dev UX. We could adapt some of the converters we have cooking up to be DOTS based before the official release (which might be good for testing) but I see more value from improving how we handle unity game objects first.

[LukasKlepper]

@haitheredavid Hi there,

i've just retested it, the operation now took 250 seconds with stopwatch for just the receiving part of speckle. I dont know why its so slow. Also interestingly I cannot even load the model in speckle.xyz. Its just loading forever. The model comes out of revit, its ~10k mep objects.

Code used to benchmark:

133: var transport = new ServerTransport(Client.Account, StreamId); 134: System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch(); 135: Debug.Log("Starting stopwatch Operations.Receive()."); 136: sw.Start(); 137: var @base = await Operations.Receive( objectId, remoteTransport: transport, onErrorAction: OnErrorAction, onProgressAction: OnProgressAction, onTotalChildrenCountKnown: OnTotalChildrenCountKnown, disposeTransports: true ); 145: sw.Stop(); 146: Debug.Log($"Stopped stopwatch Operations.Receive(). Stopwatch took {sw.Elapsed.TotalSeconds} seconds.");

The properties are there, after the stopwatch finished it just took seconds before the model was shown in unity. I dont think its a problem with the unity converter rather than the communication in the DLL between Unity and Speckle. Maybe its because its mono underneath and not native .net?

As you see the models are relatively detailed and its the basic properties of mep objects, nothing fancy.

As of the performance things we already knew that, thats why our customer machine is very beefy:

• gpu 2x RTX 3090
• cpu 5950x
• 32 gb ram
• ssd

Im sure it should be able to handle even more objects than ~10k as they are all static, but correct me if im wrong.

Im looking forward to the DOTS support, I always wanted to try it :)

[LukasKlepper]

Hi, are there any news up on this issue? Our problem still exisits within unity.