Perfomance bottleneck and possible bug in ressource loading

[Denadan]

Describe the problem

When the game wants to load a resource, the load request is queued in DataManager.foregroundRequests, a generic C# List. Before each request is added, the Find() method of the list is used to check whether the resource is already queued. Find is O(n), but when it is called by every resource the whole process becomes O(n²). The number of checks grow exponentially with number of resources, resulting in a huge performance impact.

For example, the vanilla game loads around 2K unique resources when starting a new game. Each Find() call does 1000 comparisons (on average), so there are 2M checks if only unique resources are requested. But that is not the case. In reality, over 100M checks are made. The number will further explode on a modded game with added items and mechs, which generates lot of interdependent load requests.

Provide an example of the code where the problem occurs

// in DataManager.RequestResource_Internal()
  DataManager.DataManagerLoadRequest dataManagerLoadRequest =
      this.foregroundRequests.Find((DataManager.DataManagerLoadRequest x) => 
          x.Matches(resourceType, identifier)
      );

This call generates most of the loading time in the game.

Provide an example of your proposed solution

Replace List<> with a collection that has O(1) seek time such as Dictionary<>. This also means the code need to find a resource by hash instead of by a comparator.

Provide a link to a commit in the private repo for this issue

Leave any additional comments here

When starting a new game on my hardware, there are 5K calls to RequestResource per second for 26 seconds, or 130K calls total.

[caardappel-hbs]

This work is in progress for the 1.2 release.

[Sheep-y]

Meanwhile, I attempted a BTML mod fix. It seems to improves my loadtime slightly, but I didn't test it a lot.

Turbine-20180731.zip (Development continued and released at https://github.com/Sheep-y/BattleTech_Turbine/releases )

One thing that made an obvious impact for me is I keep a separate "loading" list. Notified requests are removed from the list, which is used to detect load completion, without going through loaded resources. I tried to do it with a simple count but some resources seem to complete their load without notice; if that can be done it should run even faster.

Sourcecode included and can also be read here: https://github.com/Sheep-y/Attack-Improvement-Mod/blob/master/Turbine/src/Mod.cs

P.S. The original post was written by a Russian, editing by a German, then a Chinese, among others. A truly international effort. :grin:

[CptMoore]

there seems to be another issue with the DataManager and how the MessageCenter works, see #17

[Morphyum]

Work is in progress on HBS side so closeing this for now.