Epic: Optimising Deployment Performance

[Sewer56]

Epic

This epic tracks all issues potentially related to 'deployment performance'.
That is; the time between the user clicking Launch Game within the GUI/CLI application and being ready to start the game process.

I.e. This includes steps such as:

• Reading Metadata of all Mods to Use.
• Resolving Dependencies.
• Sorting Mods.
• Verifying Integrity of Files.
• Pushing Files to Game Folder [if applicable].

Relevant Issues

• https://github.com/Nexus-Mods/NexusMods.App/issues/84
• https://github.com/Nexus-Mods/NexusMods.App/issues/26
• https://github.com/Nexus-Mods/NexusMods.App/issues/27
• https://github.com/Nexus-Mods/NexusMods.App/issues/125 [<= improved perf in solution to this]
• https://github.com/Nexus-Mods/NexusMods.App/issues/682

General Targets

At this moment in time these targets are completely 100% arbitrary; 
that is to say 'I made them up here and now as a placeholder'. 
Everything below is up for discussion.

• <30s from-scratch deployment time.

  • This is deploying for the very first time.
  • It might not be a bad idea to express this somehow as 'deployment MB/s'.
  • Of course this will vary per game: e.g. sometimes we might need to compress stuff, merge files etc.

• ~1s repeat deployment time.

  • i.e. Time between user pressing Launch Game and it booting if no or almost no mods changed.

Please bear in mind; we developers have usually higher end machines than most end users; so think of those targets being times for a mid range machine from 2-3 years ago. i.e. Assume our target is something like a Ryzen 3600 with a SATA SSD [or high end HDD as games drive].

[halgari]

As far as metadata loading and the like goes, this problem mostly goes away when we move to a long running application with a UI. While the mod metadata sits in the IDataStore it is immutable and so all all the app needs to do is listen for changes to the loadout as a whole, then update the UI accordingly.

Since the basis of the loadout is immutable, the output of any calculations based on that loadout are also cachable. If the loadout doesn't change, the sorting won't change, and thus the proposed deployment (or "apply") plan also won't change.

With that in mind we're left with only one step on every apply: we know what the game folder should look like, does it look like that? That involves a full game folder scan and a comparison against the in-memory list of expected files. For some games this may get expensive, so we could optimize this process by using a filesystem watcher and maintaining a in-memory cache of the folder.

I guess where I'm going with all of this is that there are a pile of optimizations we get for free here due to the fact that our app will be long running, the data is immutable, and the files we apply to the game folder are not removed when the game exits. We only remove files from the game folder when the user switches to a new loadout or reverts the loadout completely (which itself is applying a loadout, just one that happens to match the game's native file configuration).

[Sewer56]

I think that under the immutable mod list/data concept; safe asynchronous deployment is also a potential possibility.

So when a user checks a mod in the mod loadout, we can start deploying the data immediately on a background thread and if they change their mind and then untick, we can then cancel the operation and roll back to the last state. Alternatively, if they tick another mod, we put it in the queue and start applying that right after doing the last operation.

Just something that comes to mind.

[halgari]

Yep, I think that's a great way to amortize the cost of the deployment.

[Sewer56]

Misc Note [Semi Related]

Ideally down the road, I'd like to migrate the Reloaded-II ecosystem (and its estimated ~35000 unique users, ~5000 active-ish [I don't do analytics]) into NMA; with NMA serving as the new launcher through either an extension or a more direct integration. [i.e. Native integration; auto-convert all metadata to NMA format; load mods specified in user's mod loadout].

It should make sense for everyone involved; the R2 users get a launcher that works natively under Linux/OSX and the perks of 1st party modding website integration; while NMA gets a boost in supported games/users and a very mature tried and tested ecosystem for code injection.

---

Thinking of it as something I'd start working on late down the backlog; before full release (as in end users can start migrating to this project kind of, release), end of this year, early next year. For now we should focus on functionality for the big games and creating a Minimum Viable Product.

Reason I bring this up is that when said time comes; I'd like to start converting games to use a null-deployment solution that would be instead handled by Reloaded-based VFS mod/extension in places where it would logically make sense and benefit the end user (e.g. Native Code based Windows Only Game w/ lots of data), or a game that requires Archive Emulation.

Because we have an extensible launcher that will eventually use extensions [either internal, and/or via plugin system]; this sort of operation could be automated; with no need for action from mod authors or end users.

[Misc note: Still finishing up work on said VFS; maybe I'll be done with it in 1-3 weeks; depends how motivated I'll be during free time :p]

---

Edit:

Note to self:

• I'll need to write a design document draft for native [plugin supported] loader integration sometime down the road.