How to trigger NGO CodeGen via custom buildpipeline?

[Laumania]

This is a long shot and I'm assuming/guessing things based on what I have researched. I also know I'm pushing the limits here - but hey somebody have to do it :)

Background

The background for this is that my game supports mods and I'm currently working on adding multiplayer to my game using NGO.

So far things are going pretty good - but there is one major issue I still haven't figured out and it's a really big deal as mod creators will be very limited if I can't get this to work.

I'm using UMod to help support mods in my game (https://assetstore.unity.com/packages/tools/integration/umod-2-0-58293). As far as I understand, it adds feature on top and around Asset Bundles.

The way UMod work is pretty nice.

As the developer of the game (me) I setup various things, like what scripts should be included in the "Mod Tools" and other settings. I then build the "Mod Tools" for my game from inside Unity. This result in a .unitypackage file.

As the mod creators (people from the community) you then open up a fresh Unity and import this .unitypackage file (Mod Tools) and get some Unity windows to create a "mod" and can then build a mod - which can include custom scripts.

The mod creator then "build mod" from a custom Unity menu made by UMod, which generates a .mod file that the game can then pick up. At least simply put - the important part here is the UMod builds the mod here. Remember that.

The problem

So, I have discovered that ClientRPCs, ServerRPCs and NetworkVariables are NOT working if a modder is using it in his custom scripts.

I have done some research that indicates (and here comes the guessing part) that NGO is doing some CodeGen at "game build time" - I guess here: https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/blob/develop/com.unity.netcode.gameobjects/Editor/CodeGen/NetworkBehaviourILPP.cs

I'm guessing that it doesn't work, because this CodeGen is not happening on the custom scripts in the mod when it's build by UMod.

UMod supports custom BuildProcessors to run custom code when building the mods like this:
using System.Collections.Generic;
using UMod.BuildEngine;
using UMod.BuildPipeline;
using UMod.BuildPipeline.Build;
using UMod.Shared.Resources;
using UnityEngine;

// Add a build processor for script assets since we are only interested in scripts that will be compiled by uMod.
// High priority means that it will run after script compilation so build assemblies collection will already be filled out with compiled assemblies.
[UModBuildProcessor(".cs", 1000, true)]
public class PatchAssembliesExample : BuildEngineProcessor
{
    // Methods
    // Called by the build engine just after compilation if the mod includes any script content, passing all script assets as a parameter (Not interested in those).
    public override void ProcessAssetBatch(BuildContext context, IEnumerable<BuildPipelineAsset> assets)
    {
        // get assemblies
        ModAssemblyEntry[] entries = context.BuildAssemblies.Assemblies;

        // Patch all
        foreach(ModAssemblyEntry entry in entries)
        {
            PatchAssembly(context, entry);
        }
    }

    private void PatchAssembly(BuildContext context, ModAssemblyEntry entry)
    {
        // Name info
        string asmName = entry.AssemblyName;

        // Get assembly image - loadable by 'Assembly.Load'
        byte[] asmImage = entry.AssemblyImage;

        // TODO - Run custom patcher code

        // Get the patched result
        byte[] asmImagePatched = ...;

        // Check for patched
        if (asmImagePatched != null)
        {
            // Remove original assembly
            context.BuildAssemblies.RemoveFromBuild(entry);

            // Register new assembly
            if (context.BuildAssemblies.RegisterAssemblyForBuild(asmImagePatched, true) == false)
                Debug.LogWarning("Failed to register patches assembly for build! : " + asmName);
        }
    }
}

So, here comes the million dollar question that will make the sky the limits for modders of my game - how can I trigger this CodeGen for the scripts in the mod?

Note that NGO is installed as a dependency to the Mod Tools, so NGO is installed in the mod creators Unity project.

[Laumania]

Hi @NoelStephensUnity

Its me again, pushing the limits - you have an idea on this one?

Its REALLY important for my game that I get this to work, else modders will be really limited making multiplayer mods.

[NoelStephensUnity]

@Laumania Hello! 👋 Hmmm... I am sure it is an order of operations type of issue... The one thing I am not 100% clear on is whether the user creating the mod is adding additional NetworkVariables and RPCs =or= if they are just extending your classes that already have the NetworkVariables and RPCs and when they extend/mod certain classes that contain NetworkVariables (most likely with custom serializable types) and RPCS those NetworkVariables and RPCs no longer function correctly.

If the later scenario is the case, then this is most likely what is happening:

• User installs your product that already has the generated assemblies with the ILPP code injected
• User then creates a mod based on those assemblies and part of the mod includes "modifying classes" contained within your already built DLLs.
• When UMod "patches the assembly" they are basically removing your package's original assembly and then replacing it with the modified version (or along those lines...where the assembly that contains the injected code is no longer used).

So...unless you are trying to preserve already existing mods (most likely is too late for already UMod generated patches), the best way to avoid losing the ILPP generated code is to create a full abstraction layer between the classes the users will modify and the classes that contain the RPCs and NetworkVariables:

• Your Project's Primary Netcode Class (in primary Assembly/Assemblies) --> i.e. PrimaryAssembly.asmdef

  • SomeCoreNetcodeClass
  • Contains RPCs and NetworkVariables

• Your Project's Customizable Classes (in secondary Assembly/Assemblies)--> i.e. CustomAssembly.asmdef

  • CustomizableClass
  • References SomeCoreNetcodeClass (but cannot derive from!)

You would then most likely need to create some form of generic interface that handles the specific behavior(s) you want to allow users to override/modify...so the CustomizableClass would implement something like ISomeCoreNetcodeClassActions that would then register itself with the referenced SomeCoreNetcodeClass component...then when specific "customizable" actions need to be invoked the SomeCoreNetcodeClass would invoke the registered ISomeCoreNetcodeClassActions.

Where SomeCoreNetcodeClass already has a "default" ISomeCoreNetcodeClassActions implementation in the event nothing is "customized".

Basically you need to only allow UMod to be able to modify assembly/assemblies that only have references to classes in your core netcode enabled assemblies but not modify the assemblies that contain those classes...thus the reason for having different assemblies... one with the compiled NGO derived classes...and one with just the interface implementations (i.e. the default) and the classes that register those implementations with the core netcode enabled classes.

This way UMod will only replace the "default interface implementation" assemblies and not the assemblies with the injected code.

I am a bit heads down currently working on some more updates for NGO, but if you don't hear back from me by the end of this week just ping me and I will try to provide you with a very simple skeleton project of what I am talking about.

[scottyboy805]

@Laumania Hello! 👋 Hmmm... I am sure it is an order of operations type of issue... The one thing I am not 100% clear on is whether the user creating the mod is adding additional NetworkVariables and RPCs =or= if they are just extending your classes that already have the NetworkVariables and RPCs and when they extend/mod certain classes that contain NetworkVariables (most likely with custom serializable types) and RPCS those NetworkVariables and RPCs no longer function correctly.

If the later scenario is the case, then this is most likely what is happening:

• User installs your product that already has the generated assemblies with the ILPP code injected
• User then creates a mod based on those assemblies and part of the mod includes "modifying classes" contained within your already built DLLs.
• When UMod "patches the assembly" they are basically removing your package's original assembly and then replacing it with the modified version (or along those lines...where the assembly that contains the injected code is no longer used).

So...unless you are trying to preserve already existing mods (most likely is too late for already UMod generated patches), the best way to avoid losing the ILPP generated code is to create a full abstraction layer between the classes the users will modify and the classes that contain the RPCs and NetworkVariables:

• Your Project's Primary Netcode Class (in primary Assembly/Assemblies) --> i.e. PrimaryAssembly.asmdef

  • SomeCoreNetcodeClass

  • Contains RPCs and NetworkVariables

• Your Project's Customizable Classes (in secondary Assembly/Assemblies)--> i.e. CustomAssembly.asmdef

  • CustomizableClass

  • References SomeCoreNetcodeClass (but cannot derive from!)

You would then most likely need to create some form of generic interface that handles the specific behavior(s) you want to allow users to override/modify...so the CustomizableClass would implement something like ISomeCoreNetcodeClassActions that would then register itself with the referenced SomeCoreNetcodeClass component...then when specific "customizable" actions need to be invoked the SomeCoreNetcodeClass would invoke the registered ISomeCoreNetcodeClassActions.

Where SomeCoreNetcodeClass already has a "default" ISomeCoreNetcodeClassActions implementation in the event nothing is "customized".

Basically you need to only allow UMod to be able to modify assembly/assemblies that only have references to classes in your core netcode enabled assemblies but not modify the assemblies that contain those classes...thus the reason for having different assemblies... one with the compiled NGO derived classes...and one with just the interface implementations (i.e. the default) and the classes that register those implementations with the core netcode enabled classes.

This way UMod will only replace the "default interface implementation" assemblies and not the assemblies with the injected code.

I am a bit heads down currently working on some more updates for NGO, but if you don't hear back from me by the end of this week just ping me and I will try to provide you with a very simple skeleton project of what I am talking about.

Hi @NoelStephensUnity, My name is Scott, co founder of Trivial Interactve who develop the uMod 2.0 modding asset mentioned here. Just wanted to offer some insight into what uMod is doing as part of its compile process and maybe you will have some more ideas with a better understanding of what is happening.

The first point to mention is that uMod does not actually patch any assemblies or game code as such. Instead it supports additive modding which means compiling additional script assemblies that work as add-ons to the game. To do this uMod runs the Roslyn compiler manually when building mods to compile the .csproj files for the mod project, and I guess since this step is done manually and not via the Unity build pipeline, the code gen is not running on those assemblies which is to be expected.

Since uMod offers extensive build events and callback, one solution that could work is to run the code gen manually just after uMod has compiled the scripts into an assembly. That was the sample code above I provided to @Laumania which is a uMod build event, but obviously we don't know how to run the code gen manually yet. I had a quick look at the code gen source and it looks to me like it is probably not designed to be run manually from some source other than the Unity build pipeline. Is that the case or are we missing some public API that we can use to trigger this process? If we can get to run the code gen manually then I think it will be problem solved and issue closed, but I suspect it may not be that easy? And just to be clear, that would need to be invoked on an already compiled managed assembly (.dll). I see there is an ICompiledAssembly interface which I guess can take care of that, but again it is all internal and probably not designed to be used by the likes of us.

I appreciate what you say about abstracting the networking aspect of the code base away so that modders don't need to deal with that at all. It does seems like a suitbale solution but my only concerns would be the amount of extra work that would mean for game developers. Ie. needing to desing an API to abstract away implementation details of the networking while making sure the API is feature complete and usable in the real world. Seems to me to be quite a complex task although it would mean that modders don't need to learn/work with netcode directly which is probably a plus point, and developers could switch networking backends without issue. Pehaps a quick example of such an abstracting layer would be helpful if you have time just to see what sort of thing we are ataling about. I guess it is probably not as simple as modders marking thier code with RPC attributes or anything like that, and instead will need to work directly with the API to send RPC's and serialized data, so a different more manualy sort of setup I would assume.

Anyway, hopefully that extra info can be somewhat helpful to provide a nice solution to this unconevntional issue/use case. Regards, Scott.

[NoelStephensUnity]

@scottyboy805 Hi Scott,

I agree that the solution I offered isn't an "optimal" path to this type of functionality. What I guess I had excluded from all of that is that currently we are spread a bit thin regarding time allocation due future development in tandem with team members working on some long standing feature requests that are a bit more time intensive.

The biggest question I have is whether this issue applies to @Laumania 's project's assemblies or if this is strictly limited to the scope of user's mod assemblies (or both)? Knowing this would help me focus what to look at specifically when I get a spot of time later today or tomorrow.

[scottyboy805]

@NoelStephensUnity Thanks for getting back so quick. Yes I understand that you probably don't have the resources to devote to such issues so that is no problem. Maybe it can be something to look into in the future but like you say there are probably more pressing issues to deal with first.

I guess this would come down to an issue effecting uMod modders only (The game portion of the code is unaffected by this issue). Essentially when a modder tries to include new C# scripts in a mod, uMod will compile those scripts manually which skips the code gen leaving netcode in what I guess would be a raw/unimplemented state. This process is completly separate from the game (A separate Unity project dedicated for creating a specific mod), so yes it only applies to code created by modders, and modders in general (for games implementing uMod 2.0), not just modders of @Laumania's game.

Hope that helps. Let me know if you need any more info.

[NoelStephensUnity]

@scottyboy805 Ahhh, so that makes more sense... user specific mod with scripts that might implement either @Laumania's game's netcode classes and/or make their own classes will not be able to add additional unique netcode behaviors (i.e. RPCs, NetworkVariables, or anything else requiring ILPP) because of how the scripts are compiled.

So, I am thinking the only real way to do that is to be able to invoke the postprocessors after you have compiled the scripts which would just require you to have access to all of the internal classes here: https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/tree/develop/com.unity.netcode.gameobjects/Editor/CodeGen

I will noodle over this a bit more, but I think exposing some of the CodeGen classes could provide enough to possibly make it possible to handle this in the PatchAssembliesExample.PatchAssembly method... The post processors basically need something like:

namespace Unity.CompilationPipeline.Common.ILPostProcessing
{
    public interface ICompiledAssembly
    {
        InMemoryAssembly InMemoryAssembly { get; }

        string Name { get; }

        string[] References { get; }

        string[] Defines { get; }
    }
}

Need to look at this side of things a bit more...

[scottyboy805]

@NoelStephensUnity Awesome, yes exposing some of the internal classes would solve the problem if that is not too much hassle. Certainly implementing something like the above interface will be ideal and will be easy for us or even developers themselfs to extend, since uMod already allows access to such information in a build event. Alternativley if it is too much work to expose some of the API or would break the intended usage etc, then even just a simple helper method like byte[] CodeGenUtil.PatchExternalAssembly(byte[] asmImage) could work for this purpose just as an example, but what ever you think if best.

Thanks for all the help.

[Laumania]

@scottyboy805 @NoelStephensUnity I'm so happy to see that this might be possible - as said - the sky will be the limit if my mod community can use RPCs and NetworkVariables :D

@scottyboy805 If we knew what method to call during the UMod build pipeline, couldn't we maybe use reflection to call it - just a a way to check if it actually works?

But of cause, some more bulletproof solution is needed.

Really excited about this progress! :D

[scottyboy805]

@Laumania Yeah looks like it will make life a lot easier if this can be added. Reflection is an idea (although a fragile one) but I don't think it can work in this case since it looks like the API will require an implementation of ICompiledAssembly, which can't be done from reflection, so some public API will need to be added I would think to get this all working nicley.

[Laumania]

Hi @NoelStephensUnity

Anything new on this? It's becoming more and more limiting for my modders that they can't do this :)

[guanaco0403]

Hi @NoelStephensUnity

Anything new on this? It's becoming more and more limiting for my modders that they can't do this :)

Yes I confirm am one of them

[Laumania]

Hi @NoelStephensUnity

Happy New Year :)

Any news on this one - because it's really really needed as modders are very limited without this.

[guanaco0403]

Yeah as one of the modders i really need to be able to use at least the NetworkVariables, its really boring that a lot of my ideas cant happen due to that @NoelStephensUnity

[NoelStephensUnity]

Apologies for not responding sooner. We have been working on various areas of NGO and this is still something we have in our queue. Since we will have to re-create a usage scenario it adds to the complexity.

Of course, NGO is open source and so if it is a true blocker then forking and making adjustments is always an option (and it would provide us with a better use case to target).

Unfortunately, we most likely will not be able to get to this until after the update that follows v1.8.0.

[Laumania]

@NoelStephensUnity We understand, thanks for your reply! :)

Any rough ETA on when NGO v1.8.0 will be release, aka when you will start looking into this, just roughly?

[NoelStephensUnity]

We had a slow down on our documentation updates which should be getting pushed in the next week (I am hoping). Due to some of the new features in NGO (i.e. Universal RPCs), we are wanting to make sure that when the package is made public the supporting documentation is ready to be pushed out to our documentation site.

Then there most likely will be another update with some additional fixes (most likely a v1.9.0)... and depending upon timing there could be a bigger update that follows (TBD)...there are several initiatives occurring simultaneously so it would be hard to give a precise ETA...normally I could give a better time frame... I want to get Kitty (one of our netcode engineers) to look over this but she is currently focused on addressing a large number of NetworkVariable/NetworkList/NetworkDictionary issues with many improvements that will be in the update following v1.8.0... so hopefully once she is done with that she can look over this and we can figure out where in the update schedule it fits the best base on her assessment.

[NoelStephensUnity]

Just following up to provide some visibility on #2813 (what Kitty has been working on that might be of interest for those who have posted here). There are still more "things in progress" that we are working through and pushing out to help improve some long-standing areas in NGO.

@ShadauxCat will circle back to take a look at this once she is done doing super-awesome things! :godmode:

[guanaco0403]

That's really nice to know big thanks 😊

[Laumania]

Sounds great! Looking forward to coming updates! :)

[guanaco0403]

Hello @NoelStephensUnity is there any updates on our request ? i saw that 1.8.0 and 1.8.1 got released and it looks great but i still cant use network variables with umod.

[Laumania]

@ShadauxCat Are you done doing super-awesome things, so you can work on this super awesome thing that my mod community is screaming for? :D

See @NoelStephensUnity 's comment above https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/issues/2747#issuecomment-1899318556

[guanaco0403]

Still no updates on the issue @NoelStephensUnity ??

[Laumania]

Still no updates on this one @NoelStephensUnity @ShadauxCat ?

[guanaco0403]

ehm... still nothing ? i kinda really need to be able to use network variables and RPC's for modding

[NoelStephensUnity]

@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.

I will add this to one of my queues...which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.

[guanaco0403]

@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.

I will add this to one of my queues...which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.

big thanks that really makes me happy have a great day

[Laumania]

@NoelStephensUnity That is super cool! Let us know when/if you want us to test some things.

[Nikmazza]

@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.

I will add this to one of my queues...which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.

Oh this is awesome! We got into this issue as well and was wondering whether or not to wait for an update. The looks promising! Thanks!

[Nikmazza]

@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.

I will add this to one of my queues...which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.

Oh this is awesome! We got into this issue as well and was wondering whether or not to wait for an update. The looks promising! Thanks!

[Laumania]

@Nikmazza Nice to see more need this - I'm sure more and more will come, however, it is a bit advanced feature, but for the ones of us needing it, it's very important :)

[Bobafet231]

@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.

I will add this to one of my queues...which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.

Same issue here, i would really like to be able to invoke thoses postprocessors but i see that some others peoples ran into that issue as well soo can't wait for that feature.

[guanaco0403]

Hello @NoelStephensUnity am just here to see if there is been some progress on our issue, i see that 2.0.0-exp5 got out recently and i would like to know if our issue is in that 2.0.0 queue ?

[Laumania]

Hi @NoelStephensUnity Any news or progress on this one? :)

[guanaco0403]

mhhh no ETA ? because i saw somewhere that 1.10 will get out verry soon soo ehm thats odd i tought after 1.9 would be 2.0 and that we will get our fix ??

[Bobafet231]

mhhh no ETA ? because i saw somewhere that 1.10 will get out verry soon soo ehm thats odd i tought after 1.9 would be 2.0 and that we will get our fix ??

Hello, I also would like to know when this feature to call the NGO CodeGen will get out, it has been a lot of time since this issue started I can see. @NoelStephensUnity

[Keltusar]

@NoelStephensUnity How come this is still not working?