Mono Compiler

[z3t0]

Hi,

In the readme you stated that the mono compiler needs to be in the same folder as the project, where can I get the mono compiler? Can you please provide a path? Thanks!

[LittleCodingFox]

You can just copy it from the Unity folder, around /Editor/Data/Mono. Copy the Mono folder into the root of your project (so it'd be the same folder where Assets is located) and it should work.

[z3t0]

Okay I see, I'll try and figure out where the unity folder is haha as I'm on linux. Thanks for the help!

[z3t0]

/opt/Unity/Editor/Data

Got it!

[z3t0]

I still have the same error, any ideas?

Unable to compile Mod Bundle 'Character'System.ComponentModel.Win32Exception: ApplicationName='/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Mono/bin/gmcs', CommandLine='-target:library -out:"/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Mods/Character/Character.dll" -r:"/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Library/ScriptAssemblies/Assembly-CSharp.dll" -r:"/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Sample_Data//Managed/UnityEngine.dll" -r:"/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Sample_Data//Managed/UnityEngine.UI.dll" "/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Mods/Character/Scripts/Character.cs"', CurrentDirectory='/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample'
  at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 
  at System.Diagnostics.Process.Start (System.Diagnostics.ProcessStartInfo startInfo) [0x00000] in <filename unknown>:0 
  at Mod.Init () [0x0037a] in /home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/Scripts/Modding/Mod.cs:128 
UnityEngine.Debug:Log(Object)
Mod:Init() (at Assets/Scripts/Modding/Mod.cs:145)
ModManager:LoadMod(String) (at Assets/Scripts/Modding/ModManager.cs:62)
ModManager:Initialize() (at Assets/Scripts/Modding/ModManager.cs:132)
ModManager:get_Instance() (at Assets/Scripts/Modding/ModManager.cs:20)
Game:Start() (at Assets/Scripts/Game.cs:9)

[LittleCodingFox]

I'm currently unable to test on Linux but i'll be able to test it on OSX tomorrow (It uses the same version of Mono).

In the meantime, could you check if your Mono folder has a file named gmcs in the bin folder?

[z3t0]

I managed to find a work around in the meantime at https://github.com/aeroson/mcs-ICodeCompiler.

Yes the folder does have gmcs. One thing that came to mind was that maybe the paths were different in linux/osx than windows so something like

Unity-CSharp-Mod/Sample/Assets/../Mono/bin/gmcs

is handled differently. I'm not too sure. However I think it would be a great idea to implement the solution used in the link above as it is fully contained.

I will look into that project in the meantime and try to understand how I can integrate these two projects.

[LittleCodingFox]

Thanks for the tip, I didn't know the paths could cause problems, and that repo will let me also make the Mono bundle smaller. I'll look into it soon.

[z3t0]

Just wanted to report that the compiler above definitely seems to have some wierd bugs... Just to make sure I'll load this project on Windows with the same code and see if the problem persists

Edit: so on that note it may not be ideal to follow the method used in the link I shared as it is non standardized and certainly has bugs

[z3t0]

Something I found useful was the article below, which states three different routes: Roslyn, CodeDom and MCS.

Currently MCS is being used (I believe? correct me if I am wrong) while Roslyn cannot be used as Unity (via Mono) uses quite an ancient version of .NET while Roslyn requires .NET 4.5 (afaik).

I am going to go ahead and try the CodeDOM route as it seems it would be much more platform agnostic and also supposedly faster.

Article: http://benohead.com/three-options-to-dynamically-execute-csharp-code/

[LittleCodingFox]

Thanks, I'll have a look at this soon and see if I can find out the best way to make things work out for this repo.

[z3t0]

Good news! I some how got it to find Mono, now I have a different error.

Unable to compile Mod Bundle 'Character': error CS0006: cannot find metadata file `/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Sample_Data//Managed/UnityEngine.dll'
error CS0006: cannot find metadata file `/home/bismarck/Documents/Development/Unity-CSharp-Mod/Sample/Assets/../Sample_Data//Managed/UnityEngine.UI.dll'

UnityEngine.Debug:Log(Object)
Mod:Init() (at Assets/Scripts/Modding/Mod.cs:133)
ModManager:LoadMod(String) (at Assets/Scripts/Modding/ModManager.cs:62)
ModManager:Initialize() (at Assets/Scripts/Modding/ModManager.cs:132)
ModManager:get_Instance() (at Assets/Scripts/Modding/ModManager.cs:20)
Game:Start() (at Assets/Scripts/Game.cs:9)

[LittleCodingFox]

Ah yes, you have to build it once and copy the Sample_Data folder to the Sample folder. So basically make a build, copy Sample_Data, into the Unity-CSharp-Mod/Sample folder. It should work that way.

[z3t0]

Ah yes, you have to build it once and copy the Sample_Data folder to the Sample folder. So basically make a build, copy Sample_Data, into the Unity-CSharp-Mod/Sample folder. It should work that way.

So I take it from this that "metadata" is referring to a ".dll"? And if so shouldn't doing the following fix it?

You may choose to enable compilation with the MODMANAGER_COMPILES_MODS define on your player. Otherwise, it'll only try to load the mod .dll. You need a Mono folder containing the Mono compiler (usually from your Unity installation) in the same folder as the project folder in order to compile.

You may choose to disable mod loading at runtime and instead load all mods from memory by compiling all mods into your project and adding the MODMANAGER_LOADS_FROM_MEMORY define to make the mod manager search for all Moddable Types in your Unity assemblies. This is useful in Mobile builds where you can't run the compiler.

Game.cs

#define MODMANAGER_COMPILES_MODS
#define MODMANAGER_LOADS_FROM_MEMORY

using UnityEngine;

public class Game : MonoBehaviour
{
    public GameObject Player;

    void Start ()
    {
        Player = ModManager.Instance.Spawn("Character", Vector3.zero);
    }
}

[LittleCodingFox]

Yes, the metadatas are usually DLLs.

You should define MODMANAGER_COMPILES_MODS on your Unity Player Settings if you want to compile the mods yourself, or MODMANAGER_LOADS_FROM_MEMORY to load the mods from your source code in Unity. It'll find all Mods and load them.

[z3t0]

Yep that did the trick! Sorry, I had no idea how to use defines in Unity

[z3t0]

Now onwards to reverse engineering this project and then replacing the Mono SDK with the compiled .dll provided in the other project... Haha hopefully it's as simple as it sounds

[LittleCodingFox]

Let me know how that goes! Don't worry, if you aren't able to do it I can do it myself when I get a chance :)

[z3t0]

Fear not! The only thing I enjoy more than solving a challenge, is finding another one :smile:

[z3t0]

When you get a chance however could you briefly outline how this project works? Just a small summary would go a long way, Thank you!

[LittleCodingFox]

Of course! Basically I find all Mods inside the Mods folder, search for a .mod file which is really a JSON, parse the JSON, read the details and configurations, if needed, then try to compile the .cs files inside it into a DLL which I can load in Unity.

Then, I search for all Moddable subclasses and register them as available moddables with the same name.

You can create instances of the Moddables yourself or do like I did and make them subclasses of MonoBehaviour. Or both, you can make it search for Moddables and ModdableBehaviours!

I used this more in depth for a game I am working on, where I could create entities and resources from Moddables.

[z3t0]

Awesome that makes a lot of the code much clearer. I'll be running through and expect to have a first beta release done in about 16 or so hours (not sure what timezone you're in so it makes more sense to refer to hours).

I was thinking it would be a great idea if we could then work on creating something branching off of this project into something a bit more inclusive and production ready which we could then put on the Asset Store?

Would working on something like that together be something you would be interested in?

[LittleCodingFox]

There's not a lot of modding support for Unity when it comes to assets, so it might be a good idea.

[z3t0]

Okay great, once I have something meaningful I will create a repo and share it with you and we can go from there.

[z3t0]

If it's okay with you I'd like to use this issue to ask some specific questions about the code. Feel free to answer only at your convenience.

[LittleCodingFox]

Sure, that sounds good

[z3t0]

ModManager.cs : ~ line 100

        Array.ForEach(AppDomain.CurrentDomain.GetAssemblies(), (Assembly) =>
        {
            Array.ForEach(Assembly.GetTypes(), (Type) =>
            {
                if(Type.IsSubclassOf(typeof(Moddable)))
                {
                    MemoryMod.Moddables.Add(Type.Name, Type);
                }
            });
        });

In this it gets all the loaded assemblies but then adds it to the MemoryMod? Is there a reason for adding Moddables to the MemoryMod? And on that note what exactly is the MemoryMod? it seems like it might be some kind of special mod that oversees other mods?

It seems that it adds "Character" as a Moddable when this snippet of code is run, however when the "Mods" folder is removed from "Assets/Scripts/Mods" which contains a duplicate of the "Character" mod which is also found in the "${Project}/Mods" folder this no longer occurs and Memory.Moddables is empty.

So then is it correct for me to assume that this snippet of code serves to search the Unity project's assets for any Moddables so that they are also loaded, and if so then I would assume they are not compiled? As they are already compiled by Unity.

And if that assumption is correct then MemoryMod serves to handle Moddables that are loaded from the Unity Project?

P.S: sorry for being so verbose with my questions, once again only reply at your own convenience :smile:

[z3t0]

Okay I think I think most of my assumptions above are correct which leads me to believe that this Mod System supports either loading from the "Assets" which includes anything inside the "${Project}/Assets OR loading from the "Mods" which is "${Project}/Mods" but never both at the same time.

If so then this itself would be a limitation that needs to be overcome in order to allow both user supplied mods and developer supplied mods.

[z3t0]

Wouldn't lines like the following cause errors on Windows? As the backslashes are replaced, I'm speaking from memory on this but I was pretty sure last I checked that Windows used backslashes for paths.

string ModDirectory = Environment.CurrentDirectory.Replace('\\', '/') + "/Mods/";

Line

[LittleCodingFox]

Regarding the memory mods and non-memory mods: You're supposed to bundle your developer mods the same way as the non-developer mods, that's why it doesn't support both.

Regarding ModDirectory: Windows actually supports forward slashes when it comes to code, so I'm basically formatting everything so it's always using forward slashes. I know this because my main platform is Windows and I've done that for a long time.

[LittleCodingFox]

And yes, loading from memory searches all loaded assemblies for compatible scripts and adds them into the dummy memory mod.

[z3t0]

Regarding the memory mods and non-memory mods: You're supposed to bundle your developer mods the same way as the non-developer mods, that's why it doesn't support both.

That's clever! I haven't gotten all the way through Mod.Init() but then I am assuming that loading ".dll" Mods (probably by copying to a place where Unity can load it) is a planned feature if it is not already included. I say this because many developers may decide to ship mods that contribute a lot to the game and that they would want to remain closed source, hence providing only a ".dll" for that and not the complete source code.

Windows actually supports forward slashes when it comes to code

Haha, that's good to know but greatly confusing. Great job Microsoft :smile:

And yes, loading from memory searches all loaded assemblies for compatible scripts and adds them into the dummy memory mod.

Okay that makes a lot more sense seeing as mods are ideally not supposed to come from within "${Project}/Assets"

[z3t0]

Wow! Just wanted to comment that your code is quite excellent to follow, after your clarification of the MemoryMod everything else seems to be implemented very well, great job!

[LittleCodingFox]

Mod.Init() tries to load the mod and compiles it (but only if MODMANAGER_COMPILES_MODS is on) before attempting to load it (Could maybe make this simpler)

And thank you kindly for that last comment about my code quality. I always try to make very readable code in general!

[z3t0]

Okay so I've finished digesting and reverse engineering :) that usually take a while longer. And two things come to mind first.

1. As per Mono should 'mcs' not be used in place of 'gmcs'
2. Doesn't this compilation method require shipping the game with the complete Mono SDK

For point two, it may be a good idea to look into CodeDOM which may require less dependencies. But still last I tried, calling CodeDOM from unity just wants gmcs... So this may be something to look into.

[LittleCodingFox]

Yeah, I wanna see if there's some better way to do these. What issues did you have with that other project?

[z3t0]

The main issue is that the binary mcs all is compiled in a non verified way which creates many variables and more directly I guess I found a case where I could not run a mod that would simply crash unity. The code itself was fine as I spent a few hours verifying so. But since the mcs is compiled oddly I suspect something in there is buggy.

I know there's a thread somewhere with a few extra details on how mcs was compiled so I'll see if I can find it and the we can try to make a modified version for this library.

[z3t0]

Here's the thread. But there weren't any responses unfortunately

http://forum.unity3d.com/threads/compiling-c-at-runtime.376611/

[z3t0]

It might be worth noting that the Mono folder used by Unity is only ~100 MB. So from that angle the current solution isn't a big deal. However, for flexibility it may be more suitable to use something like CodeDOM, which as far as I can tell is a much more elegant solution.

[z3t0]

Tada! The fruits of my labor. I will get to work on this and invite you to the repo I'm working on once I make a commit to it. But for now here's the compile mcs. Plugins.zip

[z3t0]

Unfortunately it continues to crash with my test case... Back to square one

[z3t0]

I still believe that using a compiled mcs is preferable to providing a MonoRuntime so I will go ahead and write my own implementation of aeroson's ICodeCompiler and see if I can get anywhere...

[z3t0]

Okay, this is quite basic but if you look at the following it is a very simple implementation of CodeDom that can easily be built upon.

Modding System.zip

[LittleCodingFox]

I think that might work but you shouldn't store the assemblies in memory, that will likely cause issues with Unity's unloading of assemblies and also we want the assemblies to be loadable from files anyway. The Load from Memory is mostly available because it's loaded with the game DLL itself.

[z3t0]

Okay so I am at a crossroad of sorts right now.

In either method, as you have pointed out, ideally we want to create and load a DLL. However, regardless of how we make the dll it should not be made within the unity game because then we would need to setup a system where the user has to restart the game after the mod has been compiled from Source.

This is because we should always keep dynamic compilation as an option, so instead the only solution I can come up with is to script a "launcher" that does the compiling, puts the dll in the right place and then launches the Unity game, what do you think?

[z3t0]

Do you know where the .dll needs to be placed to be detected by the built game?

EDIT: no matter where I put the compiled dll it wont get detected by the game, the dll is only detected in the Editor... which isnt all that helpful haha

[z3t0]

Managed to get dll loading with this little hack.

 string dataPath = Application.dataPath;
        #if UNITY_EDITOR
        dataPath = Application.dataPath.Replace("/Assets", "");
        #endif
        string path = dataPath.Replace("\\","/") + "/CoreMod";
        LogWriter.Log("Looking for dlls in '" + path + "'");

        List<string> dlls = new List<string>();;

        var files = Directory.GetFiles(path, "*.dll", SearchOption.AllDirectories);
        foreach (var file in files)
        {
            LogWriter.Log("DLL Found: " + file);
            dlls.Add(file.Replace("\\", "/"));
        }

        Assembly assembly = Assembly.LoadFrom(dlls[0]);
        var types = assembly.GetTypes();

        foreach (var type in types)
        {
            gameObject.AddComponent(type);
            LogWriter.Log(type.ToString());
        }

[LittleCodingFox]

Regarding the mod compilation, we already compile the mods when loading them, so the user doesn't have to wait till mods are done and have to restart, because it compiles the mods on startup.

Regarding the DLLs, they should likely remain in the same folder as the mod that is being compiled, and the DLL should be named the same name as that folder.

[LittleCodingFox]

Regarding your DLL loading, I hope that's demonstration code, because you don't want to add components to some gameobject of every component found on the mods.

[z3t0]

thanks for that catch, I was supposed to add a routine so that only those scripts with monobehvaiour are attached to a game object

[LittleCodingFox]

You can just make a list of all Types that subclass Monobehaviour