dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.02k stars 1.88k forks source link

ML.NET and Unity #1886

Closed mihaar closed 5 years ago

mihaar commented 5 years ago

System information

Issue

Unity editor reports two errors: 1.Assembly 'Library/ScriptAssemblies/Assembly-CSharp.dll' will not be loaded due to errors: Reference has errors 'MLAICore'. 2.Assembly 'Assets/Plugins/MLAICore.dll' will not be loaded due to errors: Unable to resolve reference 'Microsoft.ML.Data'. Is the assembly missing or incompatible with the current platform?

Source code / logs

using System; using Microsoft.ML; namespace MLAICore { public class Class1 { public static string SayHello() { MLContext ml = new MLContext(seed: 0);//if I comment out this line code compiles in unity return "hello from core ML"; //and i get this response, if I live it as is =>err } } }

Please paste or attach the code or logs or traces that would be helpful to diagnose the issue you are reporting.

rogancarr commented 5 years ago

Hi @mihaar ,

Thanks for filing this issue! Technically, ML.NET should work with Unity, but we haven't done comprehensive testing for Unity yet. We've added this for the 0.10 schedule (starting early January, releasing at the end of January).

One question: What version of Unity are you using?

fashrista commented 5 years ago

..I am using unity version: 2018.3.0f2 Personal. And I feel obligated to add that I am not quite sure of what I am doing, but I also find it impossible to build a class library that has it’s ML.Net dependencies included, so I could than use this lib in , let’s say .net console app. Also, if I just add Microsoft.ML dlls to unity plugins folder I also get the same errors ... I am a beginner but if I can help in any way I would be happy to. ps.:I have posted this issue under mihaar username @shauheen AN UPDATE: What did you do? 1.Unity setup: player settings/standalone/ScriptingRuntimeVersion: .NET 4.x Equivalent, player settings/standalone/APICompatibilityLevel: .NET 4.x Equivalent

  1. Made a folder: "MyUnityProject"\Assets\Plugins\ML and pasted all ML .NET dlls into it (got them on gradle). Unity excepted them with no errors.
  2. Have "copied" iris example from https://dotnet.microsoft.com/learn/machinelearning-ai/ml-dotnet-get-started-tutorial What happened? 4.Have trained succsessfully (I think) but, when trying to make a prediction mlModel.MakePredictionFunction it all goes bad with ArgumentException: method arguments are incompatible error as listed below. What did you expect? A regression prediction as I get in VisualStudio solution by doing 3. and 4.
longgood commented 5 years ago

looking forward to see it works with Unity. and announce sooner in the end of coming Jan. Though we are now seeking for the tensorflow complied with Unity.

mihaar commented 5 years ago

@rogancarr I have managet to setup unity in a way I can now import ML dlls and train models, but when trying to make a prediction I came across this error: (similar code works in .net framework 4.7.2 solution) line var predictionFunction = model.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);returns> ArgumentException: method arguments are incompatible System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure, System.Boolean allowClosed) (at :0) System.Delegate.CreateDelegate (System.Type type, System.Reflection.MethodInfo method, System.Boolean throwOnBindFailure) (at :0) System.Delegate.CreateDelegate (System.Type type, System.Reflection.MethodInfo method) (at :0) System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type delegateType) (at :0) Microsoft.ML.Runtime.Api.ApiUtils.GeneratePeek[TOwn,TRow,TValue] (System.Reflection.FieldInfo fieldInfo, System.Reflection.Emit.OpCode assignmentOpCode) (at :0) System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at :0) Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation. System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at :0) System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at :0) Microsoft.ML.Runtime.Api.ApiUtils.GeneratePeek[TOwn,TRow] (Microsoft.ML.Runtime.Api.InternalSchemaDefinition+Column column) (at :0) Microsoft.ML.Runtime.Api.DataViewConstructionUtils+DataViewBase1[TRow]..ctor (Microsoft.ML.Runtime.IHostEnvironment env, System.String name, Microsoft.ML.Runtime.Api.InternalSchemaDefinition schemaDefn) (at <e71d766983cd4fc888a6692d75dfd109>:0) Microsoft.ML.Runtime.Api.DataViewConstructionUtils+ListDataView1[TRow]..ctor (Microsoft.ML.Runtime.IHostEnvironment env, System.Collections.Generic.IList1[T] data, Microsoft.ML.Runtime.Api.InternalSchemaDefinition schemaDefn) (at <e71d766983cd4fc888a6692d75dfd109>:0) Microsoft.ML.Runtime.Api.DataViewConstructionUtils.CreateFromList[TRow] (Microsoft.ML.Runtime.IHostEnvironment env, System.Collections.Generic.IList1[T] data, Microsoft.ML.Runtime.Api.SchemaDefinition schemaDefinition) (at :0) Microsoft.ML.Runtime.Api.ComponentCreation.CreateDataView[TRow] (Microsoft.ML.Runtime.IHostEnvironment env, System.Collections.Generic.IList1[T] data, Microsoft.ML.Runtime.Api.SchemaDefinition schemaDefinition) (at <e71d766983cd4fc888a6692d75dfd109>:0) Microsoft.ML.Runtime.Data.PredictionFunction2[TSrc,TDst]..ctor (Microsoft.ML.Runtime.IHostEnvironment env, Microsoft.ML.Core.Data.ITransformer transformer) (at :0) Microsoft.ML.Runtime.Data.PredictionFunctionExtensions.MakePredictionFunction[TSrc,TDst] (Microsoft.ML.Core.Data.ITransformer transformer, Microsoft.ML.Runtime.IHostEnvironment env) (at :0) MLaiTest.TestSinglePrediction (System.Collections.Generic.List`1[T] signal, Microsoft.ML.MLContext mlContext, System.String modelPath) (at Assets/MLaiTest.cs:131) MLaiTest.Start () (at Assets/MLaiTest.cs:28)

Example is attached newTestCanMlaiDELETE.zip

fashrista commented 5 years ago

Don't know if this helps, but the iris example in unity returns the same error: ArgumentException: method arguments are incompatible. What I did to Imprt ML.NET dlls intu Unity: https://stackoverflow.com/questions/53781312/ml-net-in-unity/53781387#53781387

mihaar commented 5 years ago

As sugested by @shauheen I have tryed importing into Unity and training ML version 0.9.0. The cal: predictionEngine = model.CreatePredictionEngine<AbstractSignal, PredictedRfd>(mlContext); now returns the same error: ArgumentException: method arguments are incompatible System.Delegate.CreateDelegate (System.Type type, System.Object firstArgument, ...

(mind you, method MakePredictionFunction is now renamed to CreatePredictionEngine, but it seems the error is the same)

mihaar commented 5 years ago

EDIT: Aivaras Ivoškus from Unity says: "The issue does not seem to be related to Scripting Restrictions of Unity Engine. "

@shauheen @rogancarr @danmosemsft UNITY SCRIPTING RESTRICTIONS , says that:

System.Reflection.Emit

An AOT platform cannot implement any of the methods in the System.Reflection.Emit namespace. Note that the rest of System.Reflection is acceptable, as long as the compiler can infer that the code used via reflection needs to exist at runtime.

I think the method Microsoft.ML.ApiUtils.GeneratePeek[TOwn,TRow] (Microsoft.ML.Data.InternalSchemaDefinition+Column column) (at <22a400311822408a8bd05691f3cdc119>:0) does that exactly, or am I wrong?

mihaar commented 5 years ago

On sugestion from Unity QA team I have tryed building the app containing ML apis with IL2CPP scripting backend, but got this errors:

1. IL2CPP error for type 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1' in assembly 'C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Temp\StagingArea\Data\Managed\Microsoft.ML.Data.dll' Additional information: Build a development build for more information. Invalid method 'null' found in vtable for 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1'

2. Failed running C:\Program Files\Unity\Editor\Data\il2cpp/build/il2cpp.exe --convert-to-cpp --emit-null-checks --enable-array-bounds-check --dotnetprofile="unityaot" --enable-debugger --compile-cpp --libil2cpp-static --platform="WindowsDesktop" --architecture="x64" --configuration="Release" --outputpath="C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Temp/StagingArea/Data\Native\GameAssembly.dll" --cachedirectory="C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Assets..\Library/il2cpp_cache" --map-file-parser="C:\Program Files\Unity\Editor\Data\Tools\MapFileParser\MapFileParser.exe" --directory="C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Temp\StagingArea\Data\Managed" --generatedcppdir="C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Temp\StagingArea\Data\il2cppOutput"

stdout: IL2CPP error for type 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1' in assembly 'C:\Users\uporabnik\Desktop\newTestCanMlaiDELETE\Temp\StagingArea\Data\Managed\Microsoft.ML.Data.dll' Additional information: Build a development build for more information. Invalid method 'null' found in vtable for 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1' il2cpp.exe didn't catch exception: System.Exception: Invalid method 'null' found in vtable for 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1' at Unity.IL2CPP.Metadata.VTableBuilder.ValidateAllMethodSlots(TypeDefinition typeDefinition, IEnumerable1 slots) at Unity.IL2CPP.Metadata.VTableBuilder.VTableForType(TypeDefinition typeDefinition, Dictionary2 interfaceOffsets, Int32 currentSlot) at Unity.IL2CPP.Metadata.VTableBuilder.VTableFor(TypeReference typeReference, TypeResolver resolver) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddAssemblies(ICollection1 assemblies) at Unity.IL2CPP.AssemblyConverter.Apply() at Unity.IL2CPP.AssemblyConverter.ConvertAssemblies(IEnumerable1 assemblyDirectories, IEnumerable1 explicitAssemblies, NPath outputDir, NPath dataFolder, NPath symbolsFolder, NPath executableAssembiesFolder, NPath monoLibFolder, NPath monoEtcFolder, NPath[] searchDirectories, String entryAssemblyName, NPath[] extraTypesFiles) at il2cpp.Program.DoRun(String[] args) at il2cpp.Program.Run(String[] args) at il2cpp.Program.Main(String[] args) stderr:

Unhandled Exception: System.Exception: Invalid method 'null' found in vtable for 'Microsoft.ML.Transforms.RowShufflingTransformer/Cursor/ShufflePipe/ImplVec1' at Unity.IL2CPP.Metadata.VTableBuilder.ValidateAllMethodSlots(TypeDefinition typeDefinition, IEnumerable1 slots) at Unity.IL2CPP.Metadata.VTableBuilder.VTableForType(TypeDefinition typeDefinition, Dictionary2 interfaceOffsets, Int32 currentSlot) at Unity.IL2CPP.Metadata.VTableBuilder.VTableFor(TypeReference typeReference, TypeResolver resolver) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddVTables(IEnumerable1 types) at Unity.IL2CPP.Metadata.MetadataCollector.AddAssemblies(ICollection1 assemblies) at Unity.IL2CPP.AssemblyConverter.Apply() at Unity.IL2CPP.AssemblyConverter.ConvertAssemblies(IEnumerable1 assemblyDirectories, IEnumerable1 explicitAssemblies, NPath outputDir, NPath dataFolder, NPath symbolsFolder, NPath executableAssembiesFolder, NPath monoLibFolder, NPath monoEtcFolder, NPath[] searchDirectories, String entryAssemblyName, NPath[] extraTypesFiles) at il2cpp.Program.DoRun(String[] args) at il2cpp.Program.Run(String[] args) at il2cpp.Program.Main(String[] args)

UnityEngine.Debug:LogError(Object) UnityEditorInternal.Runner:RunProgram(Program, String, String, String, CompilerOutputParserBase) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:128) UnityEditorInternal.Runner:RunManagedProgram(String, String, String, CompilerOutputParserBase, Action1) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:73) UnityEditorInternal.IL2CPPBuilder:RunIl2CppWithArguments(List1, Action1, String) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:368) UnityEditorInternal.IL2CPPBuilder:ConvertPlayerDlltoCpp(String, String, String, Boolean) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:349) UnityEditorInternal.IL2CPPBuilder:Run() (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:211) UnityEditorInternal.IL2CPPUtils:RunIl2Cpp(String, IIl2CppPlatformProvider, Action1, RuntimeClassRegistry) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:42) DesktopStandalonePostProcessor:SetupStagingArea(BuildPostProcessArgs, HashSet1) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:245) DesktopStandalonePostProcessor:PostProcess(BuildPostProcessArgs) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:46) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr) 3. Exception: C:\Program Files\Unity\Editor\Data\il2cpp/build/il2cpp.exe did not run properly! UnityEditorInternal.Runner.RunProgram (UnityEditor.Utils.Program p, System.String exe, System.String args, System.String workingDirectory, UnityEditor.Scripting.Compilers.CompilerOutputParserBase parser) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:130) UnityEditorInternal.Runner.RunManagedProgram (System.String exe, System.String args, System.String workingDirectory, UnityEditor.Scripting.Compilers.CompilerOutputParserBase parser, System.Action1[T] setupStartInfo) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:73) UnityEditorInternal.IL2CPPBuilder.RunIl2CppWithArguments (System.Collections.Generic.List1[T] arguments, System.Action1[T] setupStartInfo, System.String workingDirectory) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:368) UnityEditorInternal.IL2CPPBuilder.ConvertPlayerDlltoCpp (System.String inputDirectory, System.String outputDirectory, System.String workingDirectory, System.Boolean platformSupportsManagedDebugging) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:349) UnityEditorInternal.IL2CPPBuilder.Run () (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:211) UnityEditorInternal.IL2CPPUtils.RunIl2Cpp (System.String stagingAreaData, UnityEditorInternal.IIl2CppPlatformProvider platformProvider, System.Action1[T] modifyOutputBeforeCompile, UnityEditor.RuntimeClassRegistry runtimeClassRegistry) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:42) DesktopStandalonePostProcessor.SetupStagingArea (UnityEditor.Modules.BuildPostProcessArgs args, System.Collections.Generic.HashSet1[T] filesToNotOverwrite) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:245) DesktopStandalonePostProcessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:46) UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Can build with mono scripting backend though.

danmoseley commented 5 years ago

@mihaar we aren't experts on Unity. However we do want it to work and @anipik will start investigating next week. Hopefully between the two of you we will get it sorted out.

asthana86 commented 5 years ago

Myself and @eerhardt spent some time with Unity 2018.3.4f1 version and have got a basic ML.NET model scoring working. These are primary workarounds we made.

Here's a unity package which shows this play.

https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/end-to-end-apps/Unity-HelloMLNET

Does this help @mihaar?

fashrista commented 5 years ago

Your example works also with binaries (0.9.0) I imported into my project. But I do not know how to implement it into a different project. Can you please do the iris example or the taxi fare example: https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/taxi-fare

Thanks !

On Wed, Feb 6, 2019 at 8:25 AM Ankit Asthana notifications@github.com wrote:

Myself and @eerhardt https://github.com/eerhardt spent some time with Unity 2018.3.4f1 version and have got a basic ML.NET model scoring working. These are primary workarounds we made.

  • Create a plugins folder in assets and add core ML.NET Nuget binaries along with all nested dependencies (e.g. Sytem.Memory) and inherent native dependencies (e.g. CPUMathNative)
  • Use lower-level API to score ML.NET models and avoid the high level convinience APIs e.g. PredictionEngine as they currently make use of reflection emit which throws up.
  • Target .NET 4.x API compatibility level (default for 2018.3.4f1)

Here's a unity package which shows this play.

https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/end-to-end-apps/Unity-HelloMLNET

Does this help @mihaar https://github.com/mihaar?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning/issues/1886#issuecomment-460925348, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj1kTAya0FRTJzsOj1u9P15IKc6Vwkf_ks5vKoNhgaJpZM4ZU87E .

fashrista commented 5 years ago

ps It is not wisible in your example how to train the model to get it in the right 'format' so that it wil take floats as parameters

On Thu, Feb 7, 2019 at 12:06 PM đibril fashrista fashrista@gmail.com wrote:

Your example works also with binaries (0.9.0) I imported into my project. But I do not know how to implement it into a different project. Can you please do the iris example or the taxi fare example:

https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/taxi-fare

Thanks !

On Wed, Feb 6, 2019 at 8:25 AM Ankit Asthana notifications@github.com wrote:

Myself and @eerhardt https://github.com/eerhardt spent some time with Unity 2018.3.4f1 version and have got a basic ML.NET model scoring working. These are primary workarounds we made.

  • Create a plugins folder in assets and add core ML.NET Nuget binaries along with all nested dependencies (e.g. Sytem.Memory) and inherent native dependencies (e.g. CPUMathNative)
  • Use lower-level API to score ML.NET models and avoid the high level convinience APIs e.g. PredictionEngine as they currently make use of reflection emit which throws up.
  • Target .NET 4.x API compatibility level (default for 2018.3.4f1)

Here's a unity package which shows this play.

https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/end-to-end-apps/Unity-HelloMLNET

Does this help @mihaar https://github.com/mihaar?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning/issues/1886#issuecomment-460925348, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj1kTAya0FRTJzsOj1u9P15IKc6Vwkf_ks5vKoNhgaJpZM4ZU87E .

rogancarr commented 5 years ago

Hi @fashrista ! The model takes the input format of the data that it was trained with. If you train a model with your data, it will take your same input data at score time.

You'll need to do two things: 1) Specify a training pipeline

  1. Load the model and score it in your app

Specify a training pipeline

a) Load the data

// Define the columns to read
var reader = mlContext.Data.CreateTextLoader(
    columns: new[]
    {
            new TextLoader.Column("Label", DataKind.R4, 0),
            new TextLoader.Column("Feature1", DataKind.R4, 1),
            new TextLoader.Column("Feature2", DataKind.R4, 2),
            new TextLoader.Column("Feature3", DataKind.R4, 3),
        },
    hasHeader: true
);

// Read the data into an IDataView
var data = reader.Read(dataFile);

b) Define a learning pipeline

// Concatenate the features to create a Feature vector.
// Normalize the data set so that for each feature, its maximum value is 1 while its minimum value is 0.
// Then append a linear regression trainer.
var pipeline = mlContext.Transforms.Concatenate("Features", new string[] {"Feature1", "Feature2", "Feature3"})
    .Append(mlContext.Transforms.Normalize("Features"))
    .Append(mlContext.BinaryClassification.Trainers.LogisticRegression());
var model = pipeline.Fit(data);

c) Save the model

using (var stream = File.Create(modelPath))
{
    mlContext.Model.Save(model, stream);
}

Load the model and score it in your app

a) Load the model

ITransformer loadedModel;
using (var stream = File.OpenRead(modelPath))
    loadedModel = mlContext.Model.Load(stream);

b) Define a class for your data and for your output

public class InputData
{
    public bool Label;
    public float Feature1;
    public float Feature2;
    public float Feature3;
}

public class ScoredData
{
    public bool Label;
    public float Feature1;
    public float Feature2;
    public float Feature3;
    public float Score;
    public float Probability;
}

c) Make a prediction engine from the model

var predictionFunc = model.CreatePredictionEngine<InputData, ScoredData>(mlContext);

d) Score your data

my inputData = GetMyInputDataFromSomewhere();
ScoredData prediction = predictionFunc.Predict(inputData);
joncham commented 5 years ago

FYI, this is the same issue as was reported for Mono (that we/Unity use). I'm looking into cherry-picking their fix: https://github.com/mono/mono/issues/9033

fashrista commented 5 years ago

I think I got some good news in a mail the other day:

Re: (Case 1109657) ML.NET and Unity Aivaras Ivoškus: Hey,

We have fixed this problem and it should not appear in the 2019.2.0a9 version of Unity. If you are still able to reproduce it on the latest version of Unity, please respond to this email.

We highly appreciate your contribution. If you have further questions, feel free to contact us.

Sincerely, Aivaras Unity QA Team_


I will be testing this shortly and will be reporting about results here.

paulallenbright commented 5 years ago

If you do how about uploading a link to a unity project that can be loaded and run right out of the box... to get started with.. ( a startup project). Nothing I've pulled down example wise will run. Either Missing DLL errors, or it puts Unity into some weird loop that requires you to kill unity.. running 2018.3.12f

Even pulling down Ml.net from nuget directly inside Unity. doesn't compile after install... dll errors..

fashrista commented 5 years ago

I will be uploading a demo with detailed instructions on how to setup a unity ML .Net project, but for now the compatible version of Unity( 2019.2.0a9) is still an alfa so, we'll have to wait a bit for it to become at least a beta. Until than you can just train the models inside Unity, but you cannot make predictions unless you find some workarround for model.predict since it uses System.Reflection.Emit which does not comply with Unity.

joncham commented 5 years ago

Reflection.Emit should work in the Editor, Standalone, and Android (with Mono).

fashrista commented 5 years ago

@joncham so sorry, my bad, shold say: ... since it calls CreateDelegate and passes a delegate type that does not have the owner type as the first argument... Is that correct?

paulallenbright commented 5 years ago

Look forward to any demo you can put up when it gets released into the wild... thanks!!!!

On Tue, Apr 9, 2019 at 2:52 AM jib fashr notifications@github.com wrote:

I will be uploading a demo with detailed instructions on how to setup a unity ML .Net project, but for now the compatible version of Unity( 2019.2.0a9) is still an alfa so, we'll have to wait a bit for it to become at least a beta. Until than you can just train the models inside Unity, but you cannot make predictions unless you find some workarround for model.predict since it uses System.Reflection.Emit which does not comply with Unity.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning/issues/1886#issuecomment-481125739, or mute the thread https://github.com/notifications/unsubscribe-auth/AJpiQjF_jNhYc5kdQ0TrNF3H6Vq1LCo5ks5vfDizgaJpZM4ZU87E .

paulallenbright commented 5 years ago

Thanks.. looking forward to anything you can do. i'm deep diving into learning unity/vr at the moment. ML would be a great addition. I wish you luck going forward!! On Tue, Apr 9, 2019 at 2:52 AM jib fashr notifications@github.com wrote:

I will be uploading a demo with detailed instructions on how to setup a unity ML .Net project, but for now the compatible version of Unity( 2019.2.0a9) is still an alfa so, we'll have to wait a bit for it to become at least a beta. Until than you can just train the models inside Unity, but you cannot make predictions unless you find some workarround for model.predict since it uses System.Reflection.Emit which does not comply with Unity.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning/issues/1886#issuecomment-481125739, or mute the thread https://github.com/notifications/unsubscribe-auth/AJpiQjF_jNhYc5kdQ0TrNF3H6Vq1LCo5ks5vfDizgaJpZM4ZU87E .

DooblyNoobly commented 5 years ago

Any luck with this? I've tried the Unity Hello World ML example and I still get the errors mentioned above on Unity 2019.2.0a13, 2019.3.0a5 and 2018.3.11f1

mihaar commented 5 years ago

Ok, as promised: @DooblyNoobly @paulallenbright As follows is a bit modyfied Iris Example from https://docs.microsoft.com/en-us/dotnet/machine-learning/tutorials/iris-clustering (this one does not work anymore due to some ML API changes)

  1. First make sure that you have the latest .net version installed and that your Unity version is at least 2019.2.0f1 (this was a preview version) or higher.
  2. Creste a new unity project. Create a Plugins folder inside your Assets folder. Import all ML .Net APIs into that folder (Might be a foolish thing to do, but I have forehand created a visual studio soution and added all those APIs to that solution via nuget, and than just copied those dll files to Assets/Plugins folder in my unity project).
  3. In Assets folder create an Data folder and paste iris.data file from https://github.com/dotnet/machinelearning/blob/master/test/data/iris.data into it.
  4. Create a script named MLuTest and paste into it the following code:

` using Microsoft.ML; using Microsoft.ML.Core.Data; using Microsoft.ML.Data; using System; using System.Collections; using System.Collections.Generic; using System.IO; using UnityEngine;

public class MLuTest : MonoBehaviour { static readonly string _dataPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "iris.data"); static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "IrisClusteringModel.zip"); MLContext mlContext; void Start() { Debug.Log("starting..."); mlContext = new MLContext(seed: 0); IDataView dataView = mlContext.Data.ReadFromTextFile(_dataPath, hasHeader: false, separatorChar: ','); string featuresColumnName = "Features"; var pipeline = mlContext.Transforms .Concatenate(featuresColumnName, "SepalLength", "SepalWidth", "PetalLength", "PetalWidth") .Append(mlContext.Clustering.Trainers.KMeans(featuresColumnName, clustersCount: 3));//read and format flowery data var model = pipeline.Fit(dataView);//train using (var fileStream = new FileStream(_modelPath, FileMode.Create, FileAccess.Write, FileShare.Write))//save trained model { mlContext.Model.Save(model, fileStream); }

    var predictor = mlContext.Model.CreatePredictionEngine<IrisData, ClusterPrediction>(model);//predict

Plugins.zip

IrisData Setosa = new IrisData { SepalLength = 5.1f, SepalWidth = 3.5f, PetalLength = 1.4f, PetalWidth = 0.2f }; Debug.Log(predictor.Predict(Setosa).PredictedClusterId); Debug.Log("...done predicting, now do what u like with it"); }

} public class IrisData { [LoadColumn(0)] public float SepalLength;

[LoadColumn(1)]
public float SepalWidth;

[LoadColumn(2)]
public float PetalLength;

[LoadColumn(3)]
public float PetalWidth;

}

public class ClusterPrediction { [ColumnName("PredictedLabel")] public uint PredictedClusterId;

[ColumnName("Score")]
public float[] Distances;

}

`


This should work right out of the box ... well it did for me. Where you could mess up is when getting api files, they could be different version from mine or just some .net framework compatible. So let me also add the content of my Plugins folder (mind you all those apis may not be necesery do the cherrypicking yourselve).
It used to be (in previous unity versions) that some player settings had to be changed but i did not have to do it. But anhoo here are mine:
![slika](https://user-images.githubusercontent.com/23123138/62822687-23f8d580-bb87-11e9-9a94-bc6e5d7b7692.png)

I hope this helps, since Unity update 19.2 I have not had any problems mentioned in previous posts in this thread.
gkapellmann commented 5 years ago

Well, I just tried this same approach and it still doenst work, Im in Unity 2019.2.0b4, and also tied with a version of 2019.3

I keep getting many errors like this:

Assembly 'Assets/Plugins/Microsoft.ML.PCA.dll' will not be loaded due to errors: Reference has errors 'Microsoft.ML.Data'.

And yes, I did the changes that @mihaar suggests.

DooblyNoobly commented 5 years ago

It’s just not possible keptin’. Btw I just ended up running ML.net as a service and connecting to it from Unity via sockets.

mihaar commented 4 years ago

Here is a simple as pants, working unity project with ML .Net plugins: http://www.filedropper.com/dotnetmldemocandelete It is as described above: Loads , trains , predicts. If U wont to use any other predictors you will have to add libs (ML .dll-s) to plugins folder. If U have any more questions, feel free to ask.

http://www.filedropper.com/dotnetmldemocandelete

gustavnygren commented 4 years ago

@mihaar I dont think we can reach your links above anymore. Would you mind sharing why you removed them or link to the solutions again?

@gkapellmann Did you ever find a solution to this?

KrasnovPavel commented 3 years ago

Hello, everyone! I managed to make ML.NET work with Unity using everything that was written here. But i have one issue. I can not use mlContext.Data.LoadFromEnumerable(). So I need to save my data to file and then load them, and I really don't like it.

I found workaround in Unity-HelloML.Net sample, but class ArrayDataViewBuilder is internal now. So I cant use it in my code.

Is there some way to use LoadFromEnumerable() (or another workaround)?

@codemzs @rogancarr @asthana86 Can you, please, reopen this issue?

eerhardt commented 3 years ago

Can you, please, reopen this issue?

Hi, @KrasnovPavel. Can you open a new issue describing your scenario? List what you are doing, what the expected results are, and what the actual results are. This will help us in addressing the issue.

mihaar commented 3 years ago

@KrasnovPavel Not sure this is a unity env thing. Have had trouble with this also outside unity env. Would sugest testing it with similar .NET framework/core version outside unity. (create a simple console app to test which.NET version is a problem)