How to actually use it inside Unity Project

[enzyme69]

Hi Keijiro, I did all you specified, like cloning the CoreML repo and put it into the specific path you mentioned, name it as StableDiffusion folder. All those done.

But I have now and empty Unity Project 3D, and not sure what to do next?

I could not find example Unity Project?

[enzyme69]

I decided to just copy all your folder repo into my Unity Project and run it anyway.

Currently running this on Mac Mini M2, and it does not have Camera by default, unlike macBookPro or iMac.

Can I make this to work with Mac Mini and iPhone camera?

[keijiro]

You can change the camera device in the Image Source component.

[enzyme69]

Hmm... will it work with 2D or 3D projects?

So far I kept getting this error: Platform Name 'macOSStandaloneServer' not supported. Supported platform names: (empty)

I am not familiar with changing camera device in the "Image Source" component.

Need your direction on how to stream from iPhone camera or webcam into this project.

[keijiro]

I'm under the impression that you don't have experience with Unity development... If so, I'd recommend doing basic tutorials before this project. Unfortunately, this project hasn't been sophisticated yet as learning material.

[enzyme69]

Okey, I'll do a bit of research first. Sorry for newbie question.

[keijiro]

Please understand that this repository is still under heavy construction. If I write a detailed instruction, it will change very soon. That's why I said it's not for learning.

I close this issue as answered.

[andrewlaboy]

@enzyme69 @keijiro the issue is the Unity version. I tried opening in 2022.2, same error. Tried editing the asmdef to get rid of Mac Standalone not supported error, and more errors of missing namespaces appeared. Switched to 2023 beta that Keijiro is on and it works.

[keijiro]

@andrewlaboy Good point. That's my bad. Now this project clearly requires 2023.1 (for the new async/await APIs). I'll update README. Thanks for pointing it out.

[andrewlaboy]

@keijiro mochiron sa

[enzyme69]

Thanks for hints, I kept digging, stuck at this. But I will try older version and see also if I could get iPhone camera to work on this.

Very curious.

[enzyme69]

Archive.zip

I am trying the TestStatic, it should work, but I got Unity crashing. Log above.

I am testing on both my old iMac 8GB M1 and mac Mini M2 32GB.

Crashing on M2.

[enzyme69]

Previously, I have been having issue with security (inside Unity), it keeps on pointing at: libStableDiffusionPlugin.dylib --> from unknown developer. But I go to finder and then right click and "open".

But then now Unity keeps on crashing.

[enzyme69]

Obtained 55 stack frames.
#0  0x0000029a76fe78 in $s21StableDiffusionPlugin11SDSetConfig_6prompt9stepCount4seed13guidanceScaleys13OpaquePointerV_AHs5Int32VAJSftF
#1  0x000002dac72bbc in  (wrapper managed-to-native) MLStableDiffusion.Plugin:_SetConfig (MLStableDiffusion.Plugin,string,int,int,single) [{0x11a889ba0} + 0xc4]  (0x2dac72af8 0x2dac72c44) [0x293182a80 - Unity Child Domain]
#2  0x000002dac72a9c in  MLStableDiffusion.Plugin:SetConfig (string,int,int,single) [{0x11a889798} + 0x3c] [./Packages/jp.keijiro.ml-stable-diffusion/Runtime/Plugin.cs :: 30u] (0x2dac72a60 0x2dac72aa8) [0x293182a80 - Unity Child Domain]
#3  0x000002dac71d68 in  MLStableDiffusion.Pipeline/<RunAsync>d__29:MoveNext () [{0x11a889648} + 0xc0] [./Packages/jp.keijiro.ml-stable-diffusion/Runtime/Pipeline.cs :: 84u] (0x2dac71ca8 0x2dac725d0) [0x293182a80 - Unity Child Domain]
#4  0x000002dac7131c in  MLStableDiffusion.Pipeline:RunAsync (UnityEngine.Texture,UnityEngine.RenderTexture) [{0x11a889600} + 0x194]  (0x2dac71188 0x2dac71348) [0x293182a80 - Unity Child Domain]
#5  0x000002dac70878 in  Tester/<RunPipelineAsync>d__18:MoveNext () [{0x296c8f730} + 0x218] [/Users/jimmygunawan/My project/Assets/Test/Tester.cs :: 76u] (0x2dac70660 0x2dac70c14) [0x293182a80 - Unity Child Domain]
#6  0x000002dac6fd04 in  Tester:RunPipelineAsync () [{0x296c8efe8} + 0x12c]  (0x2dac6fbd8 0x2dac6fd2c) [0x293182a80 - Unity Child Domain]
#7  0x000002dac6fbbc in  Tester:OnClickGenerate () [{0x296c8f010} + 0x14] [/Users/jimmygunawan/My project/Assets/Test/Tester.cs :: 88u] (0x2dac6fba8 0x2dac6fbc8) [0x293182a80 - Unity Child Domain]
#8  0x000002aaf15ccc in  UnityEngine.Events.InvokableCall:Invoke () [{0x11b0ff6d0} + 0x34] [/Users/bokken/build/output/unity/unity/Runtime/Export/UnityEvent/UnityEvent.cs :: 178u] (0x2aaf15c98 0x2aaf15ce0) [0x293182a80 - Unity Child Domain]
#9  0x000002aaf159d4 in  UnityEngine.Events.UnityEvent:Invoke () [{0x29cf0f600} + 0xb4] [/Users/bokken/build/output/unity/unity/artifacts/generated/UnityEvent/UnityEvent_0.cs :: 57u] (0x2aaf15920 0x2aaf15b18) [0x293182a80 - Unity Child Domain]
#10 0x000002dac6f37c in  UnityEngine.UI.Button:Press () [{0x296bfabe0} + 0x5c] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/UI/Core/Button.cs :: 71u] (0x2dac6f320 0x2dac6f38c) [0x293182a80 - Unity Child Domain]
#11 0x000002dac6f2c0 in  UnityEngine.UI.Button:OnPointerClick (UnityEngine.EventSystems.PointerEventData) [{0x296bfac08} + 0x28] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/UI/Core/Button.cs :: 115u] (0x2dac6f298 0x2dac6f2d0) [0x293182a80 - Unity Child Domain]
#12 0x000002dac6f278 in  UnityEngine.EventSystems.ExecuteEvents:Execute (UnityEngine.EventSystems.IPointerClickHandler,UnityEngine.EventSystems.BaseEventData) [{0x2d2077c88} + 0x50] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/ExecuteEvents.cs :: 58u] (0x2dac6f228 0x2dac6f288) [0x293182a80 - Unity Child Domain]
#13 0x000002dac6a628 in  UnityEngine.EventSystems.ExecuteEvents:Execute<T_REF> (UnityEngine.GameObject,UnityEngine.EventSystems.BaseEventData,UnityEngine.EventSystems.ExecuteEvents/EventFunction`1<T_REF>) [{0x12d8106f0} + 0x228] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/ExecuteEvents.cs :: 272u] (0x2dac6a400 0x2dac6a6f8) [0x293182a80 - Unity Child Domain]
#14 0x000002dac6ee18 in  UnityEngine.EventSystems.StandaloneInputModule:ReleaseMouse (UnityEngine.EventSystems.PointerEventData,UnityEngine.GameObject) [{0x1291770d0} + 0xa8] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/InputModules/StandaloneInputModule.cs :: 192u] (0x2dac6ed70 0x2dac6efa4) [0x293182a80 - Unity Child Domain]
#15 0x000002dac62810 in  UnityEngine.EventSystems.StandaloneInputModule:ProcessMousePress (UnityEngine.EventSystems.PointerInputModule/MouseButtonEventData) [{0x129177300} + 0x3d8] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/InputModules/StandaloneInputModule.cs :: 642u] (0x2dac62438 0x2dac62828) [0x293182a80 - Unity Child Domain]
#16 0x000002dac5c924 in  UnityEngine.EventSystems.StandaloneInputModule:ProcessMouseEvent (int) [{0x1291772b0} + 0xf4] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/InputModules/StandaloneInputModule.cs :: 547u] (0x2dac5c830 0x2dac5cb48) [0x293182a80 - Unity Child Domain]
#17 0x000002dac5c7d0 in  UnityEngine.EventSystems.StandaloneInputModule:ProcessMouseEvent () [{0x129177260} + 0x18] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/InputModules/StandaloneInputModule.cs :: 527u] (0x2dac5c7b8 0x2dac5c7dc) [0x293182a80 - Unity Child Domain]
#18 0x000002dac5bff8 in  UnityEngine.EventSystems.StandaloneInputModule:Process () [{0x129177170} + 0x70] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/InputModules/StandaloneInputModule.cs :: 282u] (0x2dac5bf88 0x2dac5c03c) [0x293182a80 - Unity Child Domain]
#19 0x000002d9838d54 in  UnityEngine.EventSystems.EventSystem:Update () [{0x1294133a0} + 0x164] [./Library/PackageCache/com.unity.ugui@1.0.0/Runtime/EventSystem/EventSystem.cs :: 533u] (0x2d9838bf0 0x2d9838e34) [0x293182a80 - Unity Child Domain]
#20 0x000002a070bec0 in  (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr) [{0x297f7df50} + 0x70]  (0x2a070be50 0x2a070bf5c) [0x293182a80 - Unity Child Domain]
#21 0x0000029354f940 in mono_jit_runtime_invoke
#22 0x000002936d4edc in do_runtime_invoke
#23 0x000002936d4e1c in mono_runtime_invoke
#24 0x00000105899760 in scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)
#25 0x00000105874d68 in ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)
#26 0x000001058381d0 in MonoBehaviour::CallMethodIfAvailable(int)
#27 0x0000010583807c in MonoBehaviour::CallUpdateMethod(int)
#28 0x000001052a4bf4 in void BaseBehaviourManager::CommonUpdate<BehaviourManager>()
#29 0x000001052a49e4 in BehaviourManager::Update()
#30 0x000001054d2804 in InitPlayerLoopCallbacks()::UpdateScriptRunBehaviourUpdateRegistrator::Forward()
#31 0x000001054c33f0 in ExecutePlayerLoop(NativePlayerLoopSystem*)
#32 0x000001054c33b8 in ExecutePlayerLoop(NativePlayerLoopSystem*)
#33 0x000001054c3810 in PlayerLoop()
#34 0x00000106419920 in EditorPlayerLoop::Execute()
#35 0x0000010641a234 in PlayerLoopController::InternalUpdateScene(bool, bool)
#36 0x000001064128c0 in PlayerLoopController::UpdateSceneIfNeededFromMainLoop()
#37 0x0000010640ff4c in Application::TickTimer()
#38 0x0000010771c3e0 in -[EditorApplication TickTimer]
#39 0x000001a7839538 in __NSFireTimer
#40 0xed750001a68a448c in __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__
#41 0x37448001a68a4134 in __CFRunLoopDoTimer
#42 0xfd368001a68a3c8c in __CFRunLoopDoTimers
#43 0x63270001a688941c in __CFRunLoopRun
#44 0x5b438001a688858c in CFRunLoopRunSpecific
#45 0x322b8001b00bddf4 in RunCurrentEventLoopInMode
#46 0xd4c8001b00bda84 in ReceiveNextEventCommon
#47 0xc7030001b00bd988 in _BlockUntilNextEventMatchingListInModeWithFilter
#48 0x97790001a9aa7f58 in _DPSNextEvent
#49 0xc260001a9aa70f4 in -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]
#50 0x99390001a9a9b558 in -[NSApplication run]
#51 0x193f8001a9a729a8 in NSApplicationMain
#52 0x3b2f000107735378 in EditorMain(int, char const**)
#53 0x000001077356f0 in main
#54 0x000001a6453f28 in start

This time, I get the crash message above.

[enzyme69]

@andrewlaboy Can you use machine like Mac Mini that has no camera? I mean I also tried static one, but keeps crashing. While on my iMac M1 8GB, it's basically too slow, the machine uses the fans, and I waited but nothing.

[keijiro]

Could you try Diffusers app?

https://apps.apple.com/us/app/diffusers/id1666309574?mt=12

It uses the same library to run the model. If it runs, this project should run too.

[enzyme69]

@keijiro Yes that Diffusers app works fine and pretty fast.

I also have DrawThingsAI app running and Automatic1111 webUI working.

My feeling is that when I use:

git lfs install
git clone https://huggingface.co/apple/coreml-stable-diffusion-2-base

I get folders but they are empty (no big model files). I have no idea why.

[keijiro]

Yes that Diffusers app works fine and pretty fast.

Okay. The fist step is cleared. Your Mac has the capability to run the Core ML SD model.

I also have DrawThingsAI app running and Automatic1111 webUI working.

That's not relevant because they don't use Core ML.

[keijiro]

My feeling is that when I use:

git lfs install git clone https://huggingface.co/apple/coreml-stable-diffusion-2-base

I get folders but they are empty (no big model files). I have no idea why.

Okay. So, were you trying to run the project without the Stable Diffusion model files? That might be the root cause of the problem.

Please solve this problem on your side. ChatGPT might be helpful for this kind of problem than me (I can't speak English).

[enzyme69]

Yes, confirmed. that's the cause of issue. I run the project with empty folders / models.

I finally manage to get the large model via git LFS etc.

Followed your directions placing the model, it's running fine for the static and dynamic example of SD inside Unity. Speed seems okey, the 3 teapots updates every few seconds at lower samples. I have not tested the one running with "FaceTime HD" because I am using Mac Mini. Would be interesting to use iPhone camera.

Thanks again for your help and this projects, I will try studying it.

[keijiro]

You can select the device with the Image Source component that is attached to the "Tester" game object. I confirmed that it works with iPhone camera wirelessly.