1.2.8 fails to receive data

Gabri94x commented 1 month ago

Describe the bug

hi there, i just tried the new release to run the phi-3 model, but, when running any model, it throws errors. I was think about problem on my project, but i have the same result in a fresh project importing the simple chatbot sample. I tried using phi-3 and the llama3 (the one i was running on the 1.2.7 versione). In additional the LLM script stopped to recognize the gpu (previous one was working fine)

unity version: 2022.3.15f1 windows 10 pro 16gb ram ddr4 i5-7600K CPU @ 3.80GHz 3.79 GHz

Steps to reproduce

install via git url the package import the sample "simple chatbot" select a model, (i tried "Phi-3-mini-4k-instruct-q4" and "Meta-Llama-3-8B-Instruct.Q4_K_M") the result is

"Curl error 56: Recv failure: Connection was reset"

"Failed to receive data UnityEngine.Debug:LogError (object) LLMUnity.LLMClient/d__77`2<LLMUnity.MultiChatResult, string>:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLMClient.cs:687) UnityEngine.UnitySynchronizationContext:ExecuteTasks ()"

LLMUnity version

1.2.8

Operating System

None

amakropoulos commented 1 month ago

Hi, do you have any other llamafile processes running on the same port eg from another project? You can check it by searching for llamafile at the task manager.

Gabri94x commented 1 month ago

i tested using the terminal tasklist | findstr /I llamafile and i have no any running llamafile. It prints only when playing the working one app (1.2.7) i made the test using two editor, the first one is the older one, i play it and it works fine. i stopped the run. i played the other one project running the 1.2.8 with the same model (llama3) and it throws the errror. i stopped the run and tried again the older versione, it keep working fine. i played the same sample scene "ChatBot" with the same model "Meta-Llama-3-8B-Instruct.Q4_K_M.gguf" the only difference is one is running the 1.2.7 from asset store, the second one is running the 1.2.8 installed from giturl

amakropoulos commented 1 month ago

Have you set the LLM variable asynchronous start? Do you see any other errors?

Gabri94x commented 1 month ago

i just tried to set the async option on, i get "no server found". i'm using the ChatBot sample of the plugin

Gabri94x commented 1 month ago

this is the log of the editor without async start

Reloading assemblies for play mode.
Reloading assemblies after forced synchronous recompile.
[Licensing::Client] Successfully resolved entitlements
Begin MonoManager ReloadAssembly
- Loaded All Assemblies, in  0.546 seconds
Refreshing native plugins compatible for Editor in 1.97 ms, found 3 plugins.
Native extension for UWP target not found
Native extension for WindowsStandalone target not found
Native extension for Android target not found
Native extension for WebGL target not found
[Licensing::Client] Successfully resolved entitlement details
[MODES] ModeService[none].Initialize
[MODES] ModeService[none].LoadModes
[MODES] Loading mode Default (0) for mode-current-id-test-llm-unity
Mono: successfully reloaded assembly
- Finished resetting the current domain, in  2.199 seconds
Domain Reload Profiling: 2745ms
    BeginReloadAssembly (412ms)
        ExecutionOrderSort (0ms)
        DisableScriptedObjects (56ms)
        BackupInstance (0ms)
        ReleaseScriptingObjects (0ms)
        CreateAndSetChildDomain (196ms)
    RebuildCommonClasses (36ms)
    RebuildNativeTypeToScriptingClass (12ms)
    initialDomainReloadingComplete (33ms)
    LoadAllAssembliesAndSetupDomain (52ms)
        LoadAssemblies (128ms)
        RebuildTransferFunctionScriptingTraits (0ms)
        AnalyzeDomain (13ms)
            TypeCache.Refresh (6ms)
                TypeCache.ScanAssembly (0ms)
            ScanForSourceGeneratedMonoScriptInfo (0ms)
            ResolveRequiredComponents (5ms)
    FinalizeReload (2199ms)
        ReleaseScriptCaches (0ms)
        RebuildScriptCaches (0ms)
        SetupLoadedEditorAssemblies (673ms)
            LogAssemblyErrors (0ms)
            InitializePlatformSupportModulesInManaged (27ms)
            SetLoadedEditorAssemblies (4ms)
            RefreshPlugins (0ms)
            BeforeProcessingInitializeOnLoad (91ms)
            ProcessInitializeOnLoadAttributes (502ms)
            ProcessInitializeOnLoadMethodAttributes (40ms)
            AfterProcessingInitializeOnLoad (8ms)
            EditorAssembliesLoaded (0ms)
        ExecutionOrderSort2 (0ms)
        AwakeInstancesAfterBackupRestoration (1026ms)
Asset Pipeline Refresh (id=8a1c6106e3e4099468465b378676aec4): Total: 2.946 seconds - Initiated by StopAssetImportingV2(ForceSynchronousImport | ForceDomainReload)
    Summary:
        Imports: total=0 (actual=0, local cache=0, cache server=0)
        Asset DB Process Time: managed=0 ms, native=153 ms
        Asset DB Callback time: managed=12 ms, native=0 ms
        Scripting: domain reloads=1, domain reload time=2779 ms, compile time=0 ms, other=0 ms
        Project Asset Count: scripts=3711, non-scripts=589
        Asset File Changes: new=0, changed=0, moved=0, deleted=0
        Scan Filter Count: 0
    InvokeBeforeRefreshCallbacks: 0.307ms
    ApplyChangesToAssetFolders: 0.103ms
    Scan: 0.001ms
    OnSourceAssetsModified: 0.001ms
    GetAllGuidsForCategorization: 0.783ms
    CategorizeAssets: 17.470ms
    ImportOutOfDateAssets: 2209.210ms (2206.687ms without children)
        CompileScripts: 0.019ms
        CollectScriptTypesHashes: 0.000ms
        ReloadNativeAssets: 0.000ms
        UnloadImportedAssets: 0.222ms
        EnsureUptoDateAssetsAreRegisteredWithGuidPM: 0.998ms
        InitializingProgressBar: 0.001ms
        PostProcessAllAssetNotificationsAddChangedAssets: 0.001ms
        OnDemandSchedulerStart: 1.282ms
    PostProcessAllAssets: 12.384ms
    GatherAllCurrentPrimaryArtifactRevisions: 0.626ms
    UnloadStreamsBegin: 0.172ms
    PersistCurrentRevisions: 0.144ms
    UnloadStreamsEnd: 0.019ms
    GenerateScriptTypeHashes: 0.212ms
    Untracked: 704.382ms
killing existing server with 14680: D:./test-llm-unity/Assets/StreamingAssets/llamafile-0.8.6.exe
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
LLMUnity.LLMUnitySetup:KillServerAfterUnityCrash (string) (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLMUnitySetup.cs:268)
LLMUnity.LLM:KillServersAfterUnityCrash () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:211)
LLMUnity.LLM/<Awake>d__47:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:225)
System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start<LLMUnity.LLM/<Awake>d__47> (LLMUnity.LLM/<Awake>d__47&)
LLMUnity.LLM:Awake ()

(Filename: ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLMUnitySetup.cs Line: 268)

Curl error 7: Failed to connect to localhost port 13333 after 2231 ms: Couldn't connect to server
Server command: D:./test-llm-unity/Assets/StreamingAssets/llamafile-0.8.6.exe  --port 13333 -m D:./test-llm-unity/Assets/StreamingAssets/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -c 512 -b 512 --log-disable --nobrowser -np 1 -ngl 9999
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
LLMUnity.LLM:RunServerCommand (string,string) (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:318)
LLMUnity.LLM/<StartLLMServer>d__55:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:365)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start<LLMUnity.LLM/<StartLLMServer>d__55> (LLMUnity.LLM/<StartLLMServer>d__55&)
LLMUnity.LLM:StartLLMServer ()
LLMUnity.LLM/<Awake>d__47:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:227)
System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start<LLMUnity.LLM/<Awake>d__47> (LLMUnity.LLM/<Awake>d__47&)
LLMUnity.LLM:Awake ()

(Filename: ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs Line: 318)

GPU failed, fallback to CPU
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
LLMUnity.LLM/<StartLLMServer>d__55:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:382)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start<LLMUnity.LLM/<StartLLMServer>d__55> (LLMUnity.LLM/<StartLLMServer>d__55&)
LLMUnity.LLM:StartLLMServer ()
LLMUnity.LLM/<Awake>d__47:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:227)
System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start<LLMUnity.LLM/<Awake>d__47> (LLMUnity.LLM/<Awake>d__47&)
LLMUnity.LLM:Awake ()

(Filename: ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs Line: 382)

Server command: D:./test-llm-unity/Assets/StreamingAssets/llamafile-0.8.6.exe  --port 13333 -m D:./test-llm-unity/Assets/StreamingAssets/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf -c 512 -b 512 --log-disable --nobrowser -np 1 -ngl 0
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
LLMUnity.LLM:RunServerCommand (string,string) (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:318)
LLMUnity.LLM/<StartLLMServer>d__55:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:385)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start<LLMUnity.LLM/<StartLLMServer>d__55> (LLMUnity.LLM/<StartLLMServer>d__55&)
LLMUnity.LLM:StartLLMServer ()
LLMUnity.LLM/<Awake>d__47:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:227)
System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start<LLMUnity.LLM/<Awake>d__47> (LLMUnity.LLM/<Awake>d__47&)
LLMUnity.LLM:Awake ()

(Filename: ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs Line: 318)

LLM Server started!
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:Log (object)
LLMUnity.LLM:CheckIfListening (string) (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs:279)
LLMUnity.LLMUnitySetup/<>c__DisplayClass1_0:<CreateProcess>b__0 (object,System.Diagnostics.DataReceivedEventArgs) (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLMUnitySetup.cs:63)
System.Diagnostics.Process:OutputReadNotifyUser (string)
System.Diagnostics.AsyncStreamReader:FlushMessageQueue ()
System.Diagnostics.AsyncStreamReader:GetLinesFromStringBuilder ()
System.Diagnostics.AsyncStreamReader:ReadBuffer (System.IAsyncResult)
System.IO.Stream/ReadWriteTask:InvokeAsyncCallback (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object)
System.IO.Stream/ReadWriteTask:System.Threading.Tasks.ITaskCompletionAction.Invoke (System.Threading.Tasks.Task)
System.Threading.Tasks.Task:FinishContinuations ()
System.Threading.Tasks.Task:FinishStageThree ()
System.Threading.Tasks.Task:FinishStageTwo ()
System.Threading.Tasks.Task:Finish (bool)
System.Threading.Tasks.Task:ExecuteWithThreadLocal (System.Threading.Tasks.Task&)
System.Threading.Tasks.Task:ExecuteEntry (bool)
System.Threading.Tasks.Task:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem ()
System.Threading.ThreadPoolWorkQueue:Dispatch ()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback ()

(Filename: ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLM.cs Line: 279)

Loaded scene 'Temp/__Backupscenes/0.backup'
    Deserialize:            3.051 ms
    Integration:            10378.319 ms
    Integration of assets:  0.003 ms
    Thread Wait Time:       0.060 ms
    Total Operation Time:   10381.434 ms
Curl error 56: Recv failure: Connection was reset
Failed to receive data
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:LogError (object)
LLMUnity.LLMClient/<PostRequest>d__77`2<LLMUnity.MultiChatResult, string>:MoveNext () (at ./Library/PackageCache/ai.undream.llm@44e5b43085/Runtime/LLMClient.cs:687)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:InvokeMoveNext (object)
System.Threading.ExecutionContext:RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
System.Runtime.CompilerServices.AsyncMethodBuilderCore/MoveNextRunner:Run ()
System.Runtime.CompilerServices.YieldAwaitable/YieldAwaiter:RunAction (object)
UnityEngine.UnitySynchronizationContext/WorkRequest:Invoke ()
UnityEngine.UnitySynchronizationContext:Exec ()
UnityEngine.UnitySynchronizationContext:ExecuteTasks ()

amakropoulos commented 1 month ago

Does the chat work anyway despite the error? Can you try using another port if not e.g. 13334? You can join Discord if you'd like to have a debugging session tomorrow

Gabri94x commented 1 month ago

@amakropoulos thank you for your help. tomorrow i will try and join the discord 👍

amakropoulos commented 1 month ago

thank you for the debugging today :star:!

For anyone observing similar behavior, the issue is related to llamafile:

Fallback to CPU without compiling GPU support is now fixed in v1.2.9

undreamai / LLMUnity