search

watson-developer-cloud / unity-sdk

:video_game: Unity SDK to use the IBM Watson services.
Apache License 2.0
571 stars 208 forks source link

[speech-to-text] Connection immediately closed in Android build, works fine in editor #691

Open jpingen opened 2 years ago

jpingen commented 2 years ago

Hi, I'm having issues with getting speech-to-text to work in an Android build. Everything works fine in the Unity Editor.

Unity version: 2021.3.6f1 Watson unity-sdk: v6.0.0 IBM unity-sdk-core: v1.2.4 Scripting backend: IL2CPP Api compatibility level: .NET Framework (I tried switching to .NET Standard 2.1, but it results in a compilation error IBMSdkCore\Connection\RESTConnector.cs(668,24): error CS0656: Missing compiler required member 'Microsoft.CSharp.RuntimeBinder.CSharpArgumentInfo.Create')

So everything works fine in the Editor, but when I run the Android build, the websocket connection is immediately closed. Similar to https://github.com/watson-developer-cloud/unity-sdk/issues/542, but I'm using the Frankfurt endpoint (eu-de). Edit: also tested on London (eu-gb) and Dallas (us-south), problem still the same.

Logs:

: RESTConnector: URL: https://iam.cloud.ibm.com/identity/token
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: RESTConnector: URL: https://iam.cloud.ibm.com/identity/token
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: RESTConnector: URL: https://api.eu-de.assistant.watson.cloud.ibm.com/v2/assistants/[redacted-assistant-id]/sessions?version=2021-11-27
: IBM.Cloud.SDK.Connection.<ProcessRequestQueue>d__33:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: [07-29-2022 08:55:14][WSConnector][WARNING] No case for URL for wss://. Replacing https:// with wss://.
: IBM.Cloud.SDK.Debug.DebugReactor:ProcessLog(LogRecord)
: IBM.Cloud.SDK.LogSystem:ProcessLog(LogRecord)
: IBM.Cloud.SDK.Log:Warning(String, String, Object[])
: IBM.Cloud.SDK.Connection.WSConnector:FixupURL(String)
: IBM.Cloud.SDK.Connection.WSConnector:CreateConnector(Authenticator, String, String, String)
: IBM.Watson.SpeechToText.V1.SpeechToTextService:CreateListenConnector()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:StartListening(OnRecognize, OnRecognizeSpeaker)
: 
: SpeechToText.CreateListenConnector(): Created listen socket. Model: nl-NL_Telephony, parsedParams:
: IBM.Watson.SpeechToText.V1.SpeechToTextService:CreateListenConnector()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:StartListening(OnRecognize, OnRecognizeSpeaker)
:
: WSConnector.Send(): Sending TextMessage message: {"action":"start","content-type":"audio/l16;rate=16000;channels=1;","inactivity_timeout":-1,"interim_results":true,"max_alternatives":1,"profanity_filter":false,"smart_formatting":true,"speaker_labels":false,"timestamps":true,"word_confidence":true,"speech_detector_sensitivity":0,5,"background_audio_suppression":0,"processing_metrics":false}
: IBM.Cloud.SDK.Connection.WSConnector:Send(Message, Boolean)
: IBM.Watson.SpeechToText.V1.SpeechToTextService:SendStart()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListen(AudioData)
:
: SpeechToText.SendStart(): SendStart() with the following params: {"action":"start","content-type":"audio/l16;rate=16000;channels=1;","inactivity_timeout":-1,"interim_results":true,"max_alternatives":1,"profanity_filter":false,"smart_formatting":true,"speaker_labels":false,"timestamps":true,"word_confidence":true,"speech_detector_sensitivity":0,5,"background_audio_suppression":0,"processing_metrics":false}
: IBM.Watson.SpeechToText.V1.SpeechToTextService:SendStart()
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListen(AudioData)
:
: WSConnector.OnWSClose(): Reason: An exception has occurred while receiving. - Code: 1006 - WasClean: False
: IBM.Cloud.SDK.Connection.WSConnector:OnWSClose(Object, CloseEventArgs)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocket:close(PayloadData, Boolean, Boolean, Boolean)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp.<>c__DisplayClass57_0:<ReadBytesAsync>b__0(IAsyncResult)
: System.Threading.Tasks.TaskToApm:Begin(Task, AsyncCallback, Object)
: IBM.Cloud.SDK.Plugins.WebSocketSharp
: SpeechToText.OnListenClosed(): OnListenClosed(), State = DISCONNECTED
: IBM.Watson.SpeechToText.V1.SpeechToTextService:OnListenClosed(WSConnector)
: IBM.Cloud.SDK.Connection.<ProcessReceiveQueue>d__49:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
:
: SpeechToText.KeepAlive(): KeepAlive exited.
: IBM.Watson.SpeechToText.V1.<KeepAlive>d__242:MoveNext()
: IBM.Cloud.SDK.Utilities.Routine:MoveNext()
: UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

I have also enabled logging on the Websocket in WSConnector:

29-7-2022 10:55:15|Warn |WSConnector.SendMessages|The current logging level has been changed to Trace.
29-7-2022 10:55:15|Debug|WebSocket.sendHandshakeRequest|A request to the server:
                         GET /v1/recognize?model=nl-NL_Telephony HTTP/1.1
                         User-Agent: websocket-sharp/1.0
                         Host: api.eu-de.speech-to-text.watson.cloud.ibm.com
                         Upgrade: websocket
                         Connection: Upgrade
                         Sec-WebSocket-Key: [redacted]
                         Sec-WebSocket-Version: 13
                         Authorization: Bearer [redacted]
29-7-2022 10:55:16|Debug|WebSocket.sendHandshakeRequest|A response to this request:
                         HTTP/1.1 101 Switching Protocols
                         sec-websocket-accept: [redacted]
                         Content-Type: application/octet-stream
                         Server: watson-gateway
                         strict-transport-security: max-age=31536000; includeSubDomains;
                         x-dp-watson-tran-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         x-request-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         x-global-transaction-id: 36a4772e-ae7f-4e2a-a996-1e37e6931fd0
                         X-EdgeConnect-MidMile-RTT: 0
                         X-EdgeConnect-Origin-MEX-Latency: 534
                         Date: Fri, 29 Jul 2022 08:55:15 GMT
                         Upgrade: websocket
                         Connection: Upgrade
29-7-2022 10:55:17|Fatal|TaskToApm.Begin|IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.TaskToApm.Begin (System.Threading.Tasks.Task task, System.AsyncCallback callback, System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at IBM.Cloud.SDK.Plugins.WebSocketSharp.Ext+<>c__DisplayClass57_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.MobileAuthenticatedStream.StartOperation (Mono.Net.Security.MobileAuthenticatedStream+OperationType type, Mono.Net.Security.AsyncProtocolRequest asyncRequest, System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.StartOperation (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.AsyncProtocolRequest.InnerRead (System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at Mono.Net.Security.MobileAuthenticatedStream.InnerRead (System.Boolean sync, System.Int32 requestedSize, System.Threading.CancellationToken cancellationToken) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task.FinishContinuations () [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[TResult].<.cctor>b__4_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Net.Sockets.Socket+<>c.<.cctor>b__367_10 (System.IAsyncResult ares) [0x00000] in <00000000000000000000000000000000>:0 
                           at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00000] in <00000000000000000000000000000000>:0 
29-7-2022 10:55:17|Trace|TaskToApm.Begin|Begin closing the connection.
29-7-2022 10:55:17|Debug|WebSocket.close|Was clean?: False
                           sent: False
                           received: False
29-7-2022 10:55:17|Trace|TaskToApm.Begin|End closing the connection.
29-7-2022 10:55:17|Info |ExecutionContext.RunInternal|The connection has already been closed.
jpingen commented 2 years ago

It appears to run fine as an Android app for mobile. However, when building an Android apk for the Oculus Quest (same setup, only added an XR rig so it runs on VR) the problem occurs.

@mediumTaj ?