[speech-to-text] Example Streaming KeepAlive exits after 30 seconds

[idialab]

Steps to reproduce

1. Open ExampleStreaming scene in Unity 2019.1.0f2.
2. Press play and wait ~30 seconds.

Expected behavior

It is my understanding that WSConnector.cs has a built in KeepAlive coroutine that sends a highHat.mp3 clip to the speech-to-text service at a defined interval in order to keep the connection alive.

Actual behavior

Either it is not sending often enough or something else is closing the connection. In any case, the KeepAlive coroutine is failing to keep the connection alive even through one session timeout.

Versions Unity 2019.1.0f2 IBM Watson SDK for Unity 3.4.1 IBM SDK Core 0.2.0

[mamoonraja]

@idialab Thanks a lot for opening the issue, I will look into it and get back to you asap.

[idialab]

@mamoonraja Were you ever able to replicate this issue? I just tried the example scene again and it seems to be working as intended now.

[mamoonraja]

@idialab I was not able to recreate, but in all fairness, I tried it twice only. Would you like to keep the open issue still? you can always re-open if you see the issue again. It can be multiple reasons why you saw the issue, it can be because of what kind of mic you were using, and/or WebSocket was not connected.

[idialab]

Actually, I may not have waited long enough when I said it was working. Though it does usually get one KeepAlive out, it pretty consistently times out at ~30 seconds. Here are the results from my console:

[mamoonraja]

Hey! I just debugged the issue. So the way streaming example is set up, the connection has been closed after 30 seconds of complete silence. If you change DetectSilence to false here, you should be fine. Please let me know if that works for you.

[idialab]

That seems to have done the trick. Session is staying alive. Though it does make me wonder what the KeepAlive is contributing. If DetectSilence is set to false, then I'm always sending audio (and getting charged for it) and that in itself should keep the session alive.

If the KeepAlive doesn't keep a session alive without me sending audio, what use is it?

[daniellovell]

I'm having the same issue and can consistently reproduce the Session Timed out message in the unedited ExampleStreaming scene in Unity 2018.3.5f1 by starting the game and waiting 30-60s.

I also share the concern that @idialab mentioned that turning DetectSilence to false will result in more billed time.

[daniellovell]

@mamoonraja I'm gonna bump this issue again since I have some more findings and an easy way to reproduce: Tested on Unity 2018.3 and Unity 2019.2, so it seems (at first glance) to be agnostic of Unity version.

If you Invoke("StopRecording", 1f) in the ExampleStreaming.cs Start function, you hand the responsibility of keeping the session open to the KeepAlive routine. You will see, without fail, that it times out at ~30s over and over.

I suspect that sending that highHat audio is simply not enough to keep the session open. Is there no way to communicate to the backend's watchdog that you are still there other than continually sending audio? This seems to be a larger issue with the API.

The only way to keep the session open without continually sending audio (not an option in most applications) is to call Start() in ExampleStreaming whenever the session times out and re-open the socket.

[mamoonraja]

@daniellovell Thanks for the ping again, apologies for the late reply as I was out of office. I am looking into the issue with the KeepAlive and will get back to you today.

[idialab]

To @daniellovell and those having the same issue with the KeepAlive, I have developed a workaround for the time being that is automatic and seems reliable:

private IEnumerator CreateService()
{
    ... //The default CreateService coroutine code with DetectSilence to true
    StartCoroutine(ReviveService());
}

private IEnumerator ReviveService()
{
    yield return new WaitWhile(() => _service.IsListening);
    Debug.Log("[SpeechToText] Reviving Service");
    Runnable.Run(CreateService());
}

I initially tried to just re-create the service in OnError as I would get a Session time out error, but found that I would not reliably get the error after ~30 of them. I tested the above solution overnight and it was still working this morning after 2500+ revivals.

[mamoonraja]

@daniellovell @idialab We have resolved issues around loading audio when sending keep alive here. If you still see the issue, you can either use the workaround mentioned above or you can use a longer audio file for keep-alive to avoid session from timing out.

[mediumTaj]

:tada: This issue has been resolved in version 4.0.1 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: