mamoonraja
commented
4 years ago @DougLilliequist Can you provide more info about the issue, i.e, steps to reproduce (more info about your script if possible), unity version, and sdk version?
Closed DougLilliequist closed 4 years ago
mamoonraja
commented
4 years ago @DougLilliequist Can you provide more info about the issue, i.e, steps to reproduce (more info about your script if possible), unity version, and sdk version?
DougLilliequist
commented
4 years ago @DougLilliequist Can you provide more info about the issue, i.e, steps to reproduce (more info about your script if possible), unity version, and sdk version?
Hey @mamoonraja ! I'll do so as soon as I'm back at the office on monday!
DougLilliequist
commented
4 years ago @mamoonraja Here is a write-up of sdk's, platforms etc:
Unity version: 2019.1.8f1 Watson sdk version: 3.5.0 Watson sdk core version: 0.3.0
As for recreating the error, the only thing required is to say (or write seeing as a string is just being passed) the required entity value that will then trigger an animation. Again: this error only happens on iOS (I don't have access to an up-to-date Android device), but everything works well on desktop!
Code used:
public class VoiceCommandDemo : MonoBehaviour
{
#region WATSON SPEECH TO TEXT KEYS
#region PLEASE SET THESE VARIABLES IN THE INSPECTOR
[Space(10)]
[Header("Speech To Text Keys")]
[Tooltip("The service URL (optional). This defaults to \"https://stream.watsonplatform.net/speech-to-text/api\"")]
[SerializeField]
private string _serviceUrl;
[Tooltip("Text field to display the results of streaming.")]
public Text ResultsField;
[Header("IAM Authentication")]
[Tooltip("The IAM apikey.")]
[SerializeField]
private string speechToTextApiKey;
[Header("Parameters")]
// https://www.ibm.com/watson/developercloud/speech-to-text/api/v1/curl.html?curl#get-model
[Tooltip("The Model to use. This defaults to en-US_BroadbandModel")]
[SerializeField]
private string _recognizeModel;
#endregion
#endregion
#region WATSON ASSISTANT KEYS
#region PLEASE SET THESE VARIABLES IN THE INSPECTOR
[Space(10)]
[Header("Assistant Keys")]
[Tooltip("The IAM apikey.")]
[SerializeField]
private string assistantApiKey;
[Tooltip("The service URL (optional). This defaults to \"https://gateway.watsonplatform.net/assistant/api\"")]
[SerializeField]
private string serviceUrl;
[Tooltip("The version date with which you would like to use the service in the form YYYY-MM-DD.")]
[SerializeField]
private string versionDate;
[Tooltip("The assistantId to run the example.")]
[SerializeField]
private string assistantId;
#endregion
#endregion
#region assistant and speech to text recognizer
private AssistantService assistant;
private SpeechToTextService speechToText;
#endregion
#region speech to text params
private int _recordingRoutine = 0;
private string _microphoneID = null;
private AudioClip _recording = null;
private int _recordingBufferSize = 1;
private int _recordingHZ = 22050;
#endregion
#region assistant params
private bool createSessionTested = false;
private bool messageTested0 = false;
// private bool messageTested1 = false;
// private bool messageTested2 = false;
// private bool messageTested3 = false;
// private bool messageTested4 = false;
private bool deleteSessionTested = false;
private string sessionId;
#endregion
#region delegates
public delegate void EmitCapturedWord(string word);
public EmitCapturedWord emitCapturedWord;
#endregion
void Start()
{
LogSystem.InstallDefaultReactors();
Runnable.Run(CreateServices());
}
private IEnumerator CreateServices()
{
// Create credential and instantiate speech to text service
Credentials speechToTextCredentials = null;
if (string.IsNullOrEmpty(speechToTextApiKey))
{
throw new IBMException("Plesae provide IAM ApiKey for the service.");
} else {
// Authenticate using iamApikey
TokenOptions tokenOptions = new TokenOptions()
{
IamApiKey = speechToTextApiKey
};
speechToTextCredentials = new Credentials(tokenOptions, _serviceUrl);
// Wait for tokendata
while (!speechToTextCredentials.HasIamTokenData())
yield return null;
speechToText = new SpeechToTextService(speechToTextCredentials);
speechToText.StreamMultipart = true;
// StartRecording();
}
// Create credential and instantiate assistant service
Credentials assistantCredentials = null;
if (string.IsNullOrEmpty(assistantApiKey))
{
throw new IBMException("Plesae provide IAM ApiKey for the service.");
} else {
// Authenticate using iamApikey
TokenOptions tokenOptions = new TokenOptions()
{
IamApiKey = assistantApiKey
};
assistantCredentials = new Credentials(tokenOptions, serviceUrl);
// Wait for tokendata
while (!assistantCredentials.HasIamTokenData())
yield return null;
Debug.Log(versionDate);
assistant = new AssistantService(versionDate, assistantCredentials);
assistant.VersionDate = versionDate;
Active = true;
assistant.CreateSession(OnCreateSession, assistantId);
while (!createSessionTested)
{
yield return null;
}
}
StartRecording();
}
public bool Active
{
get { return speechToText.IsListening; }
set
{
if (value && !speechToText.IsListening)
{
speechToText.RecognizeModel = (string.IsNullOrEmpty(_recognizeModel) ? "en-US_BroadbandModel" : _recognizeModel);
speechToText.DetectSilence = true;
speechToText.EnableWordConfidence = true;
speechToText.EnableTimestamps = true;
speechToText.SilenceThreshold = 0.0f; //setting this to 0 keeps the session alive
speechToText.MaxAlternatives = 1;
speechToText.EnableInterimResults = true;
speechToText.OnError = OnError;
speechToText.InactivityTimeout = -1;
speechToText.ProfanityFilter = false;
speechToText.SmartFormatting = true;
speechToText.SpeakerLabels = false;
speechToText.WordAlternativesThreshold = null;
speechToText.StartListening(OnRecognize, OnRecognizeSpeaker);
}
else if (!value && speechToText.IsListening)
{
speechToText.StopListening();
}
}
}
private void StartRecording()
{
if (_recordingRoutine == 0)
{
UnityObjectUtil.StartDestroyQueue();
_recordingRoutine = Runnable.Run(RecordingHandler());
}
}
private void StopRecording()
{
if (_recordingRoutine != 0)
{
Microphone.End(_microphoneID);
Runnable.Stop(_recordingRoutine);
_recordingRoutine = 0;
}
}
private void OnError(string error)
{
Active = false;
Log.Debug("ExampleStreaming.OnError()", "Error! {0}", error);
}
private IEnumerator RecordingHandler()
{
Log.Debug("ExampleStreaming.RecordingHandler()", "devices: {0}", Microphone.devices);
_recording = Microphone.Start(_microphoneID, true, _recordingBufferSize, _recordingHZ);
yield return null; // let _recordingRoutine get set..
if (_recording == null)
{
StopRecording();
yield break;
}
bool bFirstBlock = true;
int midPoint = _recording.samples / 2;
float[] samples = null;
while (_recordingRoutine != 0 && _recording != null)
{
int writePos = Microphone.GetPosition(_microphoneID);
if (writePos > _recording.samples || !Microphone.IsRecording(_microphoneID))
{
Log.Error("ExampleStreaming.RecordingHandler()", "Microphone disconnected.");
StopRecording();
yield break;
}
if ((bFirstBlock && writePos >= midPoint)
|| (!bFirstBlock && writePos < midPoint))
{
// front block is recorded, make a RecordClip and pass it onto our callback.
samples = new float[midPoint];
_recording.GetData(samples, bFirstBlock ? 0 : midPoint);
AudioData record = new AudioData();
record.MaxLevel = Mathf.Max(Mathf.Abs(Mathf.Min(samples)), Mathf.Max(samples));
record.Clip = AudioClip.Create("Recording", midPoint, _recording.channels, _recordingHZ, false);
record.Clip.SetData(samples, 0);
speechToText.OnListen(record);
bFirstBlock = !bFirstBlock;
}
else
{
// calculate the number of samples remaining until we ready for a block of audio,
// and wait that amount of time it will take to record.
int remaining = bFirstBlock ? (midPoint - writePos) : (_recording.samples - writePos);
float timeRemaining = (float)remaining / (float)_recordingHZ;
yield return new WaitForSeconds(timeRemaining);
}
}
yield break;
}
private void OnRecognize(SpeechRecognitionEvent result)
{
if (result != null && result.results.Length > 0)
{
foreach (var res in result.results)
{
foreach (var alt in res.alternatives)
{
string text = string.Format("{0} ({1}, {2:0.00})\n", alt.transcript, res.final ? "Final" : "Interim", alt.confidence);
// Log.Debug("ExampleStreaming.OnRecognize()", text);
ResultsField.text = text;
if (res.final && alt.confidence > 0)
{
string transcript_text = alt.transcript;
Debug.Log("Result: " + transcript_text + " Confidence: " + alt.confidence);
var input = new MessageInput()
{
Text = transcript_text,
Options = new MessageInputOptions()
{
ReturnContext = false
}
};
Debug.Log("Input to Assistant:" + input.Text);
assistant.Message(OnMessage, assistantId, sessionId, input);
}
}
if (res.keywords_result != null && res.keywords_result.keyword != null)
{
foreach (var keyword in res.keywords_result.keyword)
{
Log.Debug("ExampleStreaming.OnRecognize()", "keyword: {0}, confidence: {1}, start time: {2}, end time: {3}", keyword.normalized_text, keyword.confidence, keyword.start_time, keyword.end_time);
}
}
if (res.word_alternatives != null)
{
foreach (var wordAlternative in res.word_alternatives)
{
Log.Debug("ExampleStreaming.OnRecognize()", "Word alternatives found. Start time: {0} | EndTime: {1}", wordAlternative.start_time, wordAlternative.end_time);
foreach (var alternative in wordAlternative.alternatives)
Log.Debug("ExampleStreaming.OnRecognize()", "\t word: {0} | confidence: {1}", alternative.word, alternative.confidence);
}
}
}
}
}
void OnMessage(DetailedResponse<MessageResponse> resp, IBMError error)
{
if (resp != null && resp.Result.Output.Intents.Count != 0 )
{
string intent = resp.Result.Output.Intents[0].Intent;
Debug.Log("Intent: " + intent);
string currentMat = null;
string currentScale = null;
// string direction = null;
if (intent == "heading_to")
{
foreach (RuntimeEntity entity in resp.Result.Output.Entities)
{
Debug.Log("entityType: " + entity.Entity + " , value: " + entity.Value);
if(entity.Entity == "destination") {
emitCapturedWord(entity.Value);
if(entity.Value == "beach") {
Debug.Log("HAVE FUN!");
}
}
}
}
}
else
{
Debug.Log("Failed to invoke OnMessage();");
}
}
private void OnRecognizeSpeaker(SpeakerRecognitionEvent result)
{
if (result != null)
{
foreach (SpeakerLabelsResult labelResult in result.speaker_labels)
{
Log.Debug("ExampleStreaming.OnRecognizeSpeaker()", string.Format("speaker result: {0} | confidence: {3} | from: {1} | to: {2}", labelResult.speaker, labelResult.from, labelResult.to, labelResult.confidence));
}
}
}
private void OnCreateSession(DetailedResponse<SessionResponse> response, IBMError error)
{
Debug.Log(response);
Log.Debug("ExampleAssistantV2.OnCreateSession()", "Session: {0}", response.Result.SessionId);
sessionId = response.Result.SessionId;
createSessionTested = true;
}
private void OnDeleteSession(DetailedResponse<object> response, IBMError error)
{
Log.Debug("ExampleAssistantV2.OnDeleteSession()", "Session deleted.");
deleteSessionTested = true;
}
}Hope this helps and let me know if you need any more info!
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has had no recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
mediumTaj
commented
4 years ago Is this still an issue?
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has had no recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
realitygarage
commented
4 years ago getting similar result on Android running on Oculus GO, but android running on samsung tablet works fine - seems to be something with newtonsoft json conversion choking
realitygarage
commented
4 years ago not completely solved yet - but looks to be issue with AOT method - https://docs.unity3d.com/Manual/ScriptingRestrictions.html
davidjbain
commented
4 years ago I had the same error on WebGL and wanted to share my workaround. I ended up switching the newtonsoft.json plugin in the sdk with this one so I could use the AotHelper.
https://github.com/jilleJr/Newtonsoft.Json-for-Unity/wiki/Fix-AOT-using-AotHelper
I had to add two types to stop getting the errors
using Newtonsoft.Json.Utilities;
using UnityEngine;
public class AotTypeEnforcer : MonoBehaviour
{
public void Awake()
{
AotHelper.EnsureList<long>();
AotHelper.EnsureDictionary<string, string>();
}
}I hope this helps someone
realitygarage
commented
4 years ago yeah - Thanks - I guess I had also answered with more details in:
https://github.com/watson-developer-cloud/unity-sdk/issues/625#issuecomment-616550784
BTW - Do you have a solution for using their watson services SDK for Unity WebGL (which they say the do not support ?)
davidjbain
commented
4 years ago Thanks, sorry I missed that thread. I do not have anything uploaded publicly yet but I was able to recently get the assistant V2 working on WebGL thanks to Taj. You have to use the basicAutheticator with the username and password to avoid a cors issue, and in AssistantService.cs I commented out the Common.GetSdkHeaders headers parts (I was getting "old value is the empty string" error in the build). I think that is all I really had to do to get the example scene working. If you are on the slack channel it is the latest thread right now.
I made a script where I pass my speech to text data to Watson assistant, which is working fine on desktop! But when I run the same project on iOS and say the required words for triggering my desired animation, I get an error refering to a missing object reference.
Here is the error I'm logging when I say the required words as mentioned: