search

Azure-Samples / cognitive-services-speech-sdk

Sample code for the Microsoft Cognitive Services Speech SDK
MIT License
2.97k stars 1.87k forks source link

Xamarin SDK #363

Closed SOliasS closed 4 years ago

SOliasS commented 5 years ago

I was wondering if there is any news about a Xamarin.Android/ IOS version. It was teased in this request #76, but since then a year has passed and there is still no info about a xamarin version.

An alternativ would be a guide how to use it with Binding Libraries, which is possible but definitly not optimal.

jhakulin commented 5 years ago

Thanks for the question, we are working on Xamarin support as we speak and planning to release support for UWP, Android and iOS in the coming September release.

There is a ways to build your plugin using current released bits, let me know if you need more instructions on that.

SOliasS commented 5 years ago

The release in September sounds realy nice. At the moment i`m working with a Java Binding Library which is kind of a pain.

SOliasS commented 5 years ago

@jhakulin I would like to get the Enities from my queries but unfortunately i`m not able to find where\how they are returned. Can you help?

image

jhakulin commented 5 years ago

Currently for Xamarin UWP or Android, you could also build plugin using C# bindings which may be easier. See Android as an example below:

  1. Get C# dll from https://www.nuget.org/packages/Microsoft.CognitiveServices.Speech/ Extract nuget package and locate “Microsoft.CognitiveServices.Speech.csharp.dll” under microsoft.cognitiveservices.speech.1.6.0\runtimes\linux-x64\lib\netstandard2.0
  2. Get .aar package from your users directory (which is configured for Android Studio, normally C:\Users) after you have built following sample app on Android Studio https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstart-java-android Name of the .aar package you should look for is “client-sdk-1.6.0.aar” Extract .aar package and locate “libMicrosoft.CognitiveServices.Speech.core.so” of your required flavor
  3. Integrate Microsoft.CognitiveServices.Speech.csharp.dll and libMicrosoft.CognitiveServices.Speech.core.so into your Xamarin app project

Anyways we are working on this for the coming release so that you can directly use .nuget package in your Xamarin project.

SOliasS commented 5 years ago

Hey @jhakulin is there any news when the release will be finished?

I was also wondering if something like "custom wake-word" will also eventually be available in the xamarin sdk?

jhakulin commented 5 years ago

Thanks for feedback,

Plan is to release support for Xamarin UWP, Android and iOS this week.

Keyword recognizer is currently supported officially only in https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-devices-sdk. We are working on to support keyword recognizer in future Xamarin SDK releases.

SOliasS commented 5 years ago

Is it already out/where can i find it?

jhakulin commented 5 years ago

Apologies for waiting, we are still working on to finalize final bits for the release, release should be out at the beginning of coming week.

jhakulin commented 5 years ago

@SOliasS 1.7.0 release is now out and Xamarin support is released as beta version. Let us know any issues with it.

SOliasS commented 5 years ago

@jhakulin Nice. But where can i find it? ;) The documentation doesn´t look like its updated yet. Or do you have a little example how to use it?

jhakulin commented 5 years ago

Docs are being updated now, you can find Xamarin support from Microsoft.CognitiveServices.Speech nuget for UWP, Android, and iOS. Also check quickstart documentation here https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/quickstart/csharp-xamarin

SOliasS commented 5 years ago

Everything is working as expected so far. @jhakulin You guys did an amazing job, thanks.

One thing i was wondering... Is it is possible to get the entities or the probability of a result back? https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-concept-entity-types If you target the API you can get them but in the SDK i couldn't find a way to access the additional informations. These informations are quite essential for working with the SDK.

jhakulin commented 5 years ago

That is great to hear, thanks for information. Let us know if issues raises.

Related to entities/probability of results, would the Intent recognition provide the functionality you need? https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-recognize-intents-from-speech-csharp

SOliasS commented 5 years ago

No... for example if i would like the user to select something: select Product number 5. i would ether have to create a Intent for every product i have... product 1 product 2 and so on, or i would have to analyse the spoken words again to look for the number(which could be 5, fifth, and so on). Both Solutions would be quite complicated and unnecessary time consuming because luis does that already for me. With entities i would get for example a List with one Entity{Number, 5} and would not have to look by myself.

jhakulin commented 5 years ago

I will check internally what can be done for the entity request. In the meantime, would you mind to create separate issue for that ?

josephsctan commented 5 years ago

Hi,

I just downloaded Microsoft.CognitiveServices.Speech 1.8.0 from nuget and tried to run the quickstart at https://github.com/Azure-Samples/cognitive-services-speech-sdk/tree/master/quickstart/csharp/xamarin in a Oreo (Android 8.1) emulator/

And getting the same issue:
speech core so not foiund

Did something regress in v 1.8.0? Is there a way to work around this issue by copying libMicrosoft.CognitiveServices.Speech.core.so into the device manually?

jhakulin commented 5 years ago

@josephsctan Thanks for the feedback, I can reproduce some issues with x86 emulator, dll is found correctly, but app does not work correctly. I will investigate more and let you know. We did our release testing with all flavors (arm64, arm32, x86), but I have seen issues with Android emulators not been reliable also in past.

Did you try with Android HW device?

josephsctan commented 5 years ago

@jhakulin (updated) I was able to test this on an Android HW device running Android 6.0 and the same problem occurs + $exception {System.DllNotFoundException: libMicrosoft.CognitiveServices.Speech.core.so assembly:<unknown assembly> type:<unknown type> member:(null) at (wrapper managed-to-native) Microsoft.CognitiveServices.Speech.Internal.SpeechConfig.speech_config_from_subscription(intptr&,string,string) at Microsoft.CognitiveServices.Speech.SpeechConfig.FromSubscription (System.String subscriptionKey, System.String region) [0x00007] in <0dbee5398e6441379e5b8b6d03d6cece>:0 at SpotOn.AzureSpeechHelper..cctor () [0x00000] in D:\_\App1\AzureHelper\AzureSpeechHelper2.cs:29 } System.DllNotFoundException

Is there a simple workaround? I can see libMicrosoft.CognitiveServices.Speech.core.so for various architectures shipped in the nuget package, but copying them to the (emulated) device doesn't seem to work.

jhakulin commented 5 years ago

Hmm, we have not seen issues with devices which use Android 8.1, we probably have not tested Android 6.0 recently.

For workaround you can try following:

  1. Remove the nuget from all projects inside the solution using VS Package Manager
  2. Add reference to C# dll from nuget's lib\monoandroid to the main project
  3. Add Android native library from nuget's \build\monoandroid\libs directory to Android specific project, look details from here: https://docs.microsoft.com/en-us/xamarin/android/platform/native-libraries
  4. And then build Android project again
jhakulin commented 5 years ago

@josephsctan did that workaround help you forward?

Related to Xamarin and Android emulator, I was experiencing a crash and created a ticket to Xamarin side for further investigation. https://github.com/xamarin/xamarin-android/issues/3989

josephsctan commented 5 years ago

Hi @jhakulin

Thanks for the workaround. I think its working but not 100% sure. When I tried it, the original error - unable to find the libMicrosoft.CognitiveServices.Speech.core.so - went away.

Now the app is continually crashing - same error as you got in xamarin/xamarin-android#3989.

jhakulin commented 5 years ago

@josephsctan We will investigate emulator issue further, please use Android HW (if feasible) for testing.

jhakulin commented 4 years ago

x86 emulator issue will be fixed in 1.10 release (ETA at the end of Feb)

Edwardfelix08 commented 4 years ago

Hi @jhakulin, We are facing the same issue, is the issue fixed?

Thanks, Edward

jhakulin commented 4 years ago

@Edwardfelix08 Which issue you you have, emulator problem ? Do you use v1.10.0 release?

josephsctan commented 4 years ago

hey there - I'm using 1.10 now and the emulator issue's gone away

Edwardfelix08 commented 4 years ago

Hi @jhakulin , We are still facing this issue.

"https://github.com/Azure-Samples/cognitive-services-speech-sdk/issues/363#issuecomment-560172449"

Thanks Edward

jhakulin commented 4 years ago

@Edwardfelix08, do you use Speech SDK 1.10.0 version and x86 Android emulator?

upsilondynamics commented 4 years ago

We are having this issue as well, on 1.13.0 latest:

Exception thrown when calling: SpeechConfig.FromSubscription

libMicrosoft.CognitiveServices.Speech.core.so assembly:<unknown assembly> type:<unknown type> member:(null)'

jhakulin commented 4 years ago

@upsilondynamics Thanks for report, I have seen occasionally this problem, but that has disappeared after clean-up/restoring nuget packages. Could you try removing bin and obj directories from the projects of your Xamarin app and also restore nuget packages for solution ?

upsilondynamics commented 4 years ago

@jhakulin Tried that and still not working:

System.DllNotFoundException: 'libMicrosoft.CognitiveServices.Speech.core.so assembly:<unknown assembly> type:<unknown type> member:(null)

upsilondynamics commented 4 years ago

I should ask, does this happen only in the simulator? @jhakulin

jhakulin commented 4 years ago

@upsilondynamics Speech SDK supports x86 version of simulator with Xamarin, you should not see problems to load speech sdk core library with that either.

Is this first project where you have integrated speech SDK with Xamarin and the issue happened or has the problem happened first time now with 1.13.0 release and worked with previous versions of Speech SDK? I would need to be able to reproduce to investigate further.

jhakulin commented 4 years ago

@upsilondynamics Also please let me know which version Visual Studio and Xamarin.Android SDK you have?

upsilondynamics commented 4 years ago

@jhakulin

Pasted my info here:

Microsoft Visual Studio Professional 2019 Version 16.7.2 VisualStudio.16.Release/16.7.2+30413.136 Microsoft .NET Framework Version 4.8.03752

Installed Version: Professional

ADL Tools Service Provider 1.0 This package contains services used by Data Lake tools

ASA Service Provider 1.0

ASP.NET and Web Tools 2019 16.7.532.28833 ASP.NET and Web Tools 2019

ASP.NET Core Razor Language Services 16.1.0.2035807+72d099b977d3a85e65fa3b0614ca8cfc803fef02 Provides languages services for ASP.NET Core Razor.

ASP.NET Web Frameworks and Tools 2019 16.7.532.28833 For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0 16.7.532.28833 Azure App Service Tools v3.0.0

Azure Data Lake Node 1.0 This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio 2.4.6000.0 Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools 16.7.532.28833 Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio 2.4.6000.0 Microsoft Azure Stream Analytics Tools for Visual Studio

Bundler & Minifier 2.8.396 Adds support for bundling and minifying JavaScript, CSS and HTML files in any project.

C# Tools 3.7.0-6.20412.3+d3c3a44a4e7ad31cc75c59be0d3df4a19ff33878 C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Extensibility Message Bus 1.2.6 (master@34d6af2) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Fabric.DiagnosticEvents 1.0 Fabric Diagnostic Events

IntelliCode Extension 1.0 IntelliCode Visual Studio Extension Detailed Info

JetBrains ReSharper 2020.2.1 Build 202.0.20200820.125348 JetBrains ReSharper package for Microsoft Visual Studio. For more information about ReSharper, visit http://www.jetbrains.com/resharper. Copyright © 2020 JetBrains, Inc.

Microsoft Azure HDInsight Azure Node 2.4.6000.0 HDInsight Node under Azure Node

Microsoft Azure Hive Query Language Service 2.4.6000.0 Language service for Hive query

Microsoft Azure Service Fabric Tools for Visual Studio 16.0 Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Stream Analytics Language Service 2.4.6000.0 Language service for Azure Stream Analytics

Microsoft Azure Stream Analytics Node 1.0 Azure Stream Analytics Node under Azure Node

Microsoft Azure Tools 2.9 Microsoft Azure Tools for Microsoft Visual Studio 2019 - v2.9.30701.1

Microsoft Continuous Delivery Tools for Visual Studio 0.4 Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager 2.1.79+ge3567815aa.RR Install client-side libraries easily to any web project

Microsoft MI-Based Debugger 1.0 Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual Studio Tools for Containers 1.1 Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Mono Debugging for Visual Studio 16.7.5 (112c7bc) Support for debugging Mono processes with Visual Studio.

NuGet Package Manager 5.7.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension 1.0 ProjectServicesPackage Visual Studio Extension Detailed Info

SQL Server Data Tools 16.0.62008.03130 Microsoft SQL Server Data Tools

Syncfusion Xamarin Project Templates 13.4.0.53 A multi-project template for building iOS, Android, and Windows apps using Syncfusion Xamarin components.

Syncfusion Xamarin Toolbox 16.2.0.19 Syncfusion Essential Studio for Xamarin Toolbox Visual Studio extension.

SyncfusionMenu Extension 1.0 SyncfusionMenu Visual Studio Extension Detailed Info

ToolWindowHostedEditor 1.0 Hosting json editor into a tool window

TypeScript Tools 16.0.20702.2001 TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 3.7.0-6.20412.3+d3c3a44a4e7ad31cc75c59be0d3df4a19ff33878 Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 10.10.0.0 for F# 4.7 16.7.0-beta.20361.3+3ef6f0b514198c0bfa6c2c09fefe41a740b024d5 Microsoft Visual F# Tools 10.10.0.0 for F# 4.7

Visual Studio Code Debug Adapter Host Package 1.0 Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Container Tools Extensions (Preview) 1.0 View, manage, and diagnose containers within Visual Studio.

Visual Studio Tools for Containers 1.0 Visual Studio Tools for Containers

Visual Studio Tools for Kubernetes 1.0 Visual Studio Tools for Kubernetes

Visual Studio Tools for Unity 4.7.1.0 Visual Studio Tools for Unity

VisualStudio.DeviceLog 1.0 Information about my package

VisualStudio.Foo 1.0 Information about my package

VisualStudio.Mac 1.0 Mac Extension for Visual Studio

Xamarin 16.7.000.440 (d16-7@358f3c6) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 16.7.0.495 (remotes/origin/d16-7@79c0c522c) Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates 16.7.85 (1bcbbdf) Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK 11.0.2.0 (d16-7/025fde9) Xamarin.Android Reference Assemblies and MSBuild support. Mono: 83105ba Java.Interop: xamarin/java.interop/d16-7@1f3388a ProGuard: Guardsquare/proguard/proguard6.2.2@ebe9000 SQLite: xamarin/sqlite/3.32.1@1a3276b Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-7@017078f

Xamarin.iOS and Xamarin.Mac SDK 13.20.2.2 (817b6f72a) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

upsilondynamics commented 4 years ago

Also noticing that,

.Recognized result returns with an empty string of text, is the emulator expected to retrieve microphone input via the computer?

Thanks

jhakulin commented 4 years ago

@upsilondynamics Are you saying the original problem got resolved and now you see issue with microphone?

upsilondynamics commented 4 years ago

@jhakulin Oddly enough, the problem is now resolved. The only issue I am seeing is, in the emulator it returns an empty string; but works when deployed to an Android device. Have yet to confirm with iOS yet, but that;s next. Will report back if I have any other issues.

jhakulin commented 4 years ago

@upsilondynamics Thanks for information,, what might resolve the original issue ? As mentioned before I have seen that also but after a while (clean-up or nuget restore or VS project relaunch) have solve the issue. There could be a configuration issue with microphone and Xamarin Android emulator,

upsilondynamics commented 4 years ago

@jhakulin I have resolved the above issues, but with iOS I am getting this error after requesting microphone access:

System.Exception Message=Could not initialize an instance of the type 'AVFoundation.AVAudioPlayer': the native 'initWithContentsOfURL:error:' method returned nil. It is possible to ignore this condition by setting ObjCRuntime.Class.ThrowOnInitFailure to false.

jhakulin commented 4 years ago

@upsilondynamics Are you using iOS simulator or device ? Does this problem happen with Speech SDK Xamarin sample ?

upsilondynamics commented 4 years ago

@jhakulin This is on the simulator itself, I haven't tried the Xamarin Sample in it's purest form yet.

Heres the status:

Android 1 - On hardware device dictates 100% and works beauttifully 2- In simulator, won't dictate; but app doesn't crash - could be due to simulator settings?

iOS 1 - On hardware device - have not yet tested but will 2 - On simulator just crashes with above exception given.

jhakulin commented 4 years ago

@upsilondynamics Do you make request for mic permission similarly as in Xamarin sample ? https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/84c6ddbcd340c2c87b68e20581c606a1c34758f0/quickstart/csharp/xamarin/helloworld/helloworld.iOS/Services/MicrophoneService.cs#L22

Xamarin sample works well for iOS device and simulator with microphone

upsilondynamics commented 4 years ago

I did yes, it seems doing this fixed the issue:

iOSMicrophoneService.cs

`
void RequestMicPermission() { ObjCRuntime.Class.ThrowOnInitFailure = false;

        var session = AVAudioSession.SharedInstance();
        session.RequestRecordPermission((granted) =>
        {
            if(tcsPermissions != null)
                tcsPermissions.TrySetResult(granted);
        });
    }

`

jhakulin commented 4 years ago

@upsilondynamics That is interesting, thanks for information, great there is a workaround.