React Native

[johnbowdenatfacet]

Just wondering if anyone has integrated this into React Native (ios and android)?

[nshmyrev]

We can take a look on this, what platform is your priority ios or android?

[johnbowdenatfacet]

We can take a look on this, what platform is your priority ios or android?

We're hoping to support both, so both equal priority.

[erksch]

Maybe I could help with this, as I have used this library for a while and already integrated the android part over the react native bridge into our react native app (same for windows). But I have no experience for iOS.

[johnbowdenatfacet]

@erksch that would be great, if we can get both ios and android done and hopefully release as a module would be good. @nshmyrev we might need a hand exposing to IOS.

[LirryPinter]

@erksch Is your usage of the react native brigde for this library somewhere on github? I'm trying to do the same but i'm fairly new to react native and vosk.

[erksch]

@LirryPinter I could make a Gist for the android native module. For iOS please contact @nshmyrev per mail.

[LirryPinter]

@erksch Thank you a lot! I will await the Gist, and will send the email

[erksch]

@LirryPinter

Here is the Gist for the module: https://gist.github.com/erksch/f3d46b478b3912a00a4dc25726d0260c

Things to note:

• Don't forget to write an appropriate package file (refer to the native module docs for that)
• Enable Kotlin for your project, the module is written in Kotlin
• Include Klaxon (for JSON parsing) and VOSK as a dependency to your android project:

  dependencies { 
  implementation 'com.beust:klaxon:5.0.1'
  implementation 'com.alphacep:vosk-android:0.3.12'
  }

• The module works by sending events to JS like onPartialResults or onResults. The design for the events is the same as the android speech recognizer, you can change things up if you want.
• Write a proper class on JS side to handle the events and use the results as you please
• The module requires you to store a working model in filesDir/models/kaldi. You can use any other directory you like instead that may be easier accessible that the applications' files directory.
• The module is not perfect and you should adapt its API and use of partial results, results and final results to your needs.

Many of these things could be encapsulated by having a nice NPM package, we should definitely do this at some point.

[Lirry18]

@erksch

Thanks so much for the help. Sadly some of the steps seem a bit above my level of expertise.. Well most importantly the first one, because enabling Kotlin, and including the dependencies is already working. I did some googling on how to write a package file but I get mixed ways of doing it which confuses me.

Could you maybe be a bit more specific about how to write a package file for it? Thanks anyway for the gist!

[erksch]

package <your-package>

import com.facebook.react.ReactPackage
import com.facebook.react.bridge.NativeModule
import com.facebook.react.bridge.ReactApplicationContext
import com.facebook.react.uimanager.ViewManager

class KaldiRNPackage : ReactPackage {
    override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
        return listOf(KaldiRNModule(reactContext))
    }

    override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
        return emptyList()
    }
}

Should be it :) Pretty much the same as https://reactnative.dev/docs/native-modules-android#register-the-module

Then use the package in your MainApplication.java/.kt like you see in the docs also:

protected List<ReactPackage> getPackages() {
  List<ReactPackage> packages = new PackageList(this).getPackages();
  packages.add(new KaldiRNPackage());
  return packages;
} 

Then it should be available in JS as:

import { NativeModules } from 'react-native';
const Kaldi = NativeModules.Kaldi;

[Lirry18]

Hi! Thanks so much, things are going relatively smooth now. I can't seem to find the vosk dependency though..

This is the error. I also tried with version 0.3.15 since that is a later one but also no luck..

Any ideas?

[Lirry18]

Actually, I now notice on the github that vosk-android doesn't exist. Only within the api repo or as android-demo.. Could that be the reason?

[nshmyrev]

I can't seem to find the vosk dependency though

You need to add bintray maven repo in gradle:

https://github.com/alphacep/vosk-android-demo/blob/0d0a3ec212fc37e604c3fde644bf4d4ce045b77a/app/build.gradle#L5

[Lirry18]

Thanks! That worked.

My last problem now is here:

My build seems to fail at the kotlin file, it doesn't recognize the version argument for the boolean. I have to note that I did not add a model to the filesdir, could that be the problem?

@erksch Maybe you have a clue?

[erksch]

Sorry there is an error in the gist. Just remove the version parameter in the createModel call in the initialize method.

Edit: Fixed the error in the Gist.

[Lirry18]

@erksch I did that, couldn't find any other reference to this version parameter so thought it was weird. The only error is still the reactContext parameter passed in the CustomKaldiPackage.kt..

[erksch]

A yeah the variable is named wrong use the one that is the method parameter. (Fixed it in the code snippet now)

[Lirry18]

@erksch The app builds now! Coming closer to the endline. The final problem now is that when I import my native modules in App.tsx, the array is empty. I followed all the instructions, and also tried making a wrapper with a module.exports line, and importing it in App.tsx but this also doesn't work since this is a node notation I think.

Did you have this problem as well? My react-native version is 0.63.3 if that information helps..

[erksch]

Have you registered the package in your MainApplication file?

[Lirry18]

My MainApplication file is in java, heared some pp change that in to kotlin as well, but wasn't sure. The code is below, I added the package like you instructed

package com.testmoduleproject;

import com.testmoduleproject.KaldiRNPackage;
import android.app.Application;
import android.content.Context;
import com.facebook.react.PackageList;
import com.facebook.react.ReactApplication;
import com.facebook.react.ReactInstanceManager;
import com.facebook.react.ReactNativeHost;
import com.facebook.react.ReactPackage;
import com.facebook.soloader.SoLoader;
import java.lang.reflect.InvocationTargetException;
import java.util.List;

public class MainApplication extends Application implements ReactApplication {

  private final ReactNativeHost mReactNativeHost =
      new ReactNativeHost(this) {
        @Override
        public boolean getUseDeveloperSupport() {
          return BuildConfig.DEBUG;
        }

        @Override
        protected List<ReactPackage> getPackages() {
          @SuppressWarnings("UnnecessaryLocalVariable")
          List<ReactPackage> packages = new PackageList(this).getPackages();
          packages.add(new KaldiRNPackage());
          // Packages that cannot be autolinked yet can be added manually here, for example:
          // packages.add(new MyReactNativePackage());
          return packages;
        }

        @Override
        protected String getJSMainModuleName() {
          return "index";
        }
      };

  @Override
  public ReactNativeHost getReactNativeHost() {
    return mReactNativeHost;
  }

  @Override
  public void onCreate() {
    super.onCreate();
    SoLoader.init(this, /* native exopackage */ false);
    initializeFlipper(this, getReactNativeHost().getReactInstanceManager());
  }

  /**
   * Loads Flipper in React Native templates. Call this in the onCreate method with something like
   * initializeFlipper(this, getReactNativeHost().getReactInstanceManager());
   *
   * @param context
   * @param reactInstanceManager
   */
  private static void initializeFlipper(
      Context context, ReactInstanceManager reactInstanceManager) {
    if (BuildConfig.DEBUG) {
      try {
        /*
         We use reflection here to pick up the class that initializes Flipper,
        since Flipper library is not available in release mode
        */
        Class<?> aClass = Class.forName("com.testmoduleproject.ReactNativeFlipper");
        aClass
            .getMethod("initializeFlipper", Context.class, ReactInstanceManager.class)
            .invoke(null, context, reactInstanceManager);
      } catch (ClassNotFoundException e) {
        e.printStackTrace();
      } catch (NoSuchMethodException e) {
        e.printStackTrace();
      } catch (IllegalAccessException e) {
        e.printStackTrace();
      } catch (InvocationTargetException e) {
        e.printStackTrace();
      }
    }
  }
}

[erksch]

Side note: Java is fine and will not make any problems. But the Android world is moving towards Kotlin so make sure to get it in your vocabulary.

Seems good. The NativeModules should be an object that is never empty it contains dozens of default modules. Can you check that you imported and printed it correctly?

[Lirry18]

Right now I import it like this:

import { NativeModules } from 'react-native' console.log(NativeModules)

Did some googling and found that more people have issues: https://github.com/facebook/react-native/issues/26813

Going to look at it, maybe its a react issue

[Lirry18]

@erksch Seems a common problem with react native v > 0.60, can I ask you which version you are using? Maybe I can try downgrading it to that version and see if the problem persists.

[erksch]

I use 0.63. I wouldn't downgrade below 0.60 because there are many differences between 0.5 and 0.6 which may break your project. Are you using expo by any chance? Where do you view your logs, do you use the Chrome debugger?

[Lirry18]

I also used that hmm. No its a react-native cli project so no expo. Just tried the debugger, and on chrome it shows an [object Object] instead of an empty array (in the 'normal' node logger). So thats a bit better at least. Still lost on how to solve this problem

[Lirry18]

Finally found it I think! I can log it in the debugger as "Kaldi", not KaldiVOSKModule! Hopefully it will work outside of debugging in Chrome. Will keep you posted!

Edit: Also works outside the debugging environment now. Seems that a lot of developers have this problem, its probably somewhere in the core of react native.

[erksch]

Ah yeah, you can see in the Gist the module's name is "Kaldi" not "KaldiVOSKModule". https://gist.github.com/erksch/f3d46b478b3912a00a4dc25726d0260c#file-kaldivoskmodule-kt-L40

[Lirry18]

Haha all that hussle for something so small #programming. I now get the error model creation failed. The files are in the highest level (so on the level of node_modules, android, ios etc) in a map called models/kaldi. The files are maps with "am", "conf", "graph" and "ivector".

Must I place them next to the kotlin files or in main?

[erksch]

No the filesDir is not the highest level. Sorry, but this get's rather complicated to explain. Inform yourself about storing files in Android: https://developer.android.com/training/data-storage.

About filesDir here: https://developer.android.com/training/data-storage/app-specific Choose a location that you understand the best and put the files there and implement the path into the Kaldi module.

The filesDir setup I use may be not the one you want, try to use assets or something.

Maybe some general advice. I have nothing against having ambitious projects that are beyond your knowledge so you can grow on them, but take help from others only if you are definitely stuck, not when the next error occurs. If you spent 3 hours understanding the context around an error in order to solve it in the end, then you'll actually gain knowledge. I hope that didn't sound too preachy :D

[Lirry18]

I see! Im confused about why I am not getting the error '..path doesn't exist'. It doesn't matter what I put in, or what I log the promise always is rejected somehow.

[erksch]

Because when you look closely in the Gist, the "path doesnt exist" error is emitted as a JavaScript event instead of throwing an error in Kotlin. (https://reactnative.dev/docs/native-modules-android#sending-events-to-javascript)

By the way it seems than the RN docs have completely new native modules guides :D

[Lirry18]

But then the problem wasn't finding it? Else I would have got it through JS right? Anyway, I made a wrapper now like this: In a file called vosk.ts

import { NativeModules } from 'react-native';
const { Kaldi } = NativeModules
interface VoskInterface {
   initialize(): void;
   startListening(): void;
   stopListening(): void;
   destroy(): void;
}
export default Kaldi as VoskInterface;

Then I can call the functions by importing them in my App.tsx ?

[erksch]

But then the problem wasn't finding it? Else I would have got it through JS right?

Have you even read the link I sent you? It fails, and the event is emitted. You need to register an event listener to actually listen on the event. Please take the time and read through the guide and the other material that I gave you, it's all there.

[Lirry18]

I get it, the other error occurs because you made a react method around it. Now I made an eventlistener so got the error that indeed I need to fix my path to the file. Thanks for all the help!

[Lirry18]

@erksch

Did some research, understand a bit better now.

You get the Application Context through reactContext. Then you look for the filesDir. I wrote some code to list what is in there.

` val list: ArrayList = ArrayList()

   val modelsDir = File("${reactContext.filesDir.absolutePath}/models/kaldi")

    File("${reactContext.filesDir.absolutePath}").walk().forEach {
        list.add(it.toString())
    }`

And by sending it with a listener to react native I get the 'log':

No matter where I put my files, it doesn't seem to be findable. I also tried to access the applicationContext of android, which doesnt work. I tried making an assets folder with a text file, but the reactContext.assets also keeps being empty. I read both articles you send me, and val file = File(context.filesDir, filename) doesnt work.

Did you use react-native-fs or somthing on the javascript side? Sorry for asking again but im stuck now for days

[erksch]

The filesDir is a bit tricky. It is the where internal files of an application can be stored. If you want to put files there manually, you can use the Device File Explorer which you can find at the bottom on the right side of Android Studio. You can then navigate to your app's filesDir by data > data > com.youapp.whatever > files. With right click you can upload the model files there. (This is when using a real Android device that is connected to your computer).

But as you can think, setting the files manually should not be the way to go, I do it programmatically where I download model files from a server and put them there. But that's pretty off-topic and that's why I said it may not be the approach you want to use.

I guess you should continue with the assets directory. Did you make a proper assets directory (e.g. via Android Studio)?

[Lirry18]

I have a real android device connected, but have been doing everything though VSC.. That explains why creating an assets folder there was not the same. I thought both could do the job and since Android Studio is way slower I preferred VSC.

Thanks, will try your suggestion!

[erksch]

I use Android Studio for everything native and VSC for the JS stuff.

[erksch]

val file = File(context.filesDir, filename) Will not do anything when you don't run file.createNewFile().

But there are many ways to create files: https://stackoverflow.com/questions/2885173/how-do-i-create-a-file-and-write-to-it-in-java https://stackoverflow.com/questions/49677168/how-to-create-a-text-file-and-write-to-it-in-kotlin

If you're confused about Path, Files, File and so on, do some research and understand the differences it's worth it.

[JRB-y]

Any npm ackage for this ? Thanks

[Balram143]

@Lirry18 i found this type error ::Unresolved reference: kaldi i am using mac OS

[nshmyrev]

New project for react

https://github.com/riderodd/react-native-vosk