Implementing vad-react with create-react-app

[Instage]

My application was built with create-react-app, which uses react-scripts to hide the webpack config behind the scenes and this appears to be causing issues when trying to follow your guide for react-vad.

I tried installing copy-webpack-plugin, adding the new CopyPlugin to the webpack.config.js (which is found inside node_modules > react-scripts > config), and using the useMicVAD hook as per the guide. However, the hook doesn't work as I never see the console log that should be triggered by onSpeechEnd.

I set up a new React app with Webpack and followed your implementation guide and that does appear to work.

Is it possible to use react-vad with create-react-app? Are you able to provide an example of this implementation?

[ricky0123]

Hi @Instage, a long time ago I briefly looked into how you might configure an app created by create-react-app to work with the vad, but I concluded early on that it was more trouble than it was worth for me at the time. I can eventually try to provide an example but I won't have time to look into it in the near future. However, one thing that I expect would work is to create a build script that looks something like

#!/usr/bin/env bash

npm run build
cp \
    node_modules/@ricky0123/vad-web/dist/silero_vad.onnx \
    node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js \
    node_modules/onnxruntime-web/dist/*.wasm \
    build

True, you can't use the dev server, but at least you should have a working build system.

You might need to copy the files into some subdirectory of the build directory. If the script I gave doesn't work, open dev tools and try to see what paths your site is requesting the onnx/wasm/worklet files from.

[Instage]

Hello @ricky0123,

Thanks for the quick response. I've been continuing to work on integrating into a Create React App based project, and I'm still facing issues. Following your suggestion, I tried to use a custom build script to copy the necessary files, but it didn't resolve the problem. Here's a summary of the steps I've taken so far:

1. I installed the rewire package to override the Create React App's Webpack configuration without ejecting.
2. I tried creating a build.js file and using CopyWebpackPlugin to copy the required files to the build output but eventually decided to manually copy the required files (silero_vad.onnx, vad.worklet.bundle.min.js, ort-wasm.wasm, and ort-wasm-threads.wasm) from their respective locations in the node_modules directory to the public folder.
3. In my component, I tried to initialize the MicVAD instance using the MicVAD.new() method with the appropriate paths to the files, which are now being served from the public folder.
4. When attempting to initialize the MicVAD instance, I encounter the following error: "DOMException: The user aborted a request."

I have double-checked the paths to the required files, and they seem to be correct. I also confirmed that the paths are being requested successfully through the browser. Despite trying different configurations, the error persists, and I am unable to use the library in my Create React App based project.

Here is the current implementation of the Test.js component:

import React, { useState } from "react";
import { MicVAD } from "@ricky0123/vad-web";

const Test = () => {
    console.log(process.env.PUBLIC_URL);

    const [isRecording, setIsRecording] = useState(false);

    const initializeVAD = async () => {
        console.log("Initializing VAD...");
        const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

        const newVad = await MicVAD.new({
          onSpeechStart: () => {
            console.log("User started talking");
            setIsRecording(true);
          },
          onSpeechEnd: (audio) => {
            console.log("User stopped talking");
            console.log(audio);
            setIsRecording(false);
          },
          modelPath: process.env.PUBLIC_URL + '/silero_vad.onnx',
          workletPath: process.env.PUBLIC_URL + '/vad.worklet.bundle.min.js',
          wasmPaths: {
            ortWasmPath: process.env.PUBLIC_URL + '/ort-wasm.wasm',
            ortWasmThreadsPath: process.env.PUBLIC_URL + '/ort-wasm-threads.wasm',
          },
        });

        console.log("VAD initialized");
        return { stream, vad: newVad };
    };

    const handleMouseDown = async () => {
        console.log("startRecording");

        const { stream, vad } = await initializeVAD();
        await vad.start(stream);
    };

    const handleMouseUp = async () => {
        console.log("Stop recording");
        setIsRecording(false);
    };

    return (
        <div className="test-page">
            <h1>Test Page</h1>
            <p>{isRecording ? "User is speaking" : "User is not speaking"}</p>
            <button
                onMouseDown={handleMouseDown}
                onMouseUp={handleMouseUp}
            >
                {isRecording ? "Recording..." : "Hold to Talk"}
            </button>
        </div>
    );
};

export default Test;

Would you be able to provide further guidance on how to resolve this issue? If you could identify any specific steps or configurations needed for the library to work correctly within a Create React App environment, it would be greatly appreciated.

[fengyutingg]

Hi, have you resolve the problem? I meet the same problem as yours.

[LLTOMZHOU]

I have the same problem but in Next.js. I followed the example nextjs set up (same next config and forcing client-side rendering with dynamic, etc). I had to modify the code in the [node_modules/@ricky0123/vad-web/dist/asset-path.js] file because window does not exist at nextJS build time. That got me a partially working app, but upon refreshing the page after initial load, the hook would fail with the same "user aborted request" exception.

I may not be actively looking for a solution any time soon, but thought I would add some information to the issue here. Thanks for making this nice tool :-)

[ricky0123]

Thanks for letting me know, @LLTOMZHOU. I don't have time to look into this at the moment but I will get to it at some point.

[rravenel]

I'm also facing this, hosting on AWS Amplify. My evaluation app was created following Amplify's tutorial, which uses Create React App. I'm able to instantiate a useMicVAD object, but the onSpeechX handlers are never called. Also tried webpack.config.js to get it working.

[rravenel]

Here's an ugly work-around.

Import the dependencies in your index.html and define a global function that returns an instance of MicVAD. Then in App.js, you can use the useEffect hook to call that function and obtain the returned vad object. To get your hands on the audio data, pass in a callback.

index.html at end of body tag:

<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.7/dist/bundle.min.js"></script>
<script>
  window.vadit = async function vadit(onSpeechEndCB) {
    const myvad = await vad.MicVAD.new({
      startOnLoad: true,
      onSpeechStart: () => {
        console.log("onSpeechStart...");
      },
      onSpeechEnd: (audio) => {
        console.log("onSpeechEnd...");
        onSpeechEndCB(audio);
      }
    });

    return myvad;
  }
</script>

App.js in App():

 const [vad, setVad] = useState(null);

 function onSpeechEndCB(audio) {
    // use audio...
  }

  useEffect(() => {
    if (typeof window.vadit === 'function') {
      window.vadit(onSpeechEndCB).then(vad => {
        setVad(vad);
      });
    }
  }, []);

  if (vad === null) {
    console.log("awaiting vad...")
    return (<div>Loading...</div>);
  }

  vad.start();

  console.log(`vad.listening: ${vad.listening}`)

Hopefully @ricky0123 will be able to find some time for this before too long. This is the only browser friendly Silero I've been able to find.

Thanks for the all the hard work!

[amuvarma13]

Here's an ugly work-around.

Import the dependencies in your index.html and define a global function that returns an instance of MicVAD. Then in App.js, you can use the useEffect hook to call that function and obtain the returned vad object. To get your hands on the audio data, pass in a callback.

index.html at end of body tag:

<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.7/dist/bundle.min.js"></script>
<script>
  window.vadit = async function vadit(onSpeechEndCB) {
    const myvad = await vad.MicVAD.new({
      startOnLoad: true,
      onSpeechStart: () => {
        console.log("onSpeechStart...");
      },
      onSpeechEnd: (audio) => {
        console.log("onSpeechEnd...");
        onSpeechEndCB(audio);
      }
    });

    return myvad;
  }
</script>

App.js in App():

const [vad, setVad] = useState(null);

function onSpeechEndCB(audio) {
   // use audio...
 }

 useEffect(() => {
   if (typeof window.vadit === 'function') {
     window.vadit(onSpeechEndCB).then(vad => {
       setVad(vad);
     });
   }
 }, []);

 if (vad === null) {
   console.log("awaiting vad...")
   return (<div>Loading...</div>);
 }

 vad.start();

 console.log(`vad.listening: ${vad.listening}`)

Hopefully @ricky0123 will be able to find some time for this before too long. This is the only browser friendly Silero I've been able to find.

Thanks for the all the hard work!

Works for me but for some reason its slower at least than the demo on the website... any idea about why that might be?

[Sequelator123]

Dear @ricky0123,

is there a possibility that you find time to look into this? We're using the workaround provided by rravenel but the way it is integrated into our angular app is far from good...

[emirsavran]

By using the test-site code and @ricky0123's comment, https://github.com/ricky0123/vad/issues/24#issuecomment-1502638230, I finally make it work with Next.js 13.5.6.

Steps:

1. Copy the necessary files to public directory of Next, like https://github.com/ricky0123/vad/issues/24#issuecomment-1502638230:

   cp \
   node_modules/@ricky0123/vad-web/dist/silero_vad.onnx \
   node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js \
   node_modules/onnxruntime-web/dist/*.wasm \
   public

2. Add onnxruntime-web to the dependencies. Use the following line in the same file with the import of useMicVAD:

   
   // import it
   import * as ort from "onnxruntime-web";

// change the config ort.env.wasm.wasmPaths = { "ort-wasm-simd-threaded.wasm": "/ort-wasm-simd-threaded.wasm", "ort-wasm-simd.wasm": "/ort-wasm-simd.wasm", "ort-wasm.wasm": "/ort-wasm.wasm", "ort-wasm-threaded.wasm": "/ort-wasm-threaded.wasm", };

3. Use `useMicVad` with the following options:
```js
  const vad = useMicVAD({
    modelURL: "/silero_vad.onnx",
    workletURL: "/vad.worklet.bundle.min.js",
    // other options

It's not ideal but it works. I'll try to find a proper solution when I have a time.

[emirsavran]

It seems like onnx-runtime tries to load Wasm files relative to the current URL. For instance, if I delete the ort.env.wasm.wasmPaths line, it tries to load the files likehttp://localhost:3000/_next/static/chunks/pages/dashboard/ort-wasm-simd.wasm. As far as I understand, the following code supports my point: https://github.com/microsoft/onnxruntime/blob/main/js/web/lib/wasm/proxy-wrapper.ts#L126

So, we need to configure the wasmPaths anyway. My solution is to use the copy-webpack plugin, and initialize the hook and ort with the following config:

import { useMicVAD } from "@ricky0123/vad-react";
import * as ort from "onnxruntime-web";

ort.env.wasm.wasmPaths = "/_next/static/chunks/";

const vad = useMicVAD({
    modelURL: "/_next/static/chunks/silero_vad.onnx",
    workletURL: "/_next/static/chunks/vad.worklet.bundle.min.js",
});

[lachlansleight]

Heya, I managed to figure out an alternative solution based on this next.config.js file in the next-onnx repository. This works without needing to manually move any files or do any manual configuration of the ort object.

There are three steps required:

1. Move silero_vad_onnx and vad.worklet.bundle.min.js from node_modules/@ricky0123/vad-web/dist to public, as mentioned in the docs
2. Add copy-webpack-plugin to your project (npm i -D copy-webpack-plugin)
3. Add the following to your next.config.js (I'll add the whole file here for clarity, but if you have anything in your config already you'll need to combine this with whatever you've already got):

const CopyPlugin = require("copy-webpack-plugin");

const wasmPaths = [
    "./node_modules/onnxruntime-web/dist/ort-wasm.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-threaded.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd.jsep.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.jsep.wasm",
    "./node_modules/onnxruntime-web/dist/ort-training-wasm-simd.wasm",
];

/** @type {import('next').NextConfig} */
const nextConfig = {
    webpack: (config) => {
        config.resolve.fallback = {
          ...config.resolve.fallback,

          fs: false,
        };

        //local dev server - copy wasm into static/chunks/app
        config.plugins.push(
          new CopyPlugin({ patterns: wasmPaths.map(p => ({from: p, to: "static/chunks/app"})) })
        );

        //vercel - copy wasm into static/chunks
        config.plugins.push(
            new CopyPlugin({ patterns: wasmPaths.map(p => ({from: p, to: "static/chunks"})) })
          );

        return config;
      },
};

module.exports = nextConfig;

This is using next.js 14 with my project organized to use the src/app directory structure. Perhaps there's a way to avoid having to have two separate plugins, but for whatever reason I noticed that VAD was looking for its files in _next/static/chunks/app while running as a local development server, but looking in _next/static/chunks on vercel. So I just made a second plugin 😅.

[arvindmn01]

Has there been any update on using VAD with create-react-app? I have followed all the provided installation steps, I have created webpack.config.js file, but still getting this error.

Failed to parse source map from 'D:\Web_app\front_end\testing_code\test\node_modules\@ricky0123\vad-react\src\index.ts' file: Error: ENOENT: no such file or directory, open 'D:\Web_app\front_end\testing_code\test\node_modules\@ricky0123\vad-react\src\index.ts'

[ricky0123]

Hi all, yesterday I released a couple of relevant updates

• If the model file or worklet file fail to load when using the realtime vad, you should now see an error message in the console that tells you where those files are currently expected to be located
• Let's say (and I'm not talking about in a nextjs project) in the web root you have subdir/index.html and subdir/index.js where the Javascript file loads the vad but does not specify workletURL or modelURL. Previously, by default it would look for the onnx/worklet file in /subdir/vad.worklet.bundle.min.js and /subdir/silero_vad.onnx. But now, by default it will look for them in /vad.worklet.bundle.min.js and /silero_vad.onnx. This will not affect nextjs projects. For clarity, if the modelURL and workletURL options are not provided, the paths for these files is calculated as assetPath('vad.worklet.bundle.min.js') and assetPath('silero_vad.onnx') where assetPath is defined as in https://github.com/ricky0123/vad/blob/ea584aaf66d9162fb19d9bfba607e264452980c3/packages/web/src/asset-path.ts#L4-L18 This was my way of getting the package to work when using it with jsdelivr. I'm open to suggestions for improvement. My sense is that for nextjs projects the problem has been solved in the comments here (thanks!). I know that in my original comment I suggested using a script to copy certain files to public, but now I prefer using copy-webpack-plugin for everything if that is an option. You can see how I do it in the nextjs example, although going by @lachlansleight's comment, it seems as though you also need to copy to static/chunks/app (is that for new versions of nextjs?). It seems as though copy-webpack-plugin is not an option for create-react-app projects, so you would probably have to use a script or some other way to copy the files to whichever directory they belong in.
• There is now a configuration parameter ortConfig. If you are having trouble loading the wasm files, try something like

  // in the config you are passing to real or non real time vad
  ortConfig: (ort) => {
    ort.env.wasm.wasmPaths = "/the/correct/path"
  }

[lox]

I suspect the currentScript approach isn't going to work for next.js with any depth of pages, an example of what the currentScript.src is for my page is:

src="/_next/static/chunks/app/dashboard/sessions/%5Bid%5D/xxx/page.js"

Using the implementation in assetPath() is going to try and load the worklet and model relative to the current page, rather than from /_next/static/chunks.

[lox]

This is gross, but works if you have complex next.js paths:

const vad = useMicVAD({
    workletURL: "/_next/static/chunks/vad.worklet.bundle.min.js",
    modelURL: "/_next/static/chunks/silero_vad.onnx",
    modelFetcher: (path) => {
      const filename = path.split('/').pop()
      return fetch(/_next/static/chunks/${filename}).then((model) => model.arrayBuffer())
    },
    ortConfig: (ort) => {
      ort.env.wasm.wasmPaths = "/_next/static/chunks/"
    },
    onSpeechEnd: (audio) => {
      // ...
    },
  })

[joeleegithub]

Hi @ricky0123:

Are you aware that when you start myvad.start() on iPhone, the speaker output volume is cut almost by half automatically.

It also affects the volume on other opened browser programs that are currently streaming music and voice. It only happens on iPhone, not on Android and PCs. Once you stops the myvad() the speaker volume goes back to normal, for the other opened browser programs too.

This also occurs when you run Ricky's vad demo on a iPhone: https://www.vad.ricky0123.com/

Any insight would be much appreciated!

[ricky0123]

Hi @joeleegithub can you create a new issue for that? I don't think it is related to this thread

[joeleegithub]

Hi Ricky:

Thank you, Just opened up an issue #96.

Joe Lee

Sign-A-Rama

416-783-5472

From: Ricky Samore @.> Sent: Sunday, April 21, 2024 10:10 PM To: ricky0123/vad @.> Cc: Joseph Lee @.>; Mention @.> Subject: Re: [ricky0123/vad] Implementing vad-react with create-react-app (Issue #24)

Hi @joeleegithub https://github.com/joeleegithub can you create a new issue for that? I don't think it is related to this thread

— Reply to this email directly, view it on GitHub https://github.com/ricky0123/vad/issues/24#issuecomment-2068360209 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPVOWFNCXUB4J3T2NNNI53Y6RWQJAVCNFSM6AAAAAAWZOA2BCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYGM3DAMRQHE . You are receiving this because you were mentioned. https://github.com/notifications/beacon/AQPVOWDA3OYIGWAY34RYYZDY6RWQJA5CNFSM6AAAAAAWZOA2BCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTT3JCWBC.gif Message ID: @. @.> >

[wangtiger317]

This is gross, but works if you have complex next.js paths:

const vad = useMicVAD({
    workletURL: "/_next/static/chunks/vad.worklet.bundle.min.js",
    modelURL: "/_next/static/chunks/silero_vad.onnx",
    modelFetcher: (path) => {
      const filename = path.split('/').pop()
      return fetch(/_next/static/chunks/${filename}).then((model) => model.arrayBuffer())
    },
    ortConfig: (ort) => {
      ort.env.wasm.wasmPaths = "/_next/static/chunks/"
    },
    onSpeechEnd: (audio) => {
      // ...
    },
  })

this setting works on server? SyntaxError: Identifier 'ey' has already been declared (at 006b3fc2-d9877509d27d2604.js:1:52799)

I have this issue

[wangtiger317]

Here's an ugly work-around. Import the dependencies in your index.html and define a global function that returns an instance of MicVAD. Then in App.js, you can use the useEffect hook to call that function and obtain the returned vad object. To get your hands on the audio data, pass in a callback. index.html at end of body tag:

<script src="https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/ort.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.7/dist/bundle.min.js"></script>
<script>
  window.vadit = async function vadit(onSpeechEndCB) {
    const myvad = await vad.MicVAD.new({
      startOnLoad: true,
      onSpeechStart: () => {
        console.log("onSpeechStart...");
      },
      onSpeechEnd: (audio) => {
        console.log("onSpeechEnd...");
        onSpeechEndCB(audio);
      }
    });

    return myvad;
  }
</script>

App.js in App():

const [vad, setVad] = useState(null);

function onSpeechEndCB(audio) {
   // use audio...
 }

 useEffect(() => {
   if (typeof window.vadit === 'function') {
     window.vadit(onSpeechEndCB).then(vad => {
       setVad(vad);
     });
   }
 }, []);

 if (vad === null) {
   console.log("awaiting vad...")
   return (<div>Loading...</div>);
 }

 vad.start();

 console.log(`vad.listening: ${vad.listening}`)

Hopefully @ricky0123 will be able to find some time for this before too long. This is the only browser friendly Silero I've been able to find. Thanks for the all the hard work!

Works for me but for some reason its slower at least than the demo on the website... any idea about why that might be?

after download files(ort.js and bundle.min.js), after import, it gives me an error. The user aborted a request who knows how to fix it?

[shutootaki]

We implemented the following in Next.js and it worked without any problems.

1. add the following to next.config.mjs and copy the files needed for start-up, including .wasm, to the public directory

   
   import fs from "node:fs/promises";
   import path from "node:path";

async function copyFiles() { try { await fs.access("public/vad/"); } catch { await fs.mkdir("public/vad/", { recursive: true }); }

const wasmFiles = ( await fs.readdir("node_modules/onnxruntime-web/dist/") ).filter((file) => path.extname(file) === ".wasm");

await Promise.all([ fs.copyFile( "node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js", "public/vad/vad.worklet.bundle.min.js" ), fs.copyFile( "node_modules/@ricky0123/vad-web/dist/silero_vad.onnx", "public/vad/silero_vad.onnx" ), ...wasmFiles.map((file) => fs.copyFile( node_modules/onnxruntime-web/dist/${file}, public/vad/${file} ) ), ]); }

copyFiles();

2. add the path of the copied file to the useMicVAD argument.

const vad = useMicVAD({ workletURL: '/vad/vad.worklet.bundle.min.js', modelURL: '/vad/silero_vad.onnx', ortConfig(ort) { // eslint-disable-next-line @typescript-eslint/no-unsafe-member-access ort.env.wasm = { wasmPaths: { 'ort-wasm-simd-threaded.wasm': '/vad/ort-wasm-simd-threaded.wasm', 'ort-wasm-simd.wasm': '/vad/ort-wasm-simd.wasm', 'ort-wasm.wasm': '/vad/ort-wasm.wasm', 'ort-wasm-threaded.wasm': '/vad/ort-wasm-threaded.wasm', }, }; }, });

Reference:.
https://wiki.vad.ricky0123.com/en/docs/user/browser

[PrinceBaghel258025]

Heya, I managed to figure out an alternative solution based on this next.config.js file in the next-onnx repository. This works without needing to manually move any files or do any manual configuration of the ort object.

There are three steps required:

1. Move silero_vad_onnx and vad.worklet.bundle.min.js from node_modules/@ricky0123/vad-web/dist to public, as mentioned in the docs
2. Add copy-webpack-plugin to your project (npm i -D copy-webpack-plugin)
3. Add the following to your next.config.js (I'll add the whole file here for clarity, but if you have anything in your config already you'll need to combine this with whatever you've already got):

const CopyPlugin = require("copy-webpack-plugin");

const wasmPaths = [
    "./node_modules/onnxruntime-web/dist/ort-wasm.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-threaded.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd.jsep.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.wasm",
    "./node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.jsep.wasm",
    "./node_modules/onnxruntime-web/dist/ort-training-wasm-simd.wasm",
];

/** @type {import('next').NextConfig} */
const nextConfig = {
    webpack: (config) => {
        config.resolve.fallback = {
          ...config.resolve.fallback,

          fs: false,
        };

        //local dev server - copy wasm into static/chunks/app
        config.plugins.push(
          new CopyPlugin({ patterns: wasmPaths.map(p => ({from: p, to: "static/chunks/app"})) })
        );

        //vercel - copy wasm into static/chunks
        config.plugins.push(
            new CopyPlugin({ patterns: wasmPaths.map(p => ({from: p, to: "static/chunks"})) })
          );

        return config;
      },
};

module.exports = nextConfig;

This is using next.js 14 with my project organized to use the src/app directory structure. Perhaps there's a way to avoid having to have two separate plugins, but for whatever reason I noticed that VAD was looking for its files in _next/static/chunks/app while running as a local development server, but looking in _next/static/chunks on vercel. So I just made a second plugin 😅.

Are you able to do production build for it ?

[lachlansleight]

Are you able to do production build for it ?

Yes, this worked fine when deploying to vercel.