search

ricky0123 / vad

Voice activity detector (VAD) for the browser with a simple API
https://www.vad.ricky0123.com
Other
932 stars 148 forks source link

@ricky0123/vad-react, @ricky0123/vad-web does not work in monorepo setup #161

Open scottsus opened 2 days ago

scottsus commented 2 days ago

Problem

@ricky0123/vad-web, @ricky0123/vad-react not working in monorepo setup. I suspect it's because I'm serving the wasm, onnx, and js static files wrongly (since turbopack does not support CopyPlugin).

Error message

Failed to compile

Next.js (14.2.18) is outdated [(learn more)](https://nextjs.org/docs/messages/version-staleness) (turbo)
./node_modules/onnxruntime-web/dist/ort.min.js:1802:6893
Module not found
  1800 | Shader source:
  1801 | ${e}`);return r}deleteShader(e){this.gl.deleteShader(e)}bindTextureToUniform(e,o,t){let r=this.gl;r.activeTexture(r.TEXTURE0+o),this.checkError(),r.bindTexture(r.TEXTURE_2D,e),this.checkError(),r.uniform1i(t,o),this.checkError()}draw(){this.gl.drawArrays(this.gl.TRIANGLE_STRIP,0,4),this.checkError()}checkError(){if($.debug){let e=this.gl,o=e.getError(),t="";switch(o){case e.NO_ERROR:return;case e.INVALID_ENUM:t="INVALID_ENUM";break;case e.INVALID_VALUE:t="INVALID_VALUE";break;case e.INVALID_OPERATION:t="INVALID_OPERATION";break;case e.INVALID_FRAMEBUFFER_OPERATION:t="INVALID_FRAMEBUFFER_OPERATION";break;case e.OUT_OF_MEMORY:t="OUT_OF_MEMORY";break;case e.CONTEXT_LOST_WEBGL:t="CONTEXT_LOST_WEBGL";break;default:t=`Unknown WebGL Error: ${o.toString(16)}`}throw new Error(t)}}deleteTexture(e){this.gl.deleteTexture(e)}deleteProgram(e){this.gl.deleteProgram(e)}getEncoder(e,o,t=0){if(this.version===2)return new Gr(this.gl,o);switch(e){case"float":return t===1||this.isRenderFloat32Supported?new Qe(this.gl,o):new

What I tried

  1. Manually copying the wasm, onnx, js files into .next, .next/static, .next/static/chunks
  2. Placing the warm, onnx, js files into /public (fetch-ing these files at runtime to sanity check)
  3. Using onnxruntime-web@1.21.0, onnxruntime-web@1.18.0
  4. Including the modelUrl, workletUrl, and exluding it

Code

My code uses a few variants of the following:

"use client";

export function MyComponent() {
  async function setup() {
    const MicVad = (await import("@ricky0123/vad-web")).MicVAD;
    const vad = await MicVad.new({
      //   modelURL: "/silero_vad.onnx",
      //   workletURL: "/vad.worklet.bundle.min.js",
      onSpeechEnd: (audio) => {
        console.log(audio);
      },
    });
  }
  return <div>Hello from Vad</div>;
}

All of @ricky0123/vad-web, @ricky0123/vad-react, using await import vs import { } from all give the same error.

Additional context

Currently using turborepo, installed @ricky0123/vad-react, @ricky0123/vad-web, onnxruntime-web all in apps/web.

ricky0123 commented 2 days ago

Hi @scottsus by chance I have just released a relevant update in the newest version of this package. Please take a look at this section of the documentation. Basically, by default now those files will be pulled from the CDN.

You can also check out the nextjs example, which shows how to serve the files yourself instead of pulling them from the CDN. Like that project, I recommend specifically using version 1.14.0 of onnxruntime-web.

If you still have problems, please look at the network tab of the browser devtools and post here where the files are trying to be loaded from.

scottsus commented 2 days ago

Hey @ricky0123, thanks for the super fast response. Unfortunately, I am still getting the exact same error.

image

In particular, the network tab does not make any requests to a particular CDN.

image

Here are the relevant files.

Edit**: additionally, because of the CopyPlugin, I get the following warning, which may or may not be relevant:

 ⚠ Webpack is configured while Turbopack is not, which may cause problems.
 ⚠ See instructions if you need to configure Turbopack:
  https://nextjs.org/docs/app/api-reference/next-config-js/turbo

In particular, I have a curious question in your nextjs example: if the files are statically served from a CDN, why do we still need the CopyPlugin? Is it only if we specify the baseAssetPath and onnxWASMBasePath to "/"? In other words, should we remove those attributes entirely so we can fetch from CDN? But as you can see from the screenshot, that does not work for me either.

Thank you very much for your support 🙏

scottsus commented 1 day ago

Hey @ricky0123, I finally fixed my Shader source error, it turns out I was using next:14.1.1, and upgrading to next:14.2.6 or later worked 👍

However, and this is more relevant to you, but the CDN is not serving these files: image

In other words,

Here is my demo.tsx for reference, with the baseAssetPath and onnxWASMBasePath commented out to default to the CDN locations.

"use client";

import { useMicVAD, utils } from "@ricky0123/vad-react";
import { useState } from "react";

export const Demo = () => {
  const [audioList, setAudioList] = useState<string[]>([]);
  const vad = useMicVAD({
    model: "v5",
    // baseAssetPath: "/",
    // onnxWASMBasePath: "/",
    onSpeechEnd: (audio) => {
      const wavBuffer = utils.encodeWAV(audio);
      const base64 = utils.arrayBufferToBase64(wavBuffer);
      const url = `data:audio/wav;base64,${base64}`;
      setAudioList((old) => {
        return [url, ...old];
      });
    },
  });
  return (
    <div className="text-black">
      <h6>Listening</h6>
      {!vad.listening && "Not"} listening
      <h6>Loading</h6>
      {!vad.loading && "Not"} loading
      <h6>Errored</h6>
      {!vad.errored && "Not"} errored
      <h6>User Speaking</h6>
      {!vad.userSpeaking && "Not"} speaking
      <h6>Audio count</h6>
      {audioList.length}
      <h6>Start/Pause</h6>
      <button onClick={vad.pause}>Pause</button>
      <button onClick={vad.start}>Start</button>
      <button onClick={vad.toggle}>Toggle</button>
    </div>
  );
};

export default Demo;

Please let me know if it's better for us to serve the wasm files ourself. Thanks again!

ricky0123 commented 1 day ago

Hi @scottsus can you try specifically installing onnxruntime-web version 1.14.0? That version does not require ort-wasm-simd-threaded.mjs and generally seems to cause fewer problems

scottsus commented 1 day ago

Hey @ricky0123, I just did the following in my minimal Nextjs repro:

yarn remove @ricky0123/vad-react @ricky0123/vad-web onnxruntime-web
rm -rf node_modules yarn.lock
yarn add onnxruntime-web@1.14.0 @ricky0123/vad-react (this is ^0.0.27)
yarn dev

Still getting the same error, first unable to fetch the ort-wasm-simd-threaded.mjs, then

app-index.js:33 Encountered an error while loading model file https://cdn.jsdelivr.net/npm/@ricky0123/vad-web@0.0.20/dist/silero_vad_v5.onnx

Edit**: stack trace for reference image

Fetching onnx file from CDN is successful however, but the loading part is not. Any suggestions?

ricky0123 commented 21 hours ago

@scottsus the file ort-wasm-simd-threaded.mjs doesn't seem to be part of onnxruntime-web version 1.14.0. Can you check the node_modules/onnxruntime-web/package.json "version" field to make sure you successfully installed that version?

scottsus commented 20 hours ago

Hey @ricky0123, yes, I can verify inside node_modules/onnxruntime-web/package.json that it is in fact version 1.14.0

{
  "license": "MIT",
  "browser": "dist/ort-web.min.js",
  "unpkg": "dist/ort.min.js",
  "name": "onnxruntime-web",
  "repository": {
    "url": "https://github.com/Microsoft/onnxruntime.git",
    "type": "git"
  },
  "author": "fs-eire",
  "version": "1.14.0",
  "jsdelivr": "dist/ort.min.js",
  "dependencies": {
    "flatbuffers": "^1.12.0",
    "guid-typescript": "^1.0.9",
    "long": "^4.0.0",
    "onnx-proto": "^4.0.4",
    "onnxruntime-common": "~1.14.0",
    "platform": "^1.3.6"
  },
  "scripts": {
    "prepare": "tsc",
    "build:doc": "node ./script/generate-operator-md",
    "pull:wasm": "node ./script/pull-prebuilt-wasm-artifacts",
    "test:e2e": "node ./test/e2e/run",
    "build": "node ./script/build",
    "test": "tsc --build ../scripts && node ../scripts/prepare-onnx-node-tests && node ./script/test-runner-cli",
    "prepack": "node ./script/build && node ./script/prepack"
  },
  "keywords": [
    "ONNX",
    "ONNXRuntime",
    "ONNX Runtime"
  ],
  ...
}

Might it be that the vad-web package is not updated?

scottsus commented 14 hours ago

Hey @ricky0123, I found the culprit.

Repro

Using your nextjs example as a starting point, running

npm install
npm run dev

works fine. However, using yarn does not

yarn install
yarn dev

and gives the ort-wasm-simd-threaded.mjs error.

onnxruntime-common versions

I'm not very familiar with npm and yarn version resolutions, but take a look here: package-lock.json

    "node_modules/onnxruntime-common": {
      "version": "1.14.0",
      "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz",
      "integrity": "sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==",
      "license": "MIT"
    },
    "node_modules/onnxruntime-web": {
      "version": "1.14.0",
      "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz",
      "integrity": "sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==",
      "license": "MIT",
      "dependencies": {
        "flatbuffers": "^1.12.0",
        "guid-typescript": "^1.0.9",
        "long": "^4.0.0",
        "onnx-proto": "^4.0.4",
        "onnxruntime-common": "~1.14.0",
        "platform": "^1.3.6"
      }
    },

yarn.lock

onnxruntime-common@1.14.0, onnxruntime-common@~1.14.0:
  version "1.14.0"
  resolved "https://registry.yarnpkg.com/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz#2bb5dac5261269779aa5fb6536ca379657de8bf6"
  integrity sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==

onnxruntime-common@1.20.1:
  version "1.20.1"
  resolved "https://registry.yarnpkg.com/onnxruntime-common/-/onnxruntime-common-1.20.1.tgz#b42e317d4d6728745b9e8089617c8cd938d312dc"
  integrity sha512-YiU0s0IzYYC+gWvqD1HzLc46Du1sXpSiwzKb63PACIJr6LfL27VsXSXQvt68EzD3V0D5Bc0vyJTjmMxp0ylQiw==

onnxruntime-web@1.14.0:
  version "1.14.0"
  resolved "https://registry.yarnpkg.com/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz#c8cee538781b1d4c1c6b043934f4a3e6ddf1466e"
  integrity sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==
  dependencies:
    flatbuffers "^1.12.0"
    guid-typescript "^1.0.9"
    long "^4.0.0"
    onnx-proto "^4.0.4"
    onnxruntime-common "~1.14.0"
    platform "^1.3.6"

onnxruntime-web@^1.14.0:
  version "1.20.1"
  resolved "https://registry.yarnpkg.com/onnxruntime-web/-/onnxruntime-web-1.20.1.tgz#71dc4495bb75cdd56a9dc981cf27723a599189f7"
  integrity sha512-TePF6XVpLL1rWVMIl5Y9ACBQcyCNFThZON/jgElNd9Txb73CIEGlklhYR3UEr1cp5r0rbGI6nDwwrs79g7WjoA==
  dependencies:
    flatbuffers "^1.12.0"
    guid-typescript "^1.0.9"
    long "^5.2.3"
    onnxruntime-common "1.20.1"
    platform "^1.3.6"
    protobufjs "^7.2.4"

Am I tripping or does npm strictly use onnxruntime-common@1.14.0 while yarn uses onnxruntime-common@1.20.1 via onnxruntime-web@1.14.0?

npm why, yarn why

npm

npm why onnxruntime-common
onnxruntime-common@1.14.0
node_modules/onnxruntime-common
  onnxruntime-common@"1.14.0" from the root project
  onnxruntime-common@"~1.14.0" from onnxruntime-web@1.14.0
  node_modules/onnxruntime-web
    onnxruntime-web@"1.14.0" from the root project

onnxruntime-common@1.20.1
node_modules/@ricky0123/vad-react/node_modules/onnxruntime-common
  onnxruntime-common@"1.20.1" from onnxruntime-web@1.20.1
  node_modules/@ricky0123/vad-react/node_modules/onnxruntime-web
    onnxruntime-web@"^1.14.0" from @ricky0123/vad-react@0.0.27
    node_modules/@ricky0123/vad-react
      @ricky0123/vad-react@"^0.0.27" from the root project

onnxruntime-common@1.20.1
node_modules/@ricky0123/vad-web/node_modules/onnxruntime-common
  onnxruntime-common@"1.20.1" from onnxruntime-web@1.20.1
  node_modules/@ricky0123/vad-web/node_modules/onnxruntime-web
    onnxruntime-web@"^1.14.0" from @ricky0123/vad-web@0.0.21
    node_modules/@ricky0123/vad-web
      @ricky0123/vad-web@"^0.0.21" from @ricky0123/vad-react@0.0.27
      node_modules/@ricky0123/vad-react
        @ricky0123/vad-react@"^0.0.27" from the root project

yarn

yarn why onnxruntime-common
yarn why v1.22.22
[1/4] 🤔  Why do we have the module "onnxruntime-common"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
=> Found "onnxruntime-common@1.14.0"
info Has been hoisted to "onnxruntime-common"
info Reasons this module exists
   - Specified in "dependencies"
   - Hoisted from "onnxruntime-web#onnxruntime-common"
info Disk size without dependencies: "852KB"
info Disk size with unique dependencies: "852KB"
info Disk size with transitive dependencies: "852KB"
info Number of shared dependencies: 0
=> Found "@ricky0123/vad-react#onnxruntime-common@1.20.1"
info Reasons this module exists
   - "@ricky0123#vad-react#onnxruntime-web" depends on it
   - Hoisted from "@ricky0123#vad-react#onnxruntime-web#onnxruntime-common"
info Disk size without dependencies: "1.18MB"
info Disk size with unique dependencies: "1.18MB"
info Disk size with transitive dependencies: "1.18MB"
info Number of shared dependencies: 0
=> Found "@ricky0123/vad-web#onnxruntime-common@1.20.1"
info Reasons this module exists
   - "@ricky0123#vad-react#@ricky0123#vad-web#onnxruntime-web" depends on it
   - Hoisted from "@ricky0123#vad-react#@ricky0123#vad-web#onnxruntime-web#onnxruntime-common"
info Disk size without dependencies: "1.18MB"
info Disk size with unique dependencies: "1.18MB"
info Disk size with transitive dependencies: "1.18MB"
info Number of shared dependencies: 0
✨  Done in 0.09s.

force versions

I even tried

rm -rf node_modules yarn.lock
yarn add onnxruntime-common@1.14.0
yarn install --force

and I still see the 1.20.1 version there.

Might you have any suggestions on this?

scottsus commented 10 hours ago

IT'S FINALLY WORKING AFTER 3 DAYS

Shader source

It was extremely bewildering to me how I could not find any trace of the Shader source issue anywhere on the whole internet (Google, Perplexity, Stack Overflow, etc). It turns out the simple fix was to upgrade Next.js to a compatible version ✅

This was particularly frustrating because the 10,000 things I tried all gave the same error (it would be nicer to at least get a different error and make progress).

onnxruntime-web & onnxruntime-common

I only managed to discover this by accident after running npm run dev which worked, yarn dev which failed, then inspecting the package-lock.json versus the yarn.lock. Unsurprisingly (in hindsight), yarn allows for multiple versions of a particular package to exist.

Anyway, as pointed out by @ricky0123, we want to use precisely 1.14.0 for onnxruntime-web, and 1.21.1 was in fact not working with the current @ricky0123/vad-react package. What was difficult to spot as a fresh grad (not anymore after this 😤) is

  1. how yarn keeps track of package versions, including dependencies
  2. yarn.lock syntax

Even having the conviction that it's a version mismatch was absent for me, so going into the yarn.lock file was definitely not a priority, always attempting the easy fix of yarn add, yarn remove, yarn cache clean, etc. So an entry in yarn.lock looks like this:

onnxruntime-web@^1.14.0:
  version "1.20.1"
  resolved "https://registry.yarnpkg.com/onnxruntime-web/-/onnxruntime-web-1.20.1.tgz#71dc4495bb75cdd56a9dc981cf27723a599189f7"
  integrity sha512-TePF6XVpLL1rWVMIl5Y9ACBQcyCNFThZON/jgElNd9Txb73CIEGlklhYR3UEr1cp5r0rbGI6nDwwrs79g7WjoA==
  dependencies:
    flatbuffers "^1.12.0"
    guid-typescript "^1.0.9"
    long "^5.2.3"
    onnxruntime-common "1.20.1"
    platform "^1.3.6"
    protobufjs "^7.2.4"

In particular, the ^1.14.0 tells us yarn wants the latest version that is still compatible with the current major version, which in this case is actually "1.20.1"! This is where we messed up!

Knowing this, how do we apply the fix? Yarn has something called selective dependency resolution which resolves a specific dependency for a specific package. This was also quite difficult to find.

Moreover, there was learning about yarn why onnxruntime-web, yarn install --flat, and other things that were easily thrown by Claude and Cursor but needed careful understanding of how they worked. Alas I was able to get it working inside my Next.js repro with the following edit in my package.json:

  "resolutions": {
    "**/onnxruntime-web": "1.14.0"
  },

Finally, while I was able to reproduce this in a basic Next app, this would suddenly fail in my monorepo! The final fix was to put this in the root package.json. One thing I haven't figured out yet is how to scope this to a particular apps/web directory.

Red herrings

Many things were particularly distracting:

What next

Could I suggest changing the dependency in @ricky0123/vad-react to exactly 1.14.0 instead of ^1.14.0? If that were the case this entire onnxruntime thing would not have happened (but then I wouldn't have learned about this). But it's nice if other people don't suffer the same fate.

Final thanks

I'm writing this as if I won the world cup, but I think this is my first big problem where I'm trying to debug a 3p open source library in the real world, and after 3 days I finally succeeded. VAD will be big in my upcoming app and I'm incredibly ecstatic on this W.

Btw, I'm making a voice agent with my voice (cloned) and publicly available information for friends and recruiters to play around with. Just a hobby project, check it out here https://github.com/scottsus/deep-clone.

Thanks to @ricky0123 for your help 🚀