Is there any way to use this library without node.js but just with reactjs? [Question]

[NomioG]

Question. I'm trying to use twilio-video-processor with twilio-video in reactjs. But its not working correctly, even though I followed the guide in the documentation. I see there was an example project but it was using node and serving the tflite and simd. I'm not sure how does that work in ReactJs and currently I can't use the library.

Describe alternatives you've considered I feel like the documentation has very limited info. I'm not sure if you can use the library with ReactJs if you follow the documentation. ( Or maybe I'm doing something wrong )

Additional context I'm pretty sure the issue is that the required files are not being loaded correctly.

But I'm not sure how to load it properly. I tried to load it as script in the Index.html and also tried to directly load it into the js file which handling the video but neither of them worked.

It seemed like issue like this never created before and not a lot of discussion happening on Stackoverflow or any other sites.

[sarahcstringer]

@NomioG yes you should be able to use the library with ReactJS -- there are a few different dependencies here. You can load the twilio-video-processors.js from the Index.html file, but then you also need to provide a path the tflite dependencies when you run the video processors library https://twilio.github.io/twilio-video-processors.js/interfaces/virtualbackgroundprocessoroptions.html#assetspath

Can you share what you are putting for the assetsPath value, and verify that it matches where you've put all of the tflite/binary dependencies in your project?

[NomioG]

Hi @sarahcstringer I put '/virtualbackground' for assetPath and the /virtualbackground is in the public directory. I copied the files in @twilio/video-processors/dist/build to /public/virtualbackground. I feel like the issue was the project using parcel instead of react-script bundler and the public directory was not accessible. However, I realized that and set the parcel to load the files in the public directory.

Here is the code I'm using for videoTrack creation and postProcessing

            const videoTrack = await Video.createLocalVideoTrack({
            width: props.width,
            height: props.height,
            frameRate: 24
            });
            document.getElementById('video-player')?.appendChild(videoTrack.attach());

            const bg = new VideoProcessors.GaussianBlurBackgroundProcessor({
            assetsPath: '/virtualbackgrounds',
            maskBlurRadius: 10,
            blurFilterRadius: 5,
            });
            await bg.loadModel();
            videoTrack.addProcessor(bg);

I'm not sure if the tflite files are loaded correctly or not. It's saying 404 not found, which means most likely not. But, I'm not sure how to load them correctly.

the npm start script is: parcel --open --port 3000 --public-url /public

if I remove the --public-url /public and just use parcel --open --port 3000, the error becomes this:

P.S: I'm using postinstall script to copy the files from dist to public/virtualbackground

"postinstall": "rimraf public/virtualbackground && copyfiles -f node_modules/@twilio/video-processors/dist/build/* public/virtualbackground"

[sarahcstringer]

@NomioG what happens if you list the assets path as /public/virtualbackgrounds? It looks like that's where you're hosting them. Right now the video processors library is going directly to localhost:3000/virtualbackgrounds rather than looking in the public directory.

[NomioG]

@sarahcstringer If I set it to /public/virtualbackgrounds instead of /virtualbackgrounds, the error becomes

if I hover over the Unexpected token '<' line's tflite-1-0-0.js:1, the path is http://localhost:3000/public/virtualbackground/tflite-1-0-0.js:1

In this case it's saying that the createTwilioTFLiteModule is not a function, I felt like it found the files but couldn't recognize the function.

P.S: if I leave the assetPath as blank string like '' <= like this, it's gonna give me the same error as in the picture. So, I thought maybe the tflite files are not being loaded correctly.

[sarahcstringer]

@NomioG can you check the Network tab in the Chrome Developer Console and share what types of headers/content and any errors you're getting back from the requests for the tflite file?

[NomioG]

@sarahcstringer According to the Network all the files loaded without any issue.

However, If I check the each files individually, tflite.1.0.0,tflite-simd.1.0.0, and selfie_segmentation_landscape.tflite the response of the all 3 files are the same as follows:

[sarahcstringer]

Interesting, that shouldn't be the content -- can you verify that the files are correct before you copy them over to your application? Also, if you curl http://localhost:3000/public/virtualbackground/tflite-1-0-0.js do you get the same response content?

I'm wondering if somewhere in your build process you're replacing the content of the tflite files with your index.html?

[charliesantos]

@NomioG base on your screenshots, the assets are not found by the video processors. The network tab is returning 200s but the content is a default html file, which is what your server is probably returning when it cannot find the file. Before setting the assetsPath, can you please try accessing the paths directly from the browser URL? Or just use curl as @sarahcstringer mentioned. Once you're able to access it, we'll figure out the rest of the errors.

[NomioG]

@sarahcstringer @charliesantos Thanks for the response. I thought that too. The thing confused me was that it looked like the files were found but the contents were different. But in reality the files weren't in there. I thought file existed in that folder based on the error.

As for the parcel, parcel serves the static files automatically if it's in the dist folder ( Dist here is your build outDir and I'm pretty sure you can set the build outputDir dynamically.

In case if others encountering this kind of issue when using [parcel, reactjs, @twilio/video-processors]

Do the following.

use the following script to copy the necessary files to dist directory. [build output dir] The script should be in your package.json

    "postinstall": "rimraf public && copyfiles -f node_modules/@twilio/video-processors/dist/build/* dist"

dist above in the script should be your build outDir.

Next, make sure the files are copied. When parcel serve the app, it's using the index.html file located in dist

As for the assetPath it must be relative to your index.html file in dist. If the index.html and tflite-1-0-0.js both located in dist directory, you need to set assetsPath as follows:

assetsPath: '/',

So your code should be something like this:

import * as Video from 'twilio-video'
import * as VideoProcessors from '@twilio/video-processors'

// ... some other codes
const videoTrack = await Video.createLocalVideoTrack({
    width: props.width,
    height: props.height,
    frameRate: 24
});
const bg = new VideoProcessors.GaussianBlurBackgroundProcessor({
                assetsPath: '/',
                maskBlurRadius: 10,
                blurFilterRadius: 5,
            });
await bg.loadModel();
videoTrack.addProcessor(bg);

// ... Rest of your code

Then use the blur or virtual background as described in the documentation.

Best regards