search

tensorflow / tfjs

A WebGL accelerated JavaScript library for training and deploying ML models.
https://js.tensorflow.org
Apache License 2.0
18.47k stars 1.93k forks source link

Support use in Worklets, specifically AudioWorklets #4704

Open space-pope opened 3 years ago

space-pope commented 3 years ago

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

Describe the feature and the current behavior/state.

A version of tfjs that works within WorkletGlobalScope (for my use case, AudioWorkletGlobalScope) would be ideal for processing data off the main thread. Network requests are disabled in this scope, and the global object (self) is undefined. The latter prevents a tfjs distribution from even being imported in this scope, and the former would prevent functions like loadGraphModel from working, as they rely on fetching data over the network.

In addition, TensorFlow and models can't be loaded on the main thread and transferred to an existing WorkletNode via MessagePort since the objects of course contain functions and hence can't be cloned.

Will this change the current api? How?

Yes, in that model loading functions will need to be altered/added so that they accept prefetched data. I'm unclear on how much altering the use of self would change the API, or even how feasible that is.

Who will benefit with this feature?

Anyone looking to process data off the main thread in a Worklet. In my case, I'd like to do realtime audio processing using TensorFlow models in an AudioWorklet.

Any Other info.

rthadur commented 3 years ago

cc @pyu10055 @lina128

gaikwadrahul8 commented 1 year ago

Hi, @space-pope

Apologize for the delayed response and we are re-visiting our older feature requests and checking whether those feature requests implemented or not as of now and I seems like we haven't implemented this feature as of now if I'm not wrong so May I know are you still looking for this feature in TFJs ?

If someone wants to contribute for this feature then you're always welcome and please feel free to do and please refer these links Ref-1, Ref-2 . Thank you!

space-pope commented 1 year ago

Thanks for following up. I haven't been working with TFJS recently, but I still think it would be a valuable feature for those who are. I haven't been following the library's progress for awhile, so maybe there are other ways to process data in the background now?

Elitebigboss90 commented 1 year ago

It's an important feature for audio processing.

audioContext.createScriptProcessor is deprecated and soon will be gone. And that is the only way we can use tfjs for audio processing.

Right now the recommended method is AudioWorkletNode. And that is lack of full DOM and Web API access in the worklet environment.

I'd like to contribute to it. But I need some pointers so I will know where to look at.