search

greggman / virtual-webgl

Virtualize WebGL Contexts
Other
148 stars 12 forks source link

Can this be used for connecting three.js with TensorFlow.js? #2

Open EliasHasle opened 4 years ago

EliasHasle commented 4 years ago

tfjs is a JS version of TensorFlow, and successor to the experimental deeplearn.js. When WebGL is available, tfjs will accelerate operation using the GPU. tfjs has functions to create a tensor from a pixel source (with similar restrictions as texImage2D), and to create an image from a tensor.

I would like to combine it with three.js, e.g. to capture "sensory" data from three.js graphics and, conversely, augment three.js graphics with data from tfjs.

Do you think this library would be suitable for such communication, when the goal is increased performance? Or is it better to stick with communication through canvas textures?

greggman commented 4 years ago

Why don't you just try it?

I have no idea if tensorflow supports WebGL1. You'll need to prevent it from trying to use WebGL2 since this library doesn't support WebGL2 by using

  virtualWebGL.setup({
    disableWebGL2: true,
  });
EliasHasle commented 4 years ago

OK. Thanks. I may actually try it (or rather make some limited experiments with it). I suspect many small overheads from the virtualization will add up to something that may be just as bad as the expected overhead from communicating through canvas textures. But it would depend on the application. When (if) I have done some experiments, I will post an update here.

This work is very interesting anyway. :+1:

segments-tobias commented 6 months ago

@EliasHasle Did you ever experiment with something like this?

EliasHasle commented 6 months ago

@EliasHasle Did you ever experiment with something like this?

No, regrettably.