Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node

jywarren commented 6 years ago

https://www.npmjs.com/package/gl

gl lets you create a WebGL context in node.js without making a window or loading a full browser environment.

This would no longer be pure JavaScript, but for some modules this is interesting. For example, the FisheyeGL module #27 can currently only be run in a browser, and webgl-distort #64 also would be this way.

Long-term project!

Hmm, maybe also these resources:

non graphics card GL functions could be shimmed with https://github.com/Overv/JSGL
or this: https://github.com/stackgl/headless-gl

Divy123 commented 5 years ago

Sorry for being late on the fisheyeGl issue but I was facing some issues which I am going through. Will soon come up with a solution.

tech4GT commented 5 years ago

Okay I brainstormed this a lot and I came up with 2-3 solutions

We can run a headless browser like puppeteer and run this module in it's context, then get the result using the callback.
We can try to use jsdom in conjunction with gl and make a global document object that can get a web-gl context. I have been trying this out but so far it looks like this would require some changes in the fisheyeGL library by @jywarren . Since the library uses new Image() and I was not able to find anything to emulate that in node, I tried using jsdom but it does not work with the constructor, maybe we can change this to
```
document.querySelector("#image-node") || new Image();
```
We can enable web-gl support in our modules by automatically requiring gl in node context. This would however mean that code for browser and node wold have to be different, which is what I am trying to avoid here. cc @jywarren thoughts?

tech4GT commented 5 years ago

Also there's an issue with node_version for gl It was compiled for node 8.15.0 latest, so If we go with this we would have to use that version Also please do see the warning on their npm page

Divy123 commented 5 years ago

@tech4GT @jywarren I have done some research and here are some points on this: First of all the gl produces the canvas element that does not have texImage2D function method on it . So that's a drawback I think. Also @jywarren can you please explain how can https://github.com/Overv/JSGL be useful here. While researching for fisheyeGl module I went through but there seemed no good resources to move with it. Also I came across basicHTML but not of much use as there canvas.getContext() not implemented. https://github.com/WebReflection/basicHTML/issues/32. I was trying to learn webGl so that I can be helpful with this . Can you people please suggest some good resource to go forward? And also puppeteer can be quite good to go with.

Divy123 commented 5 years ago

@jywarren request your help here.

jywarren commented 5 years ago

Hmm, to @tech4gt I think it's ok to have different code in browser or node if that code split exists only inside the module context. Although we could eventually try to generalize parts to make it easier to write webgl modules.

jywarren commented 5 years ago

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

jywarren commented 5 years ago

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <jeff@unterbahn.com wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

jywarren commented 5 years ago

Although we could likely generate a dataurl from puppeteer instead of screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <jeff@unterbahn.com wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

jywarren commented 5 years ago

It sounds reasonable, and we could try to improve on the loadtime later?

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

Although we could likely generate a dataurl from puppeteer instead of screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <jeff@unterbahn.com wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

jywarren commented 5 years ago

https://github.com/GoogleChrome/puppeteer/issues/1260

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

It sounds reasonable, and we could try to improve on the loadtime later?

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

Although we could likely generate a dataurl from puppeteer instead of screenshot

On Sun, Feb 17, 2019, 4:36 PM Jeffrey Warren <jeff@unterbahn.com wrote:

https://stackoverflow.com/questions/50051250/taking-a-screenshot-of-a-page-running-webgl-using-puppeteer

On Sun, Feb 17, 2019, 4:33 PM Jeffrey Warren <jeff@unterbahn.com wrote:

For jsgl I wonder if we could use this:

https://github.com/Overv/JSGL/blob/master/README.md#glperspectivefovy-aspect-near-far

And it has texImage it looks like. It might require a rewrite of fisheyegl though. Looks like it's OpenGL, not webgl?

Would puppeteer work? Gotta look into it.

Divy123 commented 5 years ago

GoogleChrome/puppeteer#1260

In that case it can't be.

jywarren commented 5 years ago

Ah but see here: https://github.com/GoogleChrome/puppeteer/issues/1260#issuecomment-348878456

On Sun, Feb 17, 2019, 4:40 PM Slytherin <notifications@github.com wrote:

GoogleChrome/puppeteer#1260 https://github.com/GoogleChrome/puppeteer/issues/1260 In that case it can't be.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464535727, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ5hDYfl-DMWiCuW6zIEBxYZTAxJuks5vOfZ0gaJpZM4TceLs .

Divy123 commented 5 years ago

Use at your own risk though, since @aslushnikov says Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

jywarren commented 5 years ago

Hmm, do they say why?

On Sun, Feb 17, 2019, 5:18 PM Slytherin <notifications@github.com wrote:

Use at your own risk though, since @aslushnikov https://github.com/aslushnikov says Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464542054, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJyMKGhYRvpz5y4v8a_K8SfsP97O3ks5vOf86gaJpZM4TceLs .

jywarren commented 5 years ago

I think potentially we just have to start trying things and see what we can get to work.

On Sun, Feb 17, 2019, 5:20 PM Jeffrey Warren <jeff@unterbahn.com wrote:

Hmm, do they say why?

On Sun, Feb 17, 2019, 5:18 PM Slytherin <notifications@github.com wrote:

Use at your own risk though, since @aslushnikov https://github.com/aslushnikov says Emulation.setDefaultBackgroundColorOverride doesn't work well.

What say on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-464542054, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJyMKGhYRvpz5y4v8a_K8SfsP97O3ks5vOf86gaJpZM4TceLs .

Divy123 commented 5 years ago

Sure @jywarren Not mentioned specifically something about it. I think that's the only way to work around right now.

tech4GT commented 5 years ago

@jywarren Are we open to modifying your fisheye-gl coed then? I can make the changes and put in a pr.

Divy123 commented 5 years ago

@tech4GT is there something I can help out with?

tech4GT commented 5 years ago

@Divy123 Sure! Once we finalize what path we want to follow here, I'll do some groundwork and then you can help me build on it :smile:

Divy123 commented 5 years ago

Sure thanks!

jywarren commented 5 years ago

Absolutely we can do work upstream!

Noting also that I'm really not sure what happens when you run WebGL code in Node environment on a server that has no GPU...?? But I'm asking someone at Google!

On Tue, Feb 19, 2019 at 8:44 AM Slytherin notifications@github.com wrote:

Sure thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-465212325, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ9DcGiU7M4XV1efthAKstaOUSGOiks5vPCm2gaJpZM4TceLs .

harshkhandeparkar commented 5 years ago

NOOOCL is a nodejs package which runs openCL (not GLSL) shaders in nodejs. https://github.com/unbornchikken/nooocl

jywarren commented 5 years ago

oh whoaaaaa cool!

On Tue, Mar 12, 2019 at 2:45 PM Harsh Khandeparkar notifications@github.com wrote:

NOOOCL is a nodejs package which runs openCL (not GLSL) shaders in nodejs. https://github.com/unbornchikken/nooocl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-472132423, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJz18jaKfB69RJtSTphu8AG8tEqSTks5vV_WZgaJpZM4TceLs .

harshkhandeparkar commented 5 years ago

Hello! Last commit for headless-gl https://github.com/stackgl/headless-gl was 4d ago. It will compatible with latest node versions. A new version was released recently and gpu.js also will use headless-gl for v2 instead of NOOOCL

harshkhandeparkar commented 5 years ago

NOOOCL has no release. Only a pre-release, so i don't recommend using it.

harshkhandeparkar commented 5 years ago

I would suggest going with headless-gl since it is maintained to work in all recent versions of node(thought the devs say that it is not actively developed, so no new features but compatibility)

Divy123 commented 5 years ago

Can you please provide some more details about headless-gl? I went through their doc but found no way to input an image? What do you think?

harshkhandeparkar commented 5 years ago

You can pass in an unit8Array or float32Array or any array. An example would be pixels.data array which has a size of width*height*channels. This array can be given to the readPixels method of the module. Also this module provides a full canvas context, so any canvas methods available in the browser should be available here. I hope this helps.

Divy123 commented 5 years ago

Thanks a lot @HarshKhandeparkar .

jywarren commented 5 years ago

Just be aware of what will require an actual GPU and could in theory fail on Travis or in the cloud (if there is no GPU on cloud servers, for example) or what would have a fall-back solution. But I don't really know how it works, so perhaps the best way would be to try it out!

On Mon, Mar 18, 2019 at 7:42 AM Slytherin notifications@github.com wrote:

Thanks a lot @HarshKhandeparkar https://github.com/HarshKhandeparkar .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473876386, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ6JKJBZI9T1-SGRIi7FpR-zg8QFmks5vX3t_gaJpZM4TceLs .

harshkhandeparkar commented 5 years ago

Does webgl in browser fallback to cpu mode?

harshkhandeparkar commented 5 years ago

@Divy123 @jywarren if everyone uses laptop or mac, I can try this stuff on my pc with gpu and then remove the gpu and test again as I have a custom built pc. @Divy123 you can do the initial work and open a pr or tell me your branch and I will test. If you have a custom pc which you can modify, you can try it out yourself. I'm going test out gpu.js with and without gpu anyways so I can also test this if needed. Shall I do that? Please note that I will only be available from april 16th to may 2nd or 3rd, after that I cannot do anything. So if you are going to start work on this, try to do it fast. Thanks.

Divy123 commented 5 years ago

Sure @jywarren @HarshKhandeparkar Applying this year for SoC, so can I include this in my summer proposal? I will be brainstorming now and do some initial work and putting this on hold till then, is that fine?

Divy123 commented 5 years ago

I have a list of points I want to implement this summer which includes this one and the web workers. I have already started writing the web assembly code and I think these three can be a great addition to the performance. @jywarren what do you think upon this?

jywarren commented 5 years ago

This all sounds great, we'd love to have this in a SoC proposal too. Just noting however that we ideally want to try to do the GPU modules /somehow/ before June, for MapKnitter purposes (webgl-distort module) but the other optimizations can happen a bit slower. Thanks, all!

Noting also - some CPUs have an integrated GPU. I wonder if they will fall back to that. I also wonder if Travis and other cloud-based systems skip the integrated GPU since they may be purpose-built for cloud computing, and if they would have no GPU at all. Finally i wonder if some libs may fall back to pure CPU-based calculations if there's nothing else. Some testing should help us figure all this out, but one of the major target platforms will be Google Cloud or Travis, so perhaps we should consider testing on those where we can. Thank you all!

On Mon, Mar 18, 2019 at 12:10 PM Slytherin notifications@github.com wrote:

I have a list of points I want to implement this summer which includes this one and the web workers. I have already started writing the web assembly code and I think these three can be a great addition to the performance. @jywarren https://github.com/jywarren what do you think upon this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473981334, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ-_0Vtmk-hWalM0rEvcIhfSZI5bEks5vX7ppgaJpZM4TceLs .

harshkhandeparkar commented 5 years ago

Uh, I don't know much about SoC. So do you want me to test this out? If yes, when will you be able to produce the (let's say) scaffolding for running gl headlessly? Btw my implementation of gpu.js will not take time and can be done in a few days. What I 'm waiting for is the next major release of gpu.js which will allow using gpu in node for GPGPU. See issue #849 for reference.

harshkhandeparkar commented 5 years ago

My cpu has a built in gpu, so I can test it on that and we can maybe benchmark it to see if it used the gpu or used the cpu. It can be run on travis which will mostly not have a gpu and we can test if it falls back to pure cpu computation which will mostly take more time than on gpu but it can also take less time since they will have a good cpu.

harshkhandeparkar commented 5 years ago

gCloud will probably have a gpu since it is also used for ML and other similar purposes.

Divy123 commented 5 years ago

@HarshKhandeparkar will explore it better and reach out to you soon. Also thank you soo much for your such a great help. I think we have to test it on your cpu. Also @jywarren I will start working on this part as soon as wasm part is done on an initial basis probably by the mid of april and will try to complete it within May itself .

harshkhandeparkar commented 5 years ago

Ok sure. 😊

harshkhandeparkar commented 5 years ago

Btw when is SoC going to start?

Divy123 commented 5 years ago

SoC will start from may 27 but work will never stop because ultimately it's all about our performance.

jywarren commented 5 years ago

:-) I love the energy! We do also want to be sure that everyone participates at a healthy pace that ensures a good life/work balance. Thank you so much!!!

On Mon, Mar 18, 2019 at 12:45 PM Slytherin notifications@github.com wrote:

SoC will start from 27 but work will never stop because ultimately it's all about our performance.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-473999747, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ1cVWFklPlf5HdRB2XiEVrXlxgBgks5vX8KfgaJpZM4TceLs .

Divy123 commented 5 years ago

Thank you @jywarren for making this a super awesome learning platform for us.

harshkhandeparkar commented 5 years ago

@jywarren chinmay was a gsoc student right? Why doesn't he contribute anymore? Sorry if I asked something wrong or somewhere wrong. I was just curious.

harshkhandeparkar commented 5 years ago

I'm unpinning this for the time being to oin the community check in

jywarren commented 5 years ago

Chinmay was, yes - he got a job and wasn't able to keep contributing - this happens sometimes (esp. due to work restrictions sometimes) but I'm sure he is very happy to see all the work on this project that he helped to kick off!

On Mon, Mar 18, 2019 at 1:48 PM Harsh Khandeparkar notifications@github.com wrote:

I'm unpinning this for the time being to oin the community check in

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474028266, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ4547iMw9wBibcJq464XL5NDH-C3ks5vX9FOgaJpZM4TceLs .

Divy123 commented 5 years ago

@jywarren staring with headlessGl here. Just one question should I start converting the fisheye module to test out this? Or is there something else I can continue with for this?

jywarren commented 5 years ago

Yes please if you could start with the webgl-distort library that would be great!

https://github.com/jywarren/webgl-distort

Cc @tech4gt ! 🎉🎉

On Tue, Mar 19, 2019, 10:48 AM Slytherin notifications@github.com wrote:

@jywarren https://github.com/jywarren staring with headlessGl here. Just one question should I start converting the fisheye module to test out this? Or is there something else I can continue with for this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-474404009, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ9TvHFDCtcqp5vuDUM7tHto3XOX3ks5vYPifgaJpZM4TceLs .

publiclab / image-sequencer

Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node #216