Add WebGL shim in CLI/Node context to allow GL-based modules to run in Node

[jywarren]

https://www.npmjs.com/package/gl

gl lets you create a WebGL context in node.js without making a window or loading a full browser environment.

This would no longer be pure JavaScript, but for some modules this is interesting. For example, the FisheyeGL module #27 can currently only be run in a browser, and webgl-distort #64 also would be this way.

Long-term project!

---

Hmm, maybe also these resources:

• non graphics card GL functions could be shimmed with https://github.com/Overv/JSGL
• or this: https://github.com/stackgl/headless-gl

[harshkhandeparkar]

So let me ask a few questions:

1. Are all of the pixels saved to a texture?
2. Do you specifically save them to a texture or are they required from the RAM on the go?
3. If you do save the data to a texture, there are some methods(i think) which return a buffer from the texture. I think you will have to explore the MDN docs.
4. In normal GLSL(the shader language), what you do is save the data to a texture, process the data on the GPU, get the data back from the gpu after rasterisation(converting to a bitmap). Now this data can be processed in many different ways. WebGL gives you a method to directly convert it to a DataURL but there may be other methods which return different things too. Try exploring.

[jywarren]

Also is it possible you need to use extra config on your node call like shown here, for it to work?

https://github.com/stackgl/headless-gl#how-can-headless-gl-be-used-on-a-headless-linux-machine

On Mon, Mar 25, 2019, 2:45 PM Jeffrey Warren jeff@unterbahn.com wrote:

Ah, readpixels- see the example?

https://github.com/stackgl/headless-gl#example

On Mon, Mar 25, 2019, 2:44 PM Jeffrey Warren jeff@unterbahn.com wrote:

Actually webgl is a variant of 2d canvas, so it always has a canvas context. Let me look a sec..

On Mon, Mar 25, 2019, 2:40 PM Slytherin notifications@github.com wrote:

I am a bit unsure about how to get the changed pixels out of the gl format like you can see the code above here. Like there we have the context from canvas but here we directly do have gl. @HarshKhandeparkar https://github.com/HarshKhandeparkar @jywarren https://github.com/jywarren Is there a way even in webGL that I can have the changed texture data as per the transformations without using canvas.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476327274, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ_JjiBtMD5fvx6oqy99wCPkk3DnYks5vaRgRgaJpZM4TceLs .

[harshkhandeparkar]

The links you provided don't seem to be working. @jywarren

[jywarren]

Oh really? i just tried, them can you try again? It also looks like they're recommending get-pixels, but not sure if that's after an image has been exported. But these links show some examples for fetching per-pixel data!

[harshkhandeparkar]

No luck. It just opens the repo homepage i guess but not the readme. Both of them. Maybe it is because I use my phone? You may never know, github breaks in a different way every new day.

[Divy123]

Well I saw how to get data out of texture but no success till now! Still exploring. Please do help if you find something @HarshKhandeparkar @jywarren .

[jywarren]

Ah yes, on mobile view it doesn't open the full readme. But skim down the readme, it's quite informative!

On Mon, Mar 25, 2019 at 3:24 PM Slytherin notifications@github.com wrote:

Well I saw how to get data out of texture but no success till now! Still exploring. Please do help if you find something @HarshKhandeparkar https://github.com/HarshKhandeparkar @jywarren https://github.com/jywarren .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-476342360, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJxEKl5MvfYD5oD8p75KRmpragcksks5vaSJYgaJpZM4TceLs .

[harshkhandeparkar]

Ok.

[harshkhandeparkar]

O don't know what this does but it is included in the readme. Is it helpful?

process.stdout.write(['P3\n# gl.ppm\n', width, " ", height, '\n255\n'].join('')) for(var i=0; i<pixels.length; i+=4) { for(var j=0; j<3; ++j) { process.stdout.write(pixels[i+j] + ' ') }

[Divy123]

I have looked into this but the main issue how to get the data out of texture.

[harshkhandeparkar]

What does the code i posted above do?

[harshkhandeparkar]

There is a getImageData() method but it is a method of CanvasRenderingext2D object.

[Divy123]

Its writing image or pixles data as a PPM formatted image.

[Divy123]

There is a getImageData() method but it is a method of CanvasRenderingext2D object.

Again with canvas.

[harshkhandeparkar]

WebGLRenderingContext object has a canvas property which will either return a offscreenCanvas element or null. Try this out.

[tech4GT]

https://github.com/GoogleChrome/puppeteer/issues/1260 This solves an issue I was earlier facing with the puppeteer implementation, Just adding it here since it may be useful!

[Divy123]

@HarshKhandeparkar this is still not available. @tech4GT hope you can elaborate on your idea of this.

[tech4GT]

Sure @Divy123 Anything in particular you want my help on?

[Divy123]

Like I tried using headless-gl for transforming webgl-distort to be node compatible. But the problem I am facing is due to no canvas being available and attached to the context, I can not get data out of a texture. Googled a alot but no success till now. But I know a bit about headless browser concept and would be delighted if you can help me with that like some basic steps that how can we use headless browser here. I am already exploring but it would be great if you help me out with the approach we can take with puppeteer. Thanks a lot!!

[tech4GT]

Oh, the approach I tried on this was nothing special as well, I was running the headless browser inside node and the module was running on the sequencer inside the headless browser!

[tech4GT]

The issue I posted earlier relates to the fact that by default gl is disabled in puppeteer. You need to explicitly enable it.

[tech4GT]

This is not an ideal solution though since we would want to avoid running the headless browser if possible(coz it'll take up a lot of resources)

[jywarren]

Even if it takes a lot of resources (starting up the browser) I still think we should move ahead with this implementation -- it's the closest I've seen to actually working so far. We could refactor and attempt another approach afterwards, but I think we have to try! Thank you all!!!

[jywarren]

Hi @Divy123 could you open a PR with your attempt so far? I'd like to try a few things.

For example, https://github.com/stackgl/headless-gl#example says we can access the pixels like:

//Create context
var width   = 64
var height  = 64
var gl = require('gl')(width, height, { preserveDrawingBuffer: true })

//Clear screen to red
gl.clearColor(1, 0, 0, 1)
gl.clear(gl.COLOR_BUFFER_BIT)

//Write output as a PPM formatted image
var pixels = new Uint8Array(width * height * 4)
gl.readPixels(0, 0, width, height, gl.RGBA, gl.UNSIGNED_BYTE, pixels)
process.stdout.write(['P3\n# gl.ppm\n', width, " ", height, '\n255\n'].join(''))
for(var i=0; i<pixels.length; i+=4) {
  for(var j=0; j<3; ++j) {
    process.stdout.write(pixels[i+j] + ' ')
  }
}

[tech4GT]

@jywarren I have an idea, maybe I can create a binding between gl and node-canvas and add the gl based functions on node-canvas using gl! I think this might be worth a shot.

[tech4GT]

If this works, it would allow for 100% code sharing between browser and node.

[tech4GT]

@jywarren @Divy123 I think I have figured this out, what I am doing is using 2 libraries: node-canvas and gl, and I have overriden the default canvas.getContext to return an instance of 'node-gl' Apart from that I needed some more changes in the fisheye-gl code and I think this might be ready. Now do you guys think we should modify jywarren/fisheyegl or I should bake the node-specific code in the module itself?

[jywarren]

Can you push what code you have and we can see what changes would be best in which module? Thanks!!!!

On Sun, Mar 31, 2019, 4:51 AM Varun Gupta notifications@github.com wrote:

@jywarren https://github.com/jywarren @Divy123 https://github.com/Divy123 I think I have figured this out, what I am doing is using 2 libraries: node-canvas and gl, and I have overriden the default canvas.getContext to return an instance of 'node-gl' Apart from that I needed some more changes in the fisheye-gl code and I think this might be ready. Now do you guys think we should modify jywarren/fisheyegl or I should bake the node-specific code in the module itself?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-478323673, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ07a33j86jwsMQncyolIYslkh85qks5vcHcJgaJpZM4TceLs .

[tech4GT]

@jywarren I have figured out a lot of things, can you explain a little about how this code is important though. https://github.com/jywarren/fisheyegl/blob/89d28f0dee943681192a708b59c076e3d5854652/src/fisheyegl.js#L204-L217 Since we don't have window available in node context, how should we change this code?

[tech4GT]

The current code is outputting a black image and I am unable to figure out why.

[jywarren]

I think we can skip this in node because it seems to be to limit the refresh of the library to the frame rate of the browser window refresh: https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame

Hmm, maybe try a simpler webgl task to be sure the basic idea works?

On Sun, Mar 31, 2019, 10:08 AM Varun Gupta notifications@github.com wrote:

The current code is outputting a black image and I am unable to figure out why.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-478345105, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ5r3Wmz8WRn8R9zw5nQ92-TCbsbOks5vcME0gaJpZM4TceLs .

[tech4GT]

Okay there is clearly some issue with the gl module, so for now I think we should proceed with the headless browser implementation. We can circle back to the other implementation once the issues are rectified. cc @jywarren @Divy123

[Divy123]

I agree @tech4GT . Exploring puppeteer nowadays to work on this. Also I request you and @jywarren to please review my SoC proposal : https://publiclab.org/notes/lit2017001/04-01-2019/soc-proposal-image-sequencer-v3-boosting-the-performance. Thanks!!

[jywarren]

Did you open an issue to ask more about it, and if so could you link there for reference? Thank you!

[jywarren]

I will review ASAP, apologies this has been taking a lot of time this week 😄 thanks for your patience!

[Divy123]

Thanks a lot @jywarren .

[tech4GT]

@jywarren here https://github.com/stackgl/headless-gl/issues/149

[jywarren]

oh awesome. Thanks.

On Thu, Apr 4, 2019 at 10:16 AM Varun Gupta notifications@github.com wrote:

@jywarren https://github.com/jywarren here stackgl/headless-gl#149 https://github.com/stackgl/headless-gl/issues/149

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/216#issuecomment-479915661, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ016CuB23ccHHGx4DiX3hvfjkthXks5vdgk6gaJpZM4TceLs .

[tech4GT]

@jywarren @Mridul97 So I was trying this out and I think the initial reload we do for the pwa results in puppeteer execution context being destroyed on sequencer.publiclab.org, I think we need to work around it somehow. That was a temporary solution anyway I guess.

[tech4GT]

@publiclab/is-reviewers Can someone point me to the link for dist files. I am trying to add image-sequencer.min.js as a script on a page.

[harshkhandeparkar]

We are getting help from @robertleeplummerjr in using headless-gl cc @tech4GT @jywarren @Divy123 @publiclab/is-reviewers

[robertleeplummerjr]

I don't have a ton of time I can offer, already on borrowed, but I can look into getting the fisheye project running on headless-gl, and then we can connect the dots backwards to this project, so it can be loaded in. Does that sound reasonable?

[jywarren]

Wow that is very generous, thank you! We are very interested in this bc our puppeteer based approach (running a whole headless Chrome) is not very efficient. We would /love/ to work with whatever you can figure out and work to generalize a narrow case.

For what it's worth, this is another WebGL module we use that has simpler parameters (four corner coordinate pairs): https://github.com/jywarren/webgl-distort

You can see them both running in the browser here with a standard input and output:

• https://sequencer.publiclab.org/examples/#steps=webgl-distort
• https://sequencer.publiclab.org/examples/#steps=fisheye-gl

Thank you!!!

[robertleeplummerjr]

If I use headlessgl like this:

const gl = require('gl');
const context = gl(1,1);
const getPixels = require('get-pixels');
const fisheyeGL = require('../../src/fisheyegl.js');
getPixels('example/images/grid.png', function(err, pixels) {
  if (err) {
    console.log(err);
    return;
  }

  fisheyeGL({
    image: pixels.data,
    getGLContext: () => context,
  });
});

And in fisheyegl, if I force the flow to use the pixels lib mentioned above, and I change line 157 from:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);

to:

gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);

I believe it works.

Here is the full diff I did:

diff --git a/src/fisheyegl.js b/src/fisheyegl.js
index db7369b..0117c63 100644
--- a/src/fisheyegl.js
+++ b/src/fisheyegl.js
@@ -41,7 +41,7 @@ var FisheyeGl = function FisheyeGl(options){
   var image = options.image || "images/barrel-distortion.png";

   var selector = options.selector || "#canvas";
-  var gl = getGLContext(selector);
+  var gl = options.getGLContext ? options.getGLContext() : getGLContext(selector);

   var shaders = require('./shaders');

@@ -172,7 +172,7 @@ var FisheyeGl = function FisheyeGl(options){

     gl.bindTexture(gl.TEXTURE_2D, texture);

-    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, img);
+    gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);

     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
     gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR); //gl.NEAREST is also allowed, instead of gl.LINEAR, as neither mipmap.
@@ -201,6 +201,13 @@ var FisheyeGl = function FisheyeGl(options){
     return texture;
   }

+  function loadImageFromPixels(gl, pixels, callback){
+    console.log('pixels', pixels);
+    var texture = gl.createTexture();
+    loadImage(gl, pixels, callback, texture);
+    return texture;
+  }
+
   function run(animate, callback){
     var f = window.requestAnimationFrame || window.mozRequestAnimationFrame ||
       window.webkitRequestAnimationFrame || window.msRequestAnimationFrame;
@@ -272,7 +279,11 @@ var FisheyeGl = function FisheyeGl(options){
     });
   }

-  setImage(image);
+  if (typeof image === 'string') {
+    setImage(image);
+  } else {
+    loadImageFromPixels(gl, image);
+  }

   // asynchronous!
   function getImage(format) {

[robertleeplummerjr]

You'll notice I only had an image size of 1x1 above, so not a practical solution, but proof you can load in the pixels.

[harshkhandeparkar]

Wow! Thanks @robertleeplummerjr @jywarren @tech4GT @VibhorCodecianGupta @Divy123