Get as raw pixels buffer

[unbornchikken]

If I need an RGBA buffer, the supported way of doing this is that I have to create a buffer, and iterate lwip image by using getPixel method, and put RGBA pixels in it. This is extremely inefficent. For each pixel lwip will give me an array* instance holding RGBA values, that means if i have a FullHD picture, runtime will allocate 2073600 arrays of size 4. That's 64 MBytes of raw temporary data, which is about 80MBytes when v8's bookeeping extra data added.

I can go with unsupported way, and iterate over internal lwip.buffer(), but that's gonna be more nice if you support lwip.buffer() acces in the public API.

* the documentation states getPixel's result is an object, but according to src/lwip/image.cpp line 88 it's an array.

[EyalAr]

getPixel returns an object, not an array (on the JS side. reference).

About doing it the manual way you described - you don't actually need to keep all those 2073600 objects in memory; just insert the values directly into the buffer you create and let those temporary objects be GC'd.

In any case, this issue is now labeled as a feature request.

[unbornchikken]

That's worse, because runtime will allocate this objects along arrays too. Allocating this ammount of dead objects in a loop (we're in a loop when iterating through pixels of the whole image), is an overkill for the GC. See: http://jayconrod.com/posts/55/a-tour-of-v8-garbage-collection

[Maweypeyyu]

I also did some perfomance testing, having a loop with

var channelStart = width*height*multiplier;  // multiplier depending on r,g,b or a = 0, 1,2 or 3
var buffer = _image.__lwip.buffer();

for(var y = 0; y < height; y++) {
    for(var x = 0; x < width; x++) {
        var index = x + y*width;
        //matrix[index] = _image.getPixel(x, y)[_channelName];
        matrix[index] = buffer[index + channelStart];
    }
}

extracting a channel off a 512x512 image took on my machine

1387.668 ms (using getPixel())
   5.260 ms (using the buffer)

Fortunately, I saw the buffer() trick here. If you don't want to add it to the public API, maybe you can add it to the readme/wiki?

Best wishes

[EyalAr]

I have no objection adding it to the API. Perhaps something like:

image.toBuffer('rgba', function(err, buffer){
    // ...
});

The arguments presented here seem reasonable to justify this feature (speed, overstraining the GC).

I feel that a lot of use cases which would seem to need direct buffer access, could actually be resolved with more managed and safe solutions for pixels access and manipulation.

Perhaps it would also be worth adding the following two methods to the API:

/**
 * Iterate the pixels in the image.
 * A faster alternative for getPixel.
 */
image.forEach(function(x, y, r, g, b, a){
    // ...
});

/**
 * Apply a custom filter on the image.
 * A faster alternative for getPixel and setPixel.
 */
image.filter(function(x, y, r, g, b, a){
    // The return value will set the pixel at x,y.
    // Should be an RGBA object    
});

Any thoughts?

[Maweypeyyu]

I absolutely like the idea of adding it to the toBuffer() method. In fact, that was the first thing I looked at and was surprised, that you can get encoded data only.

Personally, I don't see any advantage of having an image.forEach() function, but I like the filter method as it would be convenient for some simple filters.

Thanks for the fast reply :)

[unbornchikken]

+1 for all additions. Those will be handy indeed.

[EyalAr]

One thing to note - image.toBuffer('rgba', ...) will return a copy of the image's buffer, and not the underlying buffer itself.

[Maweypeyyu]

I think that's expected behaviour :)

[greg-hornby-roam]

So you can get a copy of a buffer using image.__lwip.buffer(), how then after modifying this buffer, can I create a new image object based on this modified buffer?

[Maweypeyyu]

Currently I don't know any way, but it might be an idea to add a constructor/lwip.create() function which accepts a buffer?

[eduardoboucas]

Any news on this? It'd be great to use image.toBuffer('rgba', ...).

Thanks! 👋

[willbamford]

Would be great to have this. Also it would be nice if image.toBuffer('rgba', ...) returned the buffer in ImageData format - this would be most compatible with WebGL and HTML5 Canvas i.e. rather than r0, r1, r2, ... , g0, g1, g2 ... it would be Uint8ClampedArray withr0, g0, b0, a0, r1, g1, b1, a1 ... and a max alpha value of 255 rather than 100. Currently having to do something like:

lwip.open(resource, 'png', (err, image) => {
  var w = image.width()
  var h = image.height()
  var buffer = image.__lwip.buffer()
  var data = new Uint8ClampedArray(buffer.length)
  var indexY = 0
  var index = 0
  var s = w * h
  var i = 0
  for (var y = 0; y < h; y += 1) {
    indexY = y * w
    for (var x = 0; x < w; x += 1) {
      index = indexY + x
      data[i + 0] = buffer[index]
      data[i + 1] = buffer[index + s]
      data[i + 2] = buffer[index + s * 2]
      data[i + 3] = 2.55 * buffer[index + s * 3]
      i += 4
    }
  }
})