Pipe image data to a stream

[bensmeets]

Hi, was this available already somehow? I've seen the examples of reading from a stream, but can't figure out how to write back to a stream.

To give some context. I've seen people use imagemagick in examples, and afterwards using "pipe" to get it back in their main process' stream. Hoped this would be a good alternative (without needing IM).

[EyalAr]

@bensmeets

I haven't added streams support to the API yet (pull requests are welcome!).

I'm not sure exactly what you need, can you clarify?

Basically, if you want to stream image data out, you can get the image as a buffer and use it as a basis for a stream (see here for example). If you want to get image data into your app from a stream, you can store it in a buffer and then use lwip.open to decode the buffer.

[bensmeets]

Thanks Eyal, I'm aware that my questions must be strange. I feel like I only know half of what I need to ask the right questions :)

I'll try to clarify a bit. I'm twiggling my thumbs with MeteorJS. Using a package for that called CollectionFS which helps with managing uploaded images. All of their examples are using ImageMagick, where I would much rather have something without dependencies like lwip.

Now. The CollectionFS has support for plugging in "transformations". Somewhere in this whole setup, something is not working like it should. I've played with Async libraries, wrappers, etc. all over the place. But still no go.

Their transformation functions have this signature:

function myTransformFunction(fileObj, readStream, writeStream);

So my thought was, heck just read in the readStream, do magic lwip stuff and write back to the writeStream.... not so easy though :)

Here is one of their examples using ImageMagick (just a snippet for the syntax):

transformWrite: function(fileObj, readStream, writeStream) {
  // Transform the image into a 10x10px thumbnail
  gm(readStream, fileObj.name()).resize('10', '10').stream().pipe(writeStream);
}

is something like this possible in lwip through this new stream support you mentioned? Or is this something for the future.

Cheers.

[EyalAr]

@bensmeets Based on the docs I'd try something like this:

1. Define your own readable stream.
2. Pipe it to the writable stream provided by CollectionFS.
3. Read data from the readable stream provided by CollectionFS into a buffer, until the stream is exhausted.
4. Once you have all the data in the buffer, decode it with lwip.
5. Use lwip to manipulate the image.
6. Encode the image to a buffer and emit the data your readable stream.

Pseudo code (not tested):

transformWrite: function(fileObj, readStream, writeStream) {

    // Create a new readable stream
    var middleStream = /* new stream.Transform() */;
    middleStream.pipe(writeStream);

    // temporary cache for original image data
    var oImBuf = new Buffer();

    // original image format (for decoder)
    var ext = fileObj.extension;

    // read original image from the stream
    readStream.on('data', function(chunk) {

        oImBuf = Buffer.concat(oImBuf, chunk);

    });

    // ready to decode original image
    readStream.on('end', function() {

        lwip.open(oImBuf, ext, function(err, img) {

            // do some manipulations...
            img.batch()
                .scale(0.5)
                .sharpen(10)
                // get buffer of new manipulated image as a JPEG
                .toBuffer('jpg', function(err, nImBuf){

                    // emit the new image to the middle stream
                    /* middleStream.push(nImBuf); */

                    // close the stream
                    /* middleStream.end(); */

                });

        });

    });

}

Note that:

1. We pipe our own read stream to the framework's write stream.
2. We publish data to our stream only when we finish manipulating the image.
3. I have commented out the parts related to our custom read stream because I'm not sure this is exactly how it works. It might be needed to create a class that implements a readable stream. But that shouldn't be very complicated.

Let me know if it helps.

[bensmeets]

@EyalAr Thanks, very grateful that you took the time for such a complete comment.

I've been struggling with it a lot and finally decided that the CollectionFS is too much black box material for me. It kept throwing errors at me when this was implemented. I'm switching to something more manual that gives some more insights.

I'm keep the above code put though! The manual alternative will need it as well. When I get it working there I'll shout out.

Ohhhh the good old days of synchronous code :)......

[bensmeets]

Just to keep things together, the issue was with the syntax for streaming, after switching to something more manual (regarding CollectionFS), everything went without a problem.

[EyalAr]

@bensmeets Awesome, thanks for the update :+1:

[wmassingham]

Any update on streams? I'm trying to get an image in through busboy, process it with lwip, then send it out to MongoDB GridFS. Busboy puts out a stream and GridFS prefers a stream, but lwip is in the middle with only buffers.

[EyalAr]

@wmassingham I'll try to target it to v0.0.7. In the meantime, perhaps you can try to adapt the pseudo code I posted above.

Also, it would be good if you can post a sample of how you see streams implemented. If it was available, how would you use it?

Thanks.

[bencooling]

For what its worth this solution worked for me

lwip.open(filename, function (err, originalImage) {

  if (err) {
    throw err;
  }

  originalImage
    .batch()
    .resize(200, 200)
    .toBuffer('jpg', function (err, buffer) {

      var bufferStream = new stream.PassThrough();
      bufferStream.end( buffer );
      // bufferStream.pipe( process.stdout );

    });
});