Open FunctionDJ opened 2 weeks ago
Readable.toWeb(sharpObject)
- appears to pipe some image data, but the served image appears to be broken/blank
Use of toWeb
is probably what I'd suggest people try first when converting. Are you able to provide a minimal code sample that allows someone else to reproduce the problem you ran into?
@lovell I'm using a large, 20MB PNG file for these tests, with only 1 request at a time. I haven't tested multiple/mass requests yet and it would make the reproduction more complicated but mass requests is my actual use case which is why i'm considering performance.
This first example works both when run with Deno and with Node and has equal performance between the two JS runtimes on average in my test case:
import http from "node:http";
import sharp from "sharp";
http
.createServer((_req, res) => {
const sharpObject = sharp("img.png").webp();
sharpObject.pipe(res);
})
.listen(8000);
This second example only works in Deno and is about 1% slower on average. More importantly, the response appears to be incomplete. Chromium will log GET http://localhost:8000/ net::ERR_INCOMPLETE_CHUNKED_ENCODING 200 (OK)
in the console and the bottom ~15% of the image appear as gray and Postman reports ~800KB response size vs. the 1MB response of the first example.
import sharp from "npm:sharp";
import { Readable } from "node:stream";
Deno.serve(() => {
const sharpObject = sharp("img.png").webp();
return new Response(Readable.toWeb(sharpObject) as ReadableStream);
});
With the original post and the 3x performance difference i've mentioned, i wasn't able to reproduce this today so i must have made a mistake in input files when i did the previous testing. Please excuse me.
This looks a bit like the problem reported at https://github.com/denoland/deno/issues/24606
If performance is of concern and memory is not a limiting factor, perhaps experiment with Buffers rather than Streams to avoid many small memory allocations.
I found that Deno issue too, but i'm not using the Fresh framework or http2 (request protocol is http/1.1
according to Chromium Devtools).
As for Buffers, i'm not aware that they can be streamed. If i use sharpObject.toBuffer()
, then the code will wait for the conversion to finish, keeping all data in memory for the moment (if i understood correctly).
I've also talked to contributors on the Deno side and expectedly they told me to ask on the package (sharp) side to look into this issue.
Bun is working with the following code
import sharp from "sharp"
import { Readable } from "node:stream"
import { Buffer } from "node:buffer"
Bun.serve({
fetch: async (req) => {
const svg = await req.text()
const sharpObject = sharp(Buffer.from(svg)).webp()
return new Response(Readable.toWeb(sharpObject) as ReadableStream, {
headers: {
'Content-Type': 'image/webp',
},
})
},
})
But it getting this error message for each request, not sure why it occurs
error: Premature close
code: "ERR_STREAM_PREMATURE_CLOSE"
at new NodeError (node:stream:244:20)
at node:stream:714:47
at emit (node:events:48:48)
at emit (node:events:48:48)
at endReadableNT (node:stream:2207:27)
Question about an existing feature
What are you trying to achieve?
I'm trying to convert a
Sharp
object as returned bysharp("...").webp()
to a web standardReadableStream<Uint8Array>
for compatibility with Deno.serve() /new Response(body)
When you searched for similar issues, what did you find that might be related?
I have searched a lot, e.g. in the "issues" tab on this repo, but couldn't find any truly related issue. (Aside from #4013 maybe) I managed to make it work with:
new ReadableStream
:new ReadableStream({ start(controller) { sharpObject.on("data", chunk => controller.enqueue(chunk) } })
- but it's slightly slower than the classicsharpObject.pipe(res)
you'd use withhttp.createServer
or expressReadable.toWeb(sharpObject)
- appears to pipe some image data, but the served image appears to be broken/blank and it's significantly slower (~3x) on a 20MB input file than.pipe(res)
or the previousnew ReadableStream
option (Note:toWeb
was added in Node v17 and is still marked as experimental)createReadableStreamFromReadable
from @remix-run/node works but is ~3x as slow as.pipe(res)
. They also use their ownnew ReadableStream
. This function can be used withimport { createReadableStreamFromReadable } from "@remix-run/node"
even when not using the Remix framework.Somewhat related: Stackoverflow posts about the difference between web standard ReadableStream and Node Readable
Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question
Please provide sample image(s) that help explain this question
independent of image file