search

koajs / koa

Expressive middleware for node.js using ES2017 async functions
https://koajs.com
MIT License
35.21k stars 3.23k forks source link

Can't pipe stream to ctx.res #944

Closed knowlet closed 7 years ago

knowlet commented 7 years ago

I'm trying to pipe streams to my response but it seems didn't work

Simple test case:

const Koa = require('koa');
const app = new Koa();
const Router = require('koa-better-router');
const router = Router().loadMethods();
const request = require('request');

app.use(async (ctx, next) => {
    const start = new Date();
    await next();
    const ms = new Date() - start;
    console.log(`${ctx.method} ${ctx.url} - ${ms}ms`);
});

router.get('/', async (ctx) => {
    ctx.req.pipe(request(`https://www.google.com/?q=${ctx.query.q}`)).pipe(ctx.res);
});

app.use(router.middleware());

app.listen(3000, () => {
    console.log('listening on port 3000');
});

An error occurred:

$ node index.js 
listening on port 3000
GET / - 35ms
internal/streams/legacy.js:59
      throw er; // Unhandled stream error in pipe.
      ^

Error: write after end
    at ServerResponse.write (_http_outgoing.js:450:15)
    at Request.ondata (internal/streams/legacy.js:16:26)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:189:7)
    at IncomingMessage.<anonymous> (/Users/knowlet/test/node_modules/request/request.js:1088:12)
    at emitOne (events.js:96:13)
    at IncomingMessage.emit (events.js:189:7)
    at IncomingMessage.Readable.read (_stream_readable.js:381:10)
    at flow (_stream_readable.js:761:34)
    at resume_ (_stream_readable.js:743:3)

And req.pipe(request(...)).pipe(res) works in express.js, I'm wondering is there anything I do wrong?

$ node -v
v7.6.0
PlasmaPower commented 7 years ago

You'll want to let Koa handle the stream:

ctx.body = ctx.req.pipe(request(`https://www.google.com/?q=${ctx.query.q}`));
knowlet commented 7 years ago

It works, thanks for answering.

ChristianUlbrich commented 7 years ago

ctx.body = request(`https://www.google.com/?q=${ctx.query.q}`); will also work.

PlasmaPower commented 7 years ago

I think OP might have wanted to stream the request and response (so POST data would be forwarded).

catamphetamine commented 7 years ago

@PlasmaPower

You'll want to let Koa handle the stream:

Why? What for? ctx.res is a standard HTTP response object so there shouldn't be any need for anything else

PlasmaPower commented 7 years ago

@catamphetamine Koa can be seen as a wrapper around the Node.JS HTTP APIs. Using those APIs directly bypasses Koa. Writing to Node.JS HTTP objects managed by Koa is likely to conflict with Koa, as it does in this case. If you don't want Koa to touch the response, you can use ctx.respond = false, which also makes OP's example work perfectly. However, that is not the recommended solution, and it may cause problems with other middleware.

catamphetamine commented 7 years ago

you can use ctx.respond = false

Oh, you have that. Ok, that will do.

However, that is not the recommended solution, and it may cause problems with other middleware.

How about this

.use(async (ctx) =>
{
    // https://medium.com/@aickin/whats-new-with-server-side-rendering-in-react-16-9b0d78585d67
    ctx.res.write(header)
    await pipe(stream, ctx.res, { end: false })
    ctx.res.write(footer)
    ctx.res.end()
})

// Pipes `from` stream to `to` stream.
// Returns a `Promise`.
function pipe(from, to, options)
{
    return new Promise((resolve, reject) =>
    {
        from.pipe(to, options)
        from.on('error', reject)
        from.on('end', resolve)
    })
}
PlasmaPower commented 7 years ago

That should work, as Koa checks ctx.res.writable. You could also use the combined-stream package. Create a combined stream, append the header and footer, and then set ctx.body to the combined stream.

catamphetamine commented 7 years ago

Ok, thanks

jamesbechet commented 7 years ago

@PlasmaPower I've tried to play around with combined-stream but I'm struggling with it.

Following @catamphetamine's example, I can't get this to work:

const header = '<html><head></head><body>'
const footer = '</body></html>'
const combinedStream = CombinedStream.create()
combinedStream.write(header)
// https://medium.com/@aickin/whats-new-with-server-side-rendering-in-react-16-9b0d78585d67
const stream = renderToNodeStream()
combinedStream.append(stream)
combinedStream.pipe(ctx.body, { end : false }) // Triggers an error 'on' is not defined
combinedStream.write(footer)

I know I should let ctx.body handle the stream, but I'm not sure how in that case. ctx.body = combinedStream.pipe(.. ?

catamphetamine commented 7 years ago

@saxofficial Try multistream

import string_stream from 'string-to-stream'
import multi_stream from 'multistream'

ctx.status = 200
ctx.body = multi_stream
([
    string_stream(before_content),
    typeof content === 'string' ? string_stream(content) : content,
    string_stream(after_content)
])

It works as part of react-isomorphic-render.

jamesbechet commented 7 years ago

@catamphetamine thanks, that works 👍

lzr900515 commented 5 years ago

@catamphetamine let combinedStream = CombinedStream.create(); combinedStream.append(htmlTemplate[0]); combinedStream.append(stream); combinedStream.append(htmlTemplate[1]); ctx.body = combinedStream; it works~

don't use combinedStream.write directly. the API document said.

zachsa commented 3 years ago

For anyone using node-fetch. This seems to work:

const response = await fetch(...)
ctx.body = response.body
wengeezhang commented 3 years ago

You'll want to let Koa handle the stream:

ctx.body = ctx.req.pipe(request(`https://www.google.com/?q=${ctx.query.q}`));

ctx.req is a readable stream, so ctx.req.pipe(request(https://www.google.com/?q=${ctx.query.q})) will return the outgoing req--"request(https://www.google.com/?q=${ctx.query.q})".

Finally, koajs will pipe ctx.body to ctx.res, this will trigger error: OutgoingMessage should be write-only. Piping from it is disabled.

tataue commented 2 years ago

For anyone using node-fetch. This seems to work:

const response = await fetch(...)
ctx.body = response.body

why native fetch not ok?

alanhe421 commented 1 year ago

You'll want to let Koa handle the stream:

ctx.body = ctx.req.pipe(request(`https://www.google.com/?q=${ctx.query.q}`));

ctx.req is a readable stream, so ctx.req.pipe(request(https://www.google.com/?q=${ctx.query.q})) will return the outgoing req--"request(https://www.google.com/?q=${ctx.query.q})".

Finally, koajs will pipe ctx.body to ctx.res, this will trigger error: OutgoingMessage should be write-only. Piping from it is disabled.

ERR_STREAM_CANNOT_PIPE

koajs@2.13.1 nodejs@12.18.3

  ctx.body = ctx.req.pipe(request({
          headers, host: `${codeServerProxifier.host}`, path: url, method: ctx.method, // data: ctx.request.body,
          agent: httpAgent, timeout: 6000,
        }));
owendswang commented 8 months ago

'request' package has deprecated. Can I know how to use 'axios' to stream response ctx.body?