Can I pipe an `IncomingMessage` into `needle`?

[brianjenkins94]

Using request I was able to functionally reverse proxy a request through Express.js with the following:

import requestJs from "request";

app.all("/inbound/*", function(request, response) {
    request.pipe(requestJs({
        "url": "http://localhost:3000" + request["_parsedUrl"]["_raw"].substring("/inbound".length)
    })).pipe(response);
});

I'm having trouble accomplishing the same using needle:

import { request as needle } from "needle";

app.all("/inbound/*", function(request, response) {
    request.pipe(
        needle(request.method, "http://localhost:3000" + request["_parsedUrl"]["_raw"].substring("/inbound".length)
    )).pipe(response);
});

Since I'm just piping, I figured these two snippets would be identical, but the latter just responds with what it receives.

Any ideas?

[tomas]

Try replacing needle with needle.request in the second example.

[brianjenkins94]

No dice, I'm already pulling request out of needle in the import.

I'll see if I can put together a minimal sample that reproduces the issue.

[brianjenkins94]

Basically this:

// $ npm install express needle

const express = require("express");
const needle = require("needle");

const app = express();

app.all("/api/proxy/*", function(request, response) {
    request.pipe(
        needle.request(request.method, "https://www.wikipedia.org" + request["_parsedUrl"]["_raw"].substring("/api/proxy".length)
    )).pipe(response);
});

app.listen(8888, function() {
    console.log("Listening on port " + this.address().port);

    needle.get("http://localhost:8888/api/proxy/portal/wikipedia.org/assets/img/Wikipedia-logo-v2@2x.png", function(error, response, body) {
        if (error !== null) {
            console.error(error);
        } else {
            console.log(body);
        }
    });
});

Although now I'm getting Error [ERR_STREAM_WRITE_AFTER_END]: write after end which I haven't quite pinned down yet. because we're recieving an empty buffer from the "reverse proxy".

...but why?

[brianjenkins94]

I think the problem is here:

request.pipe(needle.request(request.method, url);

because this works fine:

needle.request(request.method, url).pipe(response);

I should be able to pipe an IncomingMessage into needle, right?

[brianjenkins94]

Is this a feature request then?

[tomas]

Yes is is!

And feel free to submit a PR if you're up for it. :)

[brianjenkins94]

As a new export named proxy or do you think there's a way to keep the current API but add in piping?

[tomas]

Whatever feels more natural. I think the first option makes more sense, right?

request.pipe(needle.post('some.url'))

[brianjenkins94]

So I would need to modify this bit:

https://github.com/tomas/needle/blob/master/lib/needle.js#L795-L803

but for this to work Needle itself would need to be an instance of WritableStream.

[tomas]

I think we'd need to return a Writable/Duplex stream instead of the Passthrough we're returning now.

And then have that stream pass whatever is written to it to the internal request, somewhere around here.

[rysi3k]

Hello, any news about this? I have similar problem :)

[tomas]

@brianjenkins94 Need any help?

[brianjenkins94]

I haven't been working on this. Would love to see it added though.

[brianjenkins94]

BountySource

[ThisSentenceIsFalse]

I think we'd need to return a Writable/Duplex stream instead of the Passthrough we're returning now.

And then have that stream pass whatever is written to it to the internal request, somewhere around here.

I feel like I have gone astray with this. What sort of semantics do you aim for in the case of redirection? I can see a few different situations:

• piping allowed in all situations, while honouring the redirection config: the original stream would be consumed by the first request - buffering the contents would be necessary to handle repeats robustly. Probably tricky to implement across plaftorms to be resilient to unexpected crashes/interference, without cancelling the benefits brought by the Streams API (garbage collector efficiency, handling large amounts of data). User is blissfully unaware of the implementation. I suppose it is rare as a paradigm, but IDK. Multiple implementation stages needed.

• piping allowed only when redirection is false: the limitation is formally specified in the API, implementation is straightforward as you described it above, API is simple and familiar, if slightly limited. One or two commits would solve it.

• piping allowed, honouring redirection, user-provided config: config option allows user to provide some buffering logic of his own liking; being aware of whatever mess he/she might be leaving behind. Defaults to simple memory buffer or whatever floats your boat, which would cover most common cases; flexible, potentially robust, but moderately compromises the library's simplicity. Multiple implementation stages required.

Second option seems to be best for a short addition? Worst case, redirection is handled by the user.