POST call using https.request returns 408

Maverick099 commented 7 months ago

Details

An overview on what the code is supposed to do:

I created a simple http request reverse proxy using express to to direct http/https requests to on-premise system for testing the application locally (connects to on-premise using VPN in local environment).

Proxy does not modify any headers sends it as is except the Accept-encoding header which it sets to utf-8, So I don't need to use any package to decode the incoming body (eg. gzip).
Proxy's backoff function is a simple exponential function if do_retry flag is enabled, which is not for testing scenario.

HTTP/s.request function implementation:

/**
 * Does HTTP/HTTPS request and returns a promise.
 * @param {{}} config Config object for the request.
 * @param {Buffer|String|{}} data Data to be sent in the request.
 * @param {Number} retries Number of retries to be done if request fails defualts to request_retries variable, @see{@link {request_retries}}.
 * @returns {Promise.<{status: Number|undefined,status_message:String|undefined,headers:http.IncomingHttpHeaders,body:any}>} resolve with  response or rejects with error.
 */
const request = (config, data = null, retries = request_retries) => {
  // eslint-disable-next-line no-unused-vars
  return new Promise((resolve, reject) => {
    try {
      // get the protocol agent
      const _agent = config.protocol === "https" ? https : http;

      delete config.protocol;

      let _data = [];
      for (let i = 0; i < retries; i++) {
        const req = _agent.request(config, (res) => {
          // collect the data
          res.on("data", (chunk) => {
            _data.push(chunk);
          });

          // create the response
          res.on("end", async () => {
            let _body;

            try {
              if (res.headers["content-type"]?.includes("application/json")) {
                // remove any JSON SYNTAXERROR wrt illegal whitespace.
                // _data = _data.map((chunk) => chunk.toString().replace(/[\n\r\s\t]+/g, " "));
                _body = JSON.parse(Buffer.concat(_data).toString("utf-8"));
              }
              //parse html for content type text/html
              else if (res.headers["content-type"]?.includes("text/html")) {
                _body = Buffer.concat(_data).toString("utf-8");
              } else {
                // check if header has encoding and use that to decode the buffer.
                _body = Buffer.concat(_data).toString(res.headers["content-encoding"] ?? "utf-8");
              }
            } catch (err) {
              _body = Buffer.concat(_data).toString();
            }

            const response = {
              status: res.statusCode,
              status_message: res.statusMessage,
              headers: res.headers,
              body: _body,
            };

            // check the condition for resolving the promise.
            if (res.statusCode >= 200 && res.statusCode < 300) {
              resolve(response);
            } else if (do_retry) {
              console.warn(`[${new Date().toISOString()}][PROXY] ${config.method} request to ${config.hostname ?? config.host + config.path} failed with status ${res.statusCode}.\nretrying...`);
              //call backoff and retry the request.
              await backoff(i);
            } else if (i === retries - 1) {
              resolve(response);
            } else {
              resolve(response);
            }
          });

          // timeout handler
          res.on("timeout", () => {
            reject(new Error(`Request to ${config.hostname ?? config.host + config.path} timed out.`));
          });

          res.on("error", (err) => {
            reject(err);
          });
        });

        // send the data depending on the type of data.
        if (data && data instanceof Buffer) {
          req.write(data);
        } else if (data && typeof data === "string") {
          req.write(data, "utf-8");
        } else if (data && typeof data === "object") {
          req.write(JSON.stringify(data), "utf-8");
        }
        // close the request
        req.end();
      }
    } catch (err) {
      reject(err);
    }
  });
};

Problem:

Able to successful do GET and HEAD calls to the on premise system, But with POST calls it fails with 408.

But when I try to do the same POST call using postman "directly", I do get a 201 status for the request.

So I tried to use postman to do the POST call "via" the proxy, to test if headers is missing and causing an 408 but I still get the same 408 error in this case as well.

So I am thinking the request function is not implemented properly.

whole code here

Node.js version

v19.8.1

Example code

Operating system

Windows, Linux

Scope

code

Module and version

http and https

preveen-stack commented 7 months ago

can you check the postman request if it is using any specific content encoding

Maverick099 commented 7 months ago

@preveen-stack No and anyways all the headers passed by postman or any client is passed as-is to via the proxy when doing the calls. Except for Accept-encoding header which is modified to accept only utf-8.

github-actions[bot] commented 1 month ago

It seems there has been no activity on this issue for a while, and it is being closed in 30 days. If you believe this issue should remain open, please leave a comment. If you need further assistance or have questions, you can also search for similar issues on Stack Overflow. Make sure to look at the README file for the most updated links.

github-actions[bot] commented 6 days ago

It seems there has been no activity on this issue for a while, and it is being closed. If you believe this issue should remain open, please leave a comment. If you need further assistance or have questions, you can also search for similar issues on Stack Overflow. Make sure to look at the README file for the most updated links.

nodejs / help