Setting Proxy URL in Next.js API routes

[kaf-lamed-beyt]

I'm using this package in my Next.js App, specifically as an API route.

Locally, everything works fine, but when I deploy to Vercel... I keep getting 500: Internal server error. So I decided to include the proxyUrl parameter... But the error still persists.

Here's what my API route looks like:

import { NextApiRequest, NextApiResponse } from "next";
import { Rettiwt } from "rettiwt-api";

type TweetId = string | string[] | undefined;
const retweetInstance = new Rettiwt({
  proxyUrl: new URL(`${process.env.NEXT_PUBLIC_PROXY_URL}/api/tweet`),
});

export default async function tweetApiRoute(
  req: NextApiRequest,
  res: NextApiResponse
) {
  const tweetId: TweetId = req.query.tweetId;
  const response = await retweetInstance.tweet.details(tweetId as string);

  res.status(200).json(response);
}

And here's the error log from Vercel.

ive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 60,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0
    },
    _header: 'POST /1.1/guest/activate.json HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/x-www-form-urlencoded\r\n' +
      'authorization: Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA\r\n' +
      'User-Agent: axios/1.6.3\r\n' +
      'Accept-Encoding: gzip, compress, deflate, br\r\n' +
      'x-vercel-id: cdg1::hrklj-1709905768948-d1a9c84a78bd\r\n' +
      'Host: api.twitter.com\r\n' +
      'Connection: close\r\n' +
      'Content-Length: 0\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: HttpsProxyAgent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      options: [Object],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype] {},
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 0,
      proxy: [URL],
      proxyHeaders: {},
      connectOpts: [Object],
      [Symbol(kCapture)]: false,
      [Symbol(AgentBaseInternalState)]: [Object]
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    joinDuplicateHeaders: undefined,
    path: '/1.1/guest/activate.json',
    _ended: false,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: false,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: true,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'Bad Request',
      client: [Socket],
      _consuming: true,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://api.twitter.com/1.1/guest/activate.json',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 16,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'api.twitter.com',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://api.twitter.com/1.1/guest/activate.json',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      authorization: [Array],
      'user-agent': [Array],
      'accept-encoding': [Array],
      'x-vercel-id': [Array],
      host: [Array]
    },
    [Symbol(errored)]: null,
    [Symbol(kHighWaterMark)]: 16384,
    [Symbol(kRejectNonStandardBodyWrites)]: false,
    [Symbol(kUniqueHeaders)]: null
  }
}
Error: Runtime exited with error: exit status 1
Runtime.ExitError

[Rishikant181]

Can you check if the proxy is working or not from you local system?

[kaf-lamed-beyt]

Hi @Rishikant181,

It isn't working locally. It throws the internal server error.

It only works when I do not use the Proxy URL

[Rishikant181]

Is the proxy working and reachable?

[kaf-lamed-beyt]

I don't think so. It doesn't work in prod. Only locally. And that's if I do not add the proxyUrl arg

[Rishikant181]

Then it seems like it's something to do with the proxy server you are using. Can you try it with some other proxy?

[kaf-lamed-beyt]

Oh! What alternatives would you recommend?

[Rishikant181]

And that's if I do not add the proxyUrl arg

What proxy were you using earlier?

[kaf-lamed-beyt]

And that's if I do not add the proxyUrl arg

What proxy were you using earlier?

I was using my app's base URL withe API route appended like so:

const retweetInstance = new Rettiwt({
  proxyUrl: new URL(`${process.env.NEXT_PUBLIC_PROXY_URL}/api/tweet`),
});

I even tried setting the proxy in package.json like so:

  "proxy": {
    "/api/tweet": {
      "target": "https://api.twitter.com/1.1",
      "changeOrigin": true
    }
  },

Now, it throws the 429 error when I checked the stack trace from my logs. I'm going to try this http-proxy-middleware package and let you know if it works or not.

[Rishikant181]

I was using my app's base URL withe API route appended like so

Ahh I see, that's the issue. You misunderstood the proxyUrl argument.

The proxyUrl arguments is actually used to set the URL of the proxy server, which will be making HTTPS requests to Twitter on your behalf. You see, Twitter Web App API block requests from cloud service providers. In order to bypass this restriction, another publicly available computer is used, through which your requests to Twitter will be routed, which will fool Twitter into thinking that the HTTP requests come from a 'normal' computer and not a cloud service provider's computer. This is called proxy-ing. You can read more about it here.

In this case, you have to use a separate proxy server (there are paid as well as freely available ones on the internet, a google search will help). After choosing a proxy server, you pass that proxy server's full url to proxyUrl arg.

[kaf-lamed-beyt]

Aha! I see.

Thank you for this detailed clarification. Quick question though... the target API endpoint is: "https://api.twitter.com/1.1", yeah?

[Rishikant181]

Yes, but you need not concern yourself with that. Everything is handle by rettiwt-api internally and all you need to do is use a proxy server. For example, if we are using a publicly available, free proxy server with IP address xxx.xxx.xxx.xxx with port yyyy, the configuration will be:

const rettiwt = new Rettiwt({ proxyUrl: new URL('http://xxx.xxx.xxx.xxx:yyyy') })

That's all is needed.

[Rishikant181]

Instead of proxyUrl, you may use just authProxyUrl as described here

If authProxyUrl doesn't help, then you may need to use a full proxy with proxyUrl.

[kaf-lamed-beyt]

Niceee!! I'll try getting a free proxy url and let you know the outcome

[kaf-lamed-beyt]

@Rishikant181, In the meantime... can you recommend where I can get proxy URLs? I tried using one from this place, but I got a refused connection error

  cause: Error: connect ECONNREFUSED 36.89.129.213:5678
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '36.89.129.213',
    port: 5678
  },

[Rishikant181]

Try multiple, whichever one works for you (that's how even I do it :P)

[kaf-lamed-beyt]

I found one after several trials! Thanks for your help.

For anyone who might run into this same issue in the future, here's a repo with an exhaustive (I think) list of proxy servers.

[Rishikant181]

Glad it worked out!