Request failed with status code 429 when use in remote/cloud environment

[taroj1205]

I am using rettiwt.user.details(id) and I am getting the error below.

{
"message": "Request failed with status code 429",
"name": "AxiosError",
"stack": "AxiosError: Request failed with status code 429\n    at eL (/var/task/.next/server/chunks/151.js:16:36057)\n    at Unzip.<anonymous> (/var/task/.next/server/chunks/151.js:18:8452)\n    at Unzip.emit (node:events:529:35)\n    at endReadableNT (node:internal/streams/readable:1368:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)",
"config": {
"transitional": {
"silentJSONParsing": true,
"forcedJSONParsing": true,
"clarifyTimeoutError": false
},
"adapter": [
"xhr",
"http"
],
"transformRequest": [
null
],
"transformResponse": [
null
],
"timeout": 0,
"xsrfCookieName": "XSRF-TOKEN",
"xsrfHeaderName": "X-XSRF-TOKEN",
"maxContentLength": -1,
"maxBodyLength": -1,
"env": {},
"headers": {
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/x-www-form-urlencoded",
"authorization": "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",
"User-Agent": "axios/1.4.0",
"Accept-Encoding": "gzip, compress, deflate, br"
},
"method": "post",
"url": "https://api.twitter.com/1.1/guest/activate.json",
"data": null
},
"code": "ERR_BAD_REQUEST",
"status": 429
}

[Rishikant181]

Status code 429 represents 'too many requests'. You might have hit the standard rate limits that comes with using Twitter web.

[taroj1205]

Status code 429 represents 'too many requests'. You might have hit the standard rate limits that comes with using Twitter web.

That was my first request for today.

[Rishikant181]

Can you provide the id whose details you were trying to fetch? It will help in trying to reproduce the error.

[taroj1205]

Can you provide the id whose details you were trying to fetch? It will help in trying to reproduce the error.

id: taroj1205

Here is my code:

export const profile = async (id: string) => {
  const rettiwt = new Rettiwt();

  try {
    const details = (await rettiwt.user.details(id)) as User;
    if (details.profileImage) {
      details.profileImage = details.profileImage.replace('_normal', '');
    }
    console.log(details);
    return details;
  } catch (error) {
    console.log(error);
    return error;
  }
};

[Rishikant181]

Strange, this works as expected on my system. Are you able to access Twitter Web from you system?

[taroj1205]

Strange, this works as expected on my system. Are you able to access Twitter Web from you system?

Yes, I can access it. Just to check, does this by any chance only support locally?

[Rishikant181]

I have always worked on it locally. It has not been tested in a remote environment, unfortunately.

Are you perhaps running this from a remote/cloud environment?

[taroj1205]

Yes, I tried to run this with Next.js hosted on vercel.

[Rishikant181]

Are you using a proxy such as cors-anywhere, specifically the publicly available cors-anywhere instance?

[taroj1205]

I'm not using a proxy. Should I? Here is my GitHub repo https://github.com/taroj1205/nextjs-twitter/blob/main/src/lib/twitter.tsx

[Rishikant181]

I'll be checking it on my Vercel deployment.

Sorry for the inconvenience :(

[sonhavietnamese]

I am using next@14.0.4, this is my working solution:

// app/actions/get-details.ts
import { Rettiwt } from 'rettiwt-api'

const rettiwt = new Rettiwt()

export const getFollowers = async () => {
  const res = await rettiwt.user.details('heysonha')

  console.log(res)
}

// app/page.tsx
import { getFollowers } from '@/app/actions/get-details'

export default async function Page() {
  await getFollowers()

  return <div>page</div>
}

[taroj1205]

I'm also using next@14.0.4, and I tried renaming the file to .tsx->.ts. It works perfectly in localhost but it still does not work (code 429) on vercel deployment.

[Rishikant181]

After testing, I can confirm that it does not work from remote environments. I'll keep this issue pinned and will be actively working on it.

[Rishikant181]

Found the culprit.

When generating a guest token using rettiwt-auth package, the token is generated by sending a POST request to the endpoint https://api.twitter.com/1.1/guest/activate.json . This is an endpoint of the older v1.1 Twitter API and there is a problem with it where it throws a BAD_REQUEST error when the request is made using Axios or NodeJS Fetch from a cloud environment. If however, the request is made using something such as node-libcurl, the request works as expected.

@omarcinkonis Can you help me out on this?

Edit 1: After some more testing, it seems I was wrong and node-libcurl too doesn't work.

[omarcinkonis]

As I understand, node-libcurl works from cloud? If so, I'm guessing that it adds different headers to the request. Check how the libcurl requests differ and it should be possible to solve the problem.

[Rishikant181]

@omarcinkonis One more thing to add, this problem only happens with the older v1.1 API endpoints. The v2 API endpoints work flawless.

[omarcinkonis]

I'm not using Next/Vercel, so I'm not willing to take this issue. I can only consult.

As I said, I suggest checking headers sent by node-libcurl (given that it works in this scenario). Since we have a POST request, Content-Type: application/x-www-form-urlencoded may be required (blind guess). I faced an issue with this header on Twitter elsewhere.

I'm not sure why there may be a difference using local/cloud instance.

[Rishikant181]

After extensive testing, I came to the following conclusions:

• As we know, fetching data without logging in uses something called a guest_token, which is fetched from the /1.1/auth/activate.json endpoint.
• This endpoint belongs to the v1.1 API of Twitter (which has been deprecated as of now, but is still used by the Twitter Web-App). The v1.1 API blocks all connections from IP ranges belonging to cloud service providers, and throws an error 429.
• Now, for fetching data, v2 API endpoints are used, which does not block connections from IP ranges belonging to cloud service providers.
• As such, when an application is deployed to a cloud environment, interacting with the API without logging in is not possible.

Therefore, @taroj1205 your only option as of now is to generate an API_KEY and use 'user' authentication instead of 'guest' authentication, when you are deploying your application to the cloud.

Thanks for you patience and feedback!

[Rishikant181]

Okay the issue has been resolved in Rettiwt-Auth.

What was the solution? The solution was simply using a proxy but the issue was that the proxy used with Rettiwt-API was not passed to Rettiwt-Auth for authentication, since Rettiwt-Auth did not have any means to use a proxy (dumb me :P). Now that Rettiwt-Auth has an option to use proxy in the alpha build, Rettiwt-API can now be made to use a proxy for authentication too, thereby allowing it's usage from within cloud platforms.

I'll be keeping this issue open, at the same time, you can track the bug here.