Rate limit exceeded

[FardinAhsanSakib]

I am trying to fetch tweets by tweet ids and getting the following output: Rate limit exceeded, waiting for 30 seconds... Rate limit exceeded, waiting for 30 seconds... Rate limit exceeded, waiting for 30 seconds... Rate limit exceeded, waiting for 30 seconds...

I am attaching my code, can you help me with it? What am I doing wrong or how can I make it work better?

const Rettiwt = require('rettiwt-api'); const { Auth } = require('rettiwt-auth'); const { TweetService } = require('rettiwt-api'); const auth = new Auth(); const fs = require('fs'); const readline = require('readline'); const outStream = fs.createWriteStream('tweets.txt');

// Helper function to write an index to a file function writeIndexToFile(index) { fs.writeFileSync("lastIndex.txt", index.toString()); }

// Helper function to read an index from a file function readIndexFromFile() { if (fs.existsSync("lastIndex.txt")) { return parseInt(fs.readFileSync("lastIndex.txt", "utf8")); } return 0; // default value if file doesn't exist }

async function fetchAndWriteTweets(credential, startIndex) { const tweetService = new TweetService(credential); const fileStream = fs.createReadStream('unique_arabicTrain-clean.csv'); const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity });

let index = 0; for await (const line of rl) { if (index < startIndex) { index++; continue; } let fetched = false; while (!fetched) { try { const tweetId = line.split(',')[1]; const tweet = await tweetService.details(tweetId); outStream.write(JSON.stringify(tweet) + '\n'); fetched = true; writeIndexToFile(index); } catch (error) { if (error.response && error.response.status === 429) { const waitTime = 30; // Wait time is 30 seconds console.log(Rate limit exceeded, waiting for ${waitTime} seconds...); await new Promise(resolve => setTimeout(resolve, waitTime * 1000)); } else { console.error(Failed to fetch Tweet at index ${index}:, error); fetched = true; } } } index++; } }

auth.getGuestCredential() .then(credential => { let startIndex = readIndexFromFile(); fetchAndWriteTweets(credential, startIndex); }) .catch(console.error);

[Rishikant181]

Did you inspect the error to check how much time Twitter is asking you to wait?

[FardinAhsanSakib]

896 seconds.

[FardinAhsanSakib]

if (error.response && error.response.status === 429) { // Get the reset timestamp from headers const resetTimestamp = error.response.headers['x-rate-limit-reset']; // Get the current timestamp const currentTimestamp = Math.floor(new Date().getTime() / 1000); // Calculate the difference in seconds const difference = resetTimestamp - currentTimestamp;

// If difference is negative (i.e., the reset time is in the past), // use a default delay of 30 seconds const waitTime = difference > 0 ? difference : 30;

console.log(Rate limit exceeded, waiting for ${waitTime} seconds...); await new Promise(resolve => setTimeout(resolve, waitTime * 1000)); }

This is how I am checking it

[Rishikant181]

Okay, so the issue is, web APIs generally use a dynamic delay upon detecting rate limit exceeding. By dynamic I mean that, upon successive rate limit breaches, the delay they will ask you to wait will go on increasing. So, introducing a delay of fixed 30 seconds at fixed intervals is not a good solution.

What might work is introducing small, randomized delays in between each successive request. Remember, web APIs check rate limit as Requests/second, the lesser number of requests you make per unit time, the less chance there is of running into Error 429.

As for the value of the randomized delay, it is generally a hit and trial method.

896 seconds.

However, before trying anything, I would advise you to wait for the said time.