Fetching tweet's details suddenly failing randomly with 404 (`NOT_FOUND`)

[Jeto143]

I have a bot running that suddenly started throwing NOT_FOUND errors when fetching tweets a few hours ago. After investigating for a bit, it seems completely random, although it appears to fail way more often than it succeeds.

It might be due to the Twitter API being (somewhat) down (can't find anything online about it), but just in case, here's a way to test it by attempting to fetch the same tweet indefinitely (waiting 5 seconds between attempts).

#!/usr/bin/env node

import { Rettiwt } from 'rettiwt-api';

const r = new Rettiwt({ logging: true });

while (true) {
  await r.tweet.details('1802350417478664517');
  await new Promise(resolve => setTimeout(resolve, 5000));
}

And here's the error:

RettiwtError: NOT_FOUND
    at HttpError.RettiwtError (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/models/errors/RettiwtError.ts:8:3)
    at HttpError (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/models/errors/HttpError.ts:17:3)
    at ErrorService.exports.ErrorService.ErrorService.createHttpError (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/services/internal/ErrorService.ts:41:10)
    at ErrorService.exports.ErrorService.ErrorService.handleHttpError (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/services/internal/ErrorService.ts:127:14)
    at ErrorService.exports.ErrorService.ErrorService.handle (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/services/internal/ErrorService.ts:157:8)
    at TweetService.<anonymous> (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/src/services/public/FetcherService.ts:205:22)
    at step (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/dist/services/public/FetcherService.js:44:23)
    at Object.throw (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/dist/services/public/FetcherService.js:25:53)
    at rejected (/home/jeto/dev/projects/seedplantr/vegan-twitter-bot/node_modules/rettiwt-api/dist/services/public/FetcherService.js:17:65)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 404
}

Note: this endpoint doesn't require authentication but it fails the same way regardless.

[Jeto143]

Looks like it's fixed now (or at least much better). I'll confirm in a few hours just to be sure!

[Jeto143]

Yep, looks like it was just a temporary outage on Twitter's side, sorry!

[Rishikant181]

Still I'll be taking a look at it in the morning (I do absolutely nothing on Saturday and Sunday :P)

[Rishikant181]

It's back again. I think that endpoint used to fetch details of a single tweet is borked. I'm checking what can be done about it.

[Rishikant181]

What I found is that the endpoint works as intended from Postman, but when the exact same request is done via NodeJS, it throws not found. Trying to find a workaround.

Edit 1: cURL works, but Axios fails.

Edit 2: Tried a library called node-libcurl to make HTTP request from inside NodeJS using cURL and it works, but axios fails.

[Jeto143]

It's responding fine for me right now (with rettiwt-api). 🤔

edit: Keeps happening randomly, though much less often than yesterday. It really seems like it's on Twitter's side.

[Rishikant181]

Please update to Rettiwt-API v3.0.2 for the fix and let me know if it's fixed.

[Rishikant181]

Let me know if any other endpoint throws a similar error, cause if it does, we might have to change from axios to cURL for making the requests.

[Jeto143]

@Rishikant181 I'm getting NOT_FOUND immediately using 3.0.2, whereas it still seems fine with 3.0.1 (bot is offline but it seems stable when I test it locally for a little while). 🤔

(Was this change intended?)

(Edit: ah just saw this so that answers my q I guess)

[Rishikant181]

The change was intended. What I did was instead of using the TweetDetailByRestId endpoint (which is an internal endpoint btw and in no way can be accessed by the Twitter Web App), I used TweetDetails endpoint, which is basically the same endpoint used by the Web App to fetch details of a Tweet, along with some of it replies.

Regarding the NOT_FOUND error, I'll have to check it once again then. It seemed to work fine by switching to replies endpoint from rettiwt-core.

[Jeto143]

@Rishikant181 Actually it seems to work when I'm logged in (the old endpoint didn't require it). I'll try for a longer time and keep you updated.

[Rishikant181]

The issue seems to be inherent in NodeJS. I tried running the same code in Bun and it works. Even the same HTTP request config works on other languages like Python, C#, Java, etc. On NodeJS, the workaround is to use node-libcurl, since it doesn't use NodeJS' native fetch library, instead, it uses libcurl library from C.

[Jeto143]

I can confirm I'm not getting any error anymore using 3.0.2 (at least not in like 10 hours).

[Rishikant181]

Still, I'll be keeping this issue open until I've worked out a permanent solution instead of a workaround.