scraper.getTweet returns 404

VanillaDevelop commented 1 year ago

On v0.5.0, using code that worked 2 days ago, I now only get 404 errors when using scraper.getTweet. For example, trying to retrieve https://twitter.com/elonmusk/status/1689963696703848449. Tested on a VPN to check if I was blocked or something but same result. Headers also seem to indicate that it's not a rate limit issue ('x-rate-limit-remaining': [ '499' ])

Can you replicate this or is it a local issue? I'd imagine it's just part of the ongoing Elon Musk self-destruct situation, but maybe you have an idea of how to fix it or what causes it.

Using code

import { Scraper } from "@the-convocation/twitter-scraper";
const scraper = new Scraper()
console.log(scraper.getTweet("1689963696703848449"))

Error (I removed some parts from the passthrough and the URL, but most of the information is there)

ApiError: Response status: 404
    at requestApi (A:\Code\test\node_modules\@the-convocation\twitter-scraper\dist\api.js:57:18)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async getTweet (A:\Code\test\node_modules\@the-convocation\twitter-scraper\dist\tweets.js:105:17) {
  response: Response {
    [...]
    [Symbol(Response internals)]: {
      url: 'https://api.twitter.com/graphql/[...]',
      status: 404,
      statusText: 'Not Found',
      headers: Headers {
        [Symbol(map)]: [Object: null prototype] {
          date: [ 'Wed, 16 Aug 2023 17:22:25 UTC' ],
          perf: [ '7626143928' ],
          server: [ 'tsa_o' ],
          'cache-control': [ 'no-cache, no-store, max-age=0' ],
          'content-length': [ '0' ],
          'x-transaction-id': [ 'b0234b43ce9dcd0f' ],
          'x-rate-limit-limit': [ '500' ],
          'x-rate-limit-reset': [ '1692207445' ],
          'x-rate-limit-remaining': [ '499' ],
          'strict-transport-security': [ 'max-age=631138519' ],
          'x-response-time': [ '109' ],
          'x-connection-hash': [
            'c2e3a1e959df9fac9591a6faaf7e126385a8ca2a0f3c36589d55e74e60b0ec96'
          ],
          connection: [ 'close' ]
        }
      },
      counter: 0
    }
  }
}

sviatoslav-lebediev commented 1 year ago

I have the same problem with getTweets. I've tried to fix and changed this url https://github.com/the-convocation/twitter-scraper/blob/1a583990676724b0ede4bd055f667582271ff90e/src/profile.ts#L148 to https://twitter.com/i/api/graphql/SAMkL5y_N9pmahSw8yy6gw/UserByScreenName I'm not sure if it's a correct URL but at least it doesn't fail. But now this line fails https://github.com/the-convocation/twitter-scraper/blob/1a583990676724b0ede4bd055f667582271ff90e/src/tweets.ts#L122 and I have no idea how to find a correct url

dandyraka commented 1 year ago

getProfile too

ImTheDeveloper commented 1 year ago

From other projects it appears using the guest token to fetch from these APIs no longer works and you'd have to switch to oauth1.0. Nitter for example is generating temporary anonymous guest users and hitting the above apis with oauth key/secrets.

karashiiro commented 1 year ago

There are still some tweet-related functions that are broken, but most of the breakages have been fixed in v0.6.0. Looks like it was entirely just API endpoints that were changed/added/removed all at once. I still need to figure out what's up with replies, though (/TweetDetail when logged-in and viewing a tweet), contributions welcome 👀

karashiiro commented 1 year ago

More fixes in #62, the only thing that still isn't working is tweet threads.