Rishikant181 / Rettiwt-API

A CLI tool and an API for fetching data from Twitter for free!
https://rishikant181.github.io/Rettiwt-API/
MIT License
319 stars 32 forks source link

Seems page 2+ of YakuCorp always has the same cursor value #307

Closed zoinkydoink closed 1 year ago

zoinkydoink commented 1 year ago

Run the following and page 2, 3 (and if u do more) they have the exact same value returning, which makes it impossible to page through that accounts tweets

async function TestHandle() {
    try {
        const rettiwt = Rettiwt();
        const user = await rettiwt.users.getUserDetails("YakuCorp");
        const page1 = await rettiwt.users.getUserTweets(user.id);

        console.log("page 1 cursor " + page1.next.value);
        const page2 = await rettiwt.users.getUserTweets(user.id, 100, page1.next.value);
        console.log("page 2 cursor " + page2.next.value);
        const page3 = await rettiwt.users.getUserTweets(user.id, 100, page2.next.value);
        console.log("page 3 cursor " + page3.next.value);

    } catch (e) {
        console.error('Error:', e);
    }

}

I assume as long as the page.next.value is not null then there is a page after, but it seems in this case there is unlimited number of pages but they all same value as page2

Rishikant181 commented 1 year ago

Just confirmed. This is actually a problem with getUserTweets method. I will be working on it but because of my ongoing exams, I'll not be able to for 3 more days.

In the meantime, you can use this method to fetch tweets, but this requires you to use cookies.

The problem is, getUserTweets method was implemented as a fail-safe after Twitter required you to login to use advanced search (the best way to fetch tweets is using advanced search). getUserTweets fetches the tweets from the target user's timeline. It is not as polished as fetching from advanced search.

The alternate method that I gave above is the one that uses advanced search, so it will work very well, although it requires using cookie.

I apologize for the inconvenience. I'll work on as soon as I possibly can.

zoinkydoink commented 1 year ago

Thank you for the reply, I am trying to avoid cookies at all costs as I do not want an twitter account tied to the queries. I am more interested in technical details on what is required and which calls are required to just get user tweets without any sort of cookies, I like to replicate this functionality in .NET instead of nodejs/typescript calls.

Are you able to provide me with any information on this. I looked at the traffic sent for the call using fiddler/http intercept, i see one call going out to guest end point but couldnt figure out all the headers etc.

Rishikant181 commented 1 year ago

I would be happy if someone implements it on other environments as well.

What I did was use Twitter from my browser and monitor the HTTP request in the browser's DevTools. It's somewhat of fun but tedious process.

I have documented all the Twitter API endpoints which I could find and have those in a Postman workspace. The problem is, I'm running Postman free tier so I can't have more than 3 people in my workspace. I'll share with you the endpoints as soon as I move it to a free alternative.

Rishikant181 commented 1 year ago

I'm back!

Now for the important thing at hand, I check that this 'anomaly' occurs only with the account 'YakuCorp'. All other (afaik) accounts work as intended. Now that I have replicated, I have started debugging.

Expect results soon!

zoinkydoink commented 1 year ago

Thank you and if you can figure out how to share the postman calls for the guest calls, I would really appreciate it I tried looking through inspect but that was a mess as there were many different things being passed. So if you can just provide that.

I think you can manually export collections to json in postman

right click a collection select export save file as json send json file

Rishikant181 commented 1 year ago

API.zip

Here you go. Not all endpoints have been well documented by me, as it's still a work in progress.

Edit: GitHub doesn't support sending .json files