Rishikant181 / Rettiwt-API

A CLI tool and an API for fetching data from Twitter for free!
https://rishikant181.github.io/Rettiwt-API/
MIT License
303 stars 31 forks source link

Can't figure out the proper usage of Cursor Parameter #518

Open Eniamza opened 2 months ago

Eniamza commented 2 months ago

I've been recently experimenting with this API, [This is a very cool one, A lot of Thanks first of all]. But I'm trying to paginate through all the users followed by the given user.

I'm aware that the first request doesn't take account of the "count" parameter and returns a cursor which can be accessed at "response.next.value"

The code snipper I'm using:

const fetchAllFollowers = async (userId, nextCursor = -1) => {

    userId = "1484596586021797889"

    let response = await rettiwt.user.following(userId, 20, nextCursor);
    let followers = response.list;

    console.log(response.next.value)
    console.log(response.list[1].userName)

    total+=followers.length;

    console.log("Current:",followers.length)
    console.log("Total:",total)

    if (response.next.value !== '0|0') {
        followers = followers.concat((await fetchAllFollowers(userId, 20, response.next.value)));
    }

    return followers;
}

This seems to work partially. The code uses the Cursor to fetch the List successfully. but it returns the same list again and again with different cursors.

Output of those console logs:

1792278726323377633|1780610968393351096
MoonfrenNFT
Current: 68
Total: 68
1792278726323377633|1780610973321658296
MoonfrenNFT
Current: 68
Total: 136
1792278726323377633|1780610978757476280
MoonfrenNFT
Current: 68
Total: 204

The list length is always consistent and the first returned user is always the same. Would be really grateful if you help me with this

Eniamza commented 2 months ago

After a few modification this code snippet seems to be working


const fetchAllFollowers = async (userId, count = 20, nextCursor = null) => {

    userId = "1484596586021797889"

    let response = await rettiwt.user.following(userId,count, nextCursor);
    let followers = response.list;

    console.log(response.next.value)
    console.log(response.list[1].userName)

    total+=followers.length;

    console.log("Current:",followers.length)
    console.log("Total:",total)

    if (response.next.value !== '0|0') {
        followers = followers.concat((await fetchAllFollowers(userId, count,  response.next.value)));
    }

    return followers;
}
  1. the initial cursor must be "null"
  2. the "Count" argument doesn't get used in Any subsequent requests. It always returned 48-49 Users per request
Rishikant181 commented 2 months ago

Yeah I was checking it out just now and it was probably because of the default cursor being "-1" instead of undefined.

As for the count, I'm checking on it now.

Eniamza commented 2 months ago

Yeah I was checking it out just now and it was probably because of the default cursor being "-1" instead of undefined. As for the count, I'm checking on it now.

Awesome!

Eniamza commented 2 months ago

Also noticed something.

The user I'm testing on has a following count of "608" on Twitter. After the iterated fetching. I'm getting 603 Followings.

image
Rishikant181 commented 2 months ago

My first guess is twitter removed custom count values altogether, although I need to confirm this.

Eniamza commented 2 months ago

This Count is accurate for ElonMusk's account which os 573.

image

And in another test, again, The total following of HtrZac on Twitter is: 2344. The final following Array variable returns a length of 2333

image
Rishikant181 commented 2 months ago

Okay just tested it and it seems like it's either bugged on Twitter's end or something is changing on Twitter, cause even using the web-app, I can't see all followers and the list just stops loading after some time. And at other times, the web-app is falling back to using the older v1.1 API endpoints instead of the new v2 endpoints for some reason.

Guess we'll just have to wait for some time for things to settle down. In the meantime, I'll see if this is intended or if something is really bugged on Twitter's side.

Rishikant181 commented 2 months ago

Just checked in with the android app and the followers and followings list is bugged there too. Seems like a bug on Twitter's end.

Eniamza commented 2 months ago

Okay just tested it and it seems like it's either bugged on Twitter's end or something is changing on Twitter, cause even using the web-app, I can't see all followers and the list just stops loading after some time. And at other times, the web-app is falling back to using the older v1.1 API endpoints instead of the new v2 endpoints for some reason. Guess we'll just have to wait for some time for things to settle down. In the meantime, I'll see if this is intended or if something is really bugged on Twitter's side.

Awesome. and Yes, I guess that's really a bug on Twitter's end. Looking forward to it. And Many Thanks for the support