Infinity Cursor Value

[gratatouilley]

Hi again😁

I planned to scrap all tweets for specified user and did try to user 1794271235284815873 which just having 166 tweets. After all tweets has been fetched, the cursor still returning value and list with pinned tweet.

Sample code:

const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const fetchAllTweets = async (id: string) => {
  let hasNextPage = true;
  let cursor: string | undefined = undefined;
  const filePath = `${id}tweets.json`;

  const user = await rettiwt.user.details(id);

  // Initialize the file by writing an empty array
  fs.writeFileSync(filePath, JSON.stringify([]));

  while (hasNextPage) {
    try {
      const response = await rettiwt.user.replies(id, 20, cursor);

      if (response && response.list) {
        const tweets = response;

        // Read current file content
        const currentData = JSON.parse(fs.readFileSync(filePath, 'utf8'));

        // Append new tweets to the current data
        currentData.push(tweets);

        // Write updated data to the file
        fs.writeFileSync(filePath, JSON.stringify(currentData, null, 2));

        cursor = response.next.value;
        hasNextPage = !!cursor;
        console.log(response.next, hasNextPage);

        await delay(3000);
      } else {
        console.error(`Invalid response structure for ${id}:`, response);
        hasNextPage = false;
      }
    } catch (err) {
      console.error(`Failed to fetch tweets for ${id}:`, err);
      hasNextPage = false;
    }
  }

};

Sample debug logs response after all tweets fetched and infinity returns pinned tweet:

RESPONSE JSON FILE

[Rishikant181]

if (response && response.list)

Here, can you try with the condition as response && response.list.length?

With only response.list, the net logical evaluates to true, even with response.list == []. However, with response.list.length, the net logical evaluates to false, when an empty list is returned, thereby stopping the fetch process.

[gratatouilley]

if (response && response.list)

Here, can you try with the condition as response && response.list.length?

With only response.list, the net logical evaluates to true, even with response.list == []. However, with response.list.length, the net logical evaluates to false, when an empty list is returned, thereby stopping the fetch process.

I think this will still pass cause as you can see the last data is still returining list which contains pinned tweet. and I just realized that the pinned tweet returns on every request

    {
      "list": [
        {
          "id": "1794272810065592439",
          "createdAt": "Sat May 25 07:42:28 +0000 2024",
          "tweetBy": {
            "id": "1794271235284815873",
            "userName": "PEBIWP",
            "fullName": "PEBIWELIANPUTRA",
            "createdAt": "Sat May 25 07:36:25 +0000 2024",
            "isVerified": false,
            "likeCount": 188,
            "followersCount": 278,
            "followingsCount": 137,
            "statusesCount": 166,
            "pinnedTweet": "1794272810065592439",
            "profileBanner": "https://pbs.twimg.com/profile_banners/1794271235284815873/1716624438",
            "profileImage": "https://pbs.twimg.com/profile_images/1794271474011996160/1Bksw3nL_normal.jpg"
          },
          "entities": {
            "hashtags": [],
            "mentionedUsers": [
              "elonmusk"
            ],
            "urls": []
          },
          "media": [
            {
              "url": "https://pbs.twimg.com/media/GOaJmxDbYAAC7uP.jpg",
              "type": "photo"
            }
          ],
          "fullText": "akun debu gue salah apaaa😭? sekedar reply dan rt doang mas @elonmusk https://t.co/oIFLGNd3gZ",
          "lang": "in",
          "quoteCount": 0,
          "replyCount": 4,
          "retweetCount": 0,
          "likeCount": 1,
          "viewCount": 429,
          "bookmarkCount": 0
        }
      ],
      "next": {
        "value": "DAABCgABGOvjMmb__zUKAAIY5o8GzhqgkQgAAwAAAAIAAA"
      }
    }

[Rishikant181]

Okay then the issue seems to be the pinned tweet. A workaround would be to check if the returned result has only the pinned tweet, which can be the exit condition.

A permanent fix will be to just remove the pinned tweet from the result on the library's side. Working on it now.

[Rishikant181]

For now, the only solution seems to be checking the following as exit conditions:

1. The returned list is empty (in case the user does not have a pinned tweet)
2. The returned list only has the pinned tweet (in case the user has a pinned tweet).

As for whether the user has a pinned tweet or not, it can be ascertained from the user details.