Error initializing Tweet, reading 'created_at'

[smaldd14]

Hey, thanks for the new updates and development time put into this project.

I'm running into an exception that looks like this:

⚠  functions: TypeError: Cannot read properties of undefined (reading 'created_at')
at new Tweet (/path/to/node_modules/rettiwt-api/dist/models/data/Tweet.js:24:39)

It looks like it is failing on the line this.createdAt = tweet.legacy.created_at;

I am using this call to search:

const sinceDate = new Date();
 sinceDate.setDate(oneDayAgo.getDate() - 1);
const tweets = await rettiwt.tweet.search({
      fromUsers: ["elonmusk"],
      startDate: sinceDate,
  });

If it helps, I am doing some slight manipulation to the tweet's fullText and printing it out:

  tweet: {"id":"1752052871124623515","createdAt":"Mon Jan 29 19:35:30 +0000 2024","tweetBy":{"id":"44196397","userName":"elonmusk","fullName":"Elon Musk","createdAt":"Tue Jun 02 20:12:29 +0000 2009","description":"(CTO) Chief Troll Officer","isVerified":false,"favouritesCount":40880,"followersCount":170556912,"followingsCount":523,"statusesCount":36852,"location":"Trōllheim","profileBanner":"https://pbs.twimg.com/profile_banners/44196397/1690621312","profileImage":"https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg"},"entities":{"hashtags":[],"urls":[],"mentionedUsers":["RepThomasMassie"]},"fullText":"@RepThomasMassie: Is there anyone you don’t want to bomb?.\nelonmusk: @RepThomasMassie 🎯.","replyTo":"1752045649875304689","lang":"qme","quoteCount":5,"replyCount":73,"retweetCount":55,"likeCount":648,"viewCount":7673,"bookmarkCount":9}

[smaldd14]

This is no longer happening for me... I was seeing strange behavior with TOO_MANY_REQUESTS errors around the same time too.

Not sure what happened, but this seems to be fixed now

[Rishikant181]

Still, I'm gonna take a look at it, just in case.

[smaldd14]

@Rishikant181 Thanks. I'm seeing the error occuring again. This time searching for username "arvidkahl" and since the last day.

I think it could be happening with the call to get details of a replyTo or quoted tweet. The error seems to be occuring around this line: await rettiwt.tweet.details(tweet.replyTo);

If it helps, here's the entire code snippet:

export async function getTweetsByUser(username: string, sinceDate: Date): Promise<Tweet[]> {
    const tweets = await rettiwt.tweet.search({
        fromUsers: [username],
        startDate: sinceDate,
    });

    let userTweets: Tweet[] = tweets.list //.filter((tweet) => !tweet.replyTo);
    const tweetPromises = userTweets.map(async (tweet) => {
      try {
        if (!!tweet.quoted || tweet.fullText.startsWith('RT') || tweet.fullText.startsWith('QT')) {
          const quotedTweet = await rettiwt.tweet.details(tweet.quoted);
          tweet.fullText = '@' + quotedTweet.tweetBy.userName + ': ' + quotedTweet.fullText + '\n@' + tweet.tweetBy.userName +': ' + tweet.fullText;
        } else if (!!tweet.replyTo) {
          const replyToTweet = await rettiwt.tweet.details(tweet.replyTo);
          tweet.fullText = '@' + replyToTweet.tweetBy.userName + ': ' + replyToTweet.fullText + '\n@' + tweet.tweetBy.userName +': ' + tweet.fullText;
        } else {
          tweet.fullText = '@' + tweet.tweetBy.userName + ': ' + tweet.fullText;
        }
        return tweet;
    });
    userTweets = await Promise.all(tweetPromises);

    return userTweets;
}

I've dug deeper and found the tweet that seems to be causing the error and I printed it out. This is what it seems to be failing on:

error parsing tweet data: TypeError: Cannot read properties of undefined (reading 'created_at')
tweet: {"id":"1752089833097334989","createdAt":"Mon Jan 29 22:02:23 +0000 2024","tweetBy":{"id":"343990983","userName":"arvidkahl","fullName":"Arvid Kahl","createdAt":"Thu Jul 28 11:30:56 +0000 2011","description":"Building https://t.co/B1R2YOxw9F in Public. Raising all the boats with kindness.\n\n🎙️ https://t.co/6w69DZmi8H · ✍️ https://t.co/lpnor5rsTW · 🗞️ https://t.co/dY99qs7qHQ · 📚 https://t.co/cHkXgWNeCT","isVerified":false,"favouritesCount":141943,"followersCount":133521,"followingsCount":17414,"statusesCount":74379,"location":"Ontario, Canada","pinnedTweet":"1743639220206391414","profileBanner":"https://pbs.twimg.com/profile_banners/343990983/1683397562","profileImage":"https://pbs.twimg.com/profile_images/1201525049766883328/QPimCC9z_normal.jpg"},"entities":{"hashtags":[],"urls":[],"mentionedUsers":[]},"fullText":"Oh, and: please retweet this. If you can get just a handful of founders to think about calling, you’re already doing massive work.","replyTo":"1752084354455539768","lang":"en","quoteCount":0,"replyCount":1,"retweetCount":0,"likeCount":0,"viewCount":1388,"bookmarkCount":0}

[Rishikant181]

Checking

[Rishikant181]

Can you post the output with logging on? That way I can see exactly at when step and on which data the error is thrown.

[smaldd14]

@Rishikant181 Ah, I'm having trouble reproducing it again, so I do not have the full stack trace.

I can tell you that the exception was thrown at node_modules/rettiwt-api/dist/models/data/Tweet.js line 24 it's during the Tweet ctor where this.createdAt = tweet.legacy.created_at;

Also, remembering the stack trace, I believe that function was called from node_modules/rettiwt-api/dist/services/internal/FetcherService.js line 255 during FetcherService.prototype.deserializeData

// If the item is a valid raw tweet
            if (item && item.__typename == 'Tweet' && item.rest_id) {
                // Logging
                this.logger.log(Logging_1.ELogActions.DESERIALIZE, { type: item.__typename, id: item.rest_id });
                // Adding deserialized Tweet to list
                deserializedList.push(new Tweet_1.Tweet(item));
            }

during this condition

[Rishikant181]

Does this issue still occur?

[smaldd14]

It is finicky. Sometimes it occurs, sometimes it does not. Not sure if you want to leave the issue open in case anyone else runs into it, but as a "workaround" (not really, but a way to keep moving) I just wrapped the code giving me a problem in a try/catch and log the error. This might not be a sufficient work around for some, but I don't necessarily care about each and every tweet given back, so it worked for me.

e.g.

let userTweets: Tweet[] = tweets.list;
const tweetPromises = userTweets.map(async (tweet) => {
      try {
        if (!!tweet.quoted || tweet.fullText.startsWith('RT') || tweet.fullText.startsWith('QT')) {
          const quotedTweet = await rettiwt.tweet.details(tweet.quoted);
          tweet.fullText = '@' + quotedTweet.tweetBy.userName + ': ' + quotedTweet.fullText + '\n@' + tweet.tweetBy.userName +': ' + tweet.fullText;
        } else if (!!tweet.replyTo) {
          const replyToTweet = await rettiwt.tweet.details(tweet.replyTo);
          tweet.fullText = '@' + replyToTweet.tweetBy.userName + ': ' + replyToTweet.fullText + '\n@' + tweet.tweetBy.userName +': ' + tweet.fullText;
        } else {
          tweet.fullText = '@' + tweet.tweetBy.userName + ': ' + tweet.fullText;
        }
        return tweet;
     } catch (error) {
        console.log(error);
        return null;
     }
    });
// filter out nulls
userTweets = (await Promise.all(tweetPromises)).filter((tweet): tweet is Tweet => tweet !== null);