Missing tweets when using stream function

[TomRadford]

Thank you so much for this amazing lib!

I seem to be missing occasional tweets when using rettiwt.tweet.stream, ie: I only get every second or third one at seemingly random intervals. Is there something I'm missing here (note I've wrapped the loop in a while to try restart after failures):

const rettiwt = new Rettiwt({
    apiKey: serverEnvs.RETTIWT_API_KEY,
});

const MAX_RETRIES = 5;
const RETRY_DELAY = 5 * 60000; // 5 minutes
const REFRESH_DELAY = 20000;

export const streamTweets = async () => {
    let retries = 0;
    const account = await rettiwt.user.details(ACCOUNT.id);

    while (true) {
        try {
            console.log('Starting tweet stream...');
            for await (const tweet of rettiwt.tweet.stream(
                { fromUsers: [ACCOUNT.userName] },
                REFRESH_DELAY
            )) {
                console.log('NEW TWEET!');
                console.log(tweet.fullText);
            }
        } catch (err) {
            console.error('Error in tweet stream:', err);
            retries++;

            if (retries > MAX_RETRIES) {
                console.error(
                    `Max retries (${MAX_RETRIES}) reached. Waiting for ${RETRY_DELAY / 60000} minutes before restarting.`
                );
                await sleep(RETRY_DELAY);
                retries = 0;
            } else {
                console.log(`Retrying in 5 seconds... (Attempt ${retries} of ${MAX_RETRIES})`);
                await sleep(5000);
            }
        }
    }
};

Any help would be greatly appreciated!

[Rishikant181]

Let me try to reproduce the issue. I'll get back with my observations.

[Rishikant181]

I just tested it against my alt account and it seems to work fine.

If you can send a timeline of the tweets that it fetched (i.e, the actual time the tweet was posted at, can be obtained from the createdAt field) and the actual time the tweet was printed out, that would be very helpful.

[TomRadford]

Thanks so much for your insights!

I think I can do you one better and provide more context (let me know if there's an issue with posting account usernames and will remove):

I'm streaming updates from the MetrorailWC account. For some reason, tweets starting with #SouthernLineCT weren't being detected, however, when adding includeWords: ['#SouthernLineCT'] to rettiwt.tweet.stream's params, I was able to retrieve them. Very strange behaviour and I'm not sure if it might be out of scope of this lib? I tried with two accounts (burner and personal) and the problem persisted.

Something I also noticed is that tweets with no location like this: https://x.com/MetrorailWC/status/1851666155179553085

seem to show up fine.

But when there's a location attached like this:

https://x.com/MetrorailWC/status/1851524753057542159

they dont.

Using rettiwt.tweet.details I'm still able to retrieve the tweet using its id, but not when using rettiwt.tweet.search or rettiwt.tweet.stream Perhaps there's a param missing to also get tweets with location data?

So I'm thinking it might have something to do with tweets with location data being filtered out. Any insights would be much appreciated :)

Edit: Sadly looks like here's another tweet the filter missed (that doesn't have location attached): https://x.com/metrorailwc/status/1851835748095336726?s=46

[Rishikant181]

That's a good pointer. I'm checking it.

[Rishikant181]

@TomRadford Got the issue. It's the same one which was causing issues in quoted tweets a few weeks back. And the tweets with limited visibility are the culprit.

TL:DR, tweets with similar attributes are error out and skipped:

[TomRadford]

@Rishikant181 Thank you so much for investigating! Please let me know if I can assist in any way with testing anything further or reviewing a PR 🙌

[Rishikant181]

@TomRadford Please update to v4.1.4