Exporting Blocklists Stops After 20k

[SysAdminDoc]

Greetings,

I have a lot of blocked accounts that I'm trying to backup/export the lists. I left the export process working overnight from the blocklist and am finding that it stops loading after 20k accounts. Any idea how I can make a complete backup?

[SysAdminDoc]

I think Twitter has put some sort of anti-script mechanism in place after it queries about 20k accounts from the blocklist. Each repeat attempt when I try to get a full list it will get around the 20k range before the "Something is wrong. Try reloading" screen and the extension stops. I'm curious if the extension can be updated so that it can pause after it queries 10k accounts for say 30 minutes (assuming that's how long the Twitter API is blocking the call for accounts) and then resume the querying?

[dmstern]

Hey @SysAdminDoc,

from the Twitter API Docs:

The blocks lookup GET endpoint allows you to see which accounts you’ve blocked on behalf of an authorized user. This endpoint has a rate limit of 15 requests per 15 minutes per user.

This may cause the problem. Although we don't call this endpoint directly but through the twitter front-end, there might this limitation you mentioned. Yes, maybe we could add some sort of timeout to bypass this limit when collecting the blocked accounts.

Currently I don't have the time, but if you (or anyone else) want to try it, feel free to file a PR. :)

Maybe have a look at src/LikersBlocker.ts:263:

this.collectedUsers.push(userHandle);

There's also a function called stopScrolling(). Maybe we could add a timer that triggers a setTimeout or check if a specific amount of accounts already has been collected before we proceed to startScrolling().

[SysAdminDoc]

Ah man. I'm in above my head.

[KatayR]

It stops waaay earlier for me.

Everything was fine few months ago but now I cant export all my block list. It stops at around 3k. If I zoom out the page it collects accounts faster but it still stops around 8k.