search

DIGITALCRIMINAL / ArchivedUltimaScraper

Scrape content from OnlyFans and Fansly
GNU General Public License v3.0
953 stars 39 forks source link

OF: No longer able to scrape usernames? #1088

Open lordoffools opened 1 year ago

lordoffools commented 1 year ago

Seems to be timing out. Flipping over to web shows that the account is temporarily blocked/throttled.

Started today-ish. Anyone else seeing this?

Trying to tweak to see if I can diagnose, but I'm coming up empty.

lordoffools commented 1 year ago

Ping

Anyone else experiencing this? I have it set to 1 thread, and have ~700 subscriptions.

jp1151419 commented 1 year ago

Same. It seems like they implemented another rate limiting "feature"

lordoffools commented 1 year ago

They're killing me with these annoying rate limits.

I just suspended all my subscriptions as a result. Spend too much money with them to not be able to have content for offline viewing.

lordoffools commented 1 year ago

Yeah... everything triggers the rate limiting lockout now. I'm running just a single thread.

Scraping profiles triggers it (though sometimes it passes). Once you pick a creator profile to scrape, it hits the limit and locks out just during the initial post scrape.

:(

lordoffools commented 1 year ago

Anyone have a workaround to slow down the scraping to not trigger this?

Occasionally, it does scrape the usernames. Pick one, and it stalls when scraping Posts. It never goes past that (and thus creator doesn't have a lot of posts to begin with).

jp1151419 commented 1 year ago

It seems to run ok if given only a single thread and only parsing a single user at a time. I enter a couple (3-4) at a time spaced by line breaks and run short batches like that with longer pauses between them. Definitely not feasible at scale, just enough to run out a couple subscriptions.

ex:

4 5 6

lordoffools commented 1 year ago

Single thread seems to bring up a list of users half the time for me, half the time it triggers the lockout just while scraping the list of users (~600 total).

I can't get past that, however. It locks out while scraping main posts of a single user after that, even users with ~200 posts or so.

naodrej commented 1 year ago

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

lordoffools commented 1 year ago

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

Good advice. Will give that a shot and report back. Thank you!

naodrej commented 1 year ago

The only way I've gotten it to work so far is wait 2-3 minutes between each action and doing individual models one at a time. So I run it, it shows the list of models (idk about 600, i follow about 200), I wait 2-3 minutes, select the model, scrape/download. When that finishes, wait 2-3 minutes, and repeat.

Good advice. Will give that a shot and report back. Thank you!

You may have to extend the wait times to 5-6 mins depending how much content the model has.

jp1151419 commented 1 year ago

I was getting a few accounts to scrape by going very slowly as we discussed but as of today it seems the paying accounts don't appear in the list anymore, no matter how long the subscription still has before it expires.