search

adultmm / AdultMediaManager

Adult Media Manager is the ultimate media manager for your adult movies and videos. Organize your content for Kodi, Plex, and other media centers.
192 stars 30 forks source link

IAFD 403 forbidden error #121

Closed DrunkSith closed 3 months ago

DrunkSith commented 4 months ago

I can't seem to scrape anything from IAFD.

wkearney99 commented 4 months ago

Same here. Get error 403 within AMM. Using same URL in a browser successfully loads the page.

And if I take the title.rme/id= field and use that as a manual ID it also throws the 403 error.

Is the User Agent string editable in AMM? Might that be something iafd is blocking?

wkearney99 commented 4 months ago

Is this related? https://github.com/adultmm/AdultMediaManager/issues/108#issuecomment-2042704756

AMM utilizes DuckDuckGo for searches on the IAFD if no results are found during the onsite search. Occasionally, DuckDuckGo blocks too many searches, so AMM attempts accessing DuckDuckGo via the Tor network. However, in your case, this was unsuccessful five times. I'll take a look at the DDG search.

adultmm commented 4 months ago

Guys, please attach the corresponding log file. Thanks

wkearney99 commented 4 months ago

It's repeatable. No scrapes are successful from iafd. This is a simple test. It has one movie in the database, and ONLY the iafd scraper is enabled.

2024-07-15.csv

adultmm commented 4 months ago

Thank you. I can see the 403 error in the log, but it works for me. I'll try to reproduce.

wkearney99 commented 4 months ago

It would really help if we had a bit more indications in the log about how a connection was being made. Using proxies, tor, etc. It'd help point the finger toward the actual problem.

adultmm commented 4 months ago

The actual problem - as usual - is the ClouFlare protection. Me, and many others work on solving these challenges: FlareSolverr, CF-Clearance-Scraper, etc. If you can download the content of the URL with a curl or any other tool, you have solved the problem. Send the solution, and I'll implement it in AMM.

adultmm commented 4 months ago

Fixed in v0.12.12.

adultmm commented 4 months ago

Try out the standalone version that works: v1.0.0-beta1.

adultmm commented 4 months ago

@wkearney99 @DrunkSith v1.0.0-beta1 scrapes IAFD for you? Thanks

adultmm commented 4 months ago

Cody has an issue with the v1.0.0-beta1 version as well: https://github.com/adultmm/AdultMediaManager/issues/123#issuecomment-2241492370

wkearney99 commented 4 months ago

Nope, 1.0.0.b1 immediate 403 error, same as with 12.11. 2024-07-26.csv

Fetterbr61 commented 4 months ago

aaa-004 cant scrape any japanese movies, especially femdom sites.

wkearney99 commented 3 months ago

The problem seems to come and go. What's the code logic behind the scenes that would affect this?

adultmm commented 3 months ago

As I can see, IAFD changed someting on their CF settings and now AMM can scrape their pages. Please reopen, if the problem appears again.

wkearney99 commented 1 month ago

Seems to be a problem again. 403 errors. Even a paste of the manual ID entry fails with a 403.