[Bug]: requests unable to handle cloudflare check

[thecoolreaver]

Version

falocalrepo 4.4.5

What happened?

Cloudflare has enabled new checkers which prevent

How to reproduce the bug?

attempt to run program and returns cloudflare check error

Relevant log output

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/falocalrepo/__main__.py", line 59, in main
    exit(app.main(standalone_mode=False) or 0)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/falocalrepo/console/download.py", line 165, in download_users
    api: FAAPI = open_api(db, ctx)
  File "/usr/local/lib/python3.10/dist-packages/falocalrepo/console/util.py", line 199, in open_api
    if check_login and not api.login_status:
  File "/usr/local/lib/python3.10/dist-packages/faapi/base.py", line 118, in login_status
    return parse_loggedin_user(self.get_parsed("login", skip_auth_check=True)) is not None
  File "/usr/local/lib/python3.10/dist-packages/faapi/base.py", line 146, in get_parsed
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://www.furaffinity.net/login

[thecoolreaver]

doing a test via python cloudscraper

import cloudscraper 

scraper = cloudscraper.create_scraper(delay=10, browser="chrome") 
content = scraper.get("https://www.furaffinity.net/").text 

print(content)

returns this cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.

[thecoolreaver]

Rerunning the test now shows its fine. Back to normal? Ill leave this open for the moment but you may close if needed

[MatteoCampinoti94]

I am aware of the issue and have already tried using cfscrape with no success.

As soon as I find a solution I’ll update the library/add a comment here :)

[MatteoCampinoti94]

Rerunning the test now shows it’s fine. Back to normal? Ill leave this open for the moment but you may close if needed

@thecoolreaver Did you perchance use the same cookies in a login session you use on your browser? This same issue happened before and the “fix” last time was to simply open the page in a browser so cloudflare would clear those particular cookies.

[thecoolreaver]

No I did not, I have a separate account and this is ran on a different ISP.

The tests were just python's requests and cloudscraper going direct to the main site to test cloudflare's Captcha response which did not include any login cookies or sessions at that time.

[MatteoCampinoti94]

Fixed it!

As I suspected, using the cookies in a browser solves the issue and clears the cookies.

Alternatively, new cookies can be created with a new login session.

Using cloudscraper or cfscrape is not necessary (not useful as we saw). Once the cookies are cleared they can be used with a normal requests.Session object, and as long as those cookies are added to the session's cookie jar, then all requests will succeed :)

[thecoolreaver]

Ok

I will drop this code I did find for future ref in case the option of having to switch is needed. This will probably not be the best as it requires a chromium browser to be installed so it can be ran headless but this does get around the v2 Cloudflare Captcha

undetected-chromedriver

from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import undetected_chromedriver as uc

driver = uc.Chrome(headless=True,use_subprocess=False)

driver.get('https://www.furaffinity.net/')
print('current url %s' % driver.current_url)

try:
    WebDriverWait(driver,15).until(EC.title_contains('moment'))
except TimeoutException:
    pass

print('current url %s' % driver.current_url)

try:
    WebDriverWait(driver,15).until(EC.title_contains('Affinity'))
    print('PASSED CLOUDFLARE!')

except TimeoutException:
    print('timeout')
    print('current url %s' % driver.current_url)

print(driver.page_source)
driver.quit

[MatteoCampinoti94]

Thanks for sharing!

I am actually testing Selenium for falocalrepo to simplify the login process for users who aren’t so versed with cli tools and browsers’ inspection tools.