search

FlareSolverr / FlareSolverr

Proxy server to bypass Cloudflare protection
MIT License
5.99k stars 533 forks source link

[multiple sites] Error solving the challenge. Timeout after X seconds - challenge loop #1036

Open MrTyton opened 4 months ago

MrTyton commented 4 months ago

Have you checked our README?

Have you followed our Troubleshooting?

Is there already an issue for your problem?

Have you checked the discussions?

Environment

- FlareSolverr version: 3.3.13
- Last working FlareSolverr version: Unsure, but was working on Monday (2024/01/08)
- Operating system: Docker Unraid
- Are you using Docker: yes
- FlareSolverr User-Agent (see log traces or / endpoint): Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
- Are you using a VPN: no
- Are you using a Proxy: no
- Are you using Captcha Solver: No
- If using captcha solver, which one:
- URL to test this issue: https://www.fanfiction.net/s/14145272/1/In-Your-Wildest-Dreams

Description

Using FanFicFare to scrape from fanfiction.net. Nothing's changed with my config, but it stopped working this week.

Logged Error Messages

01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     FlareSolverr 3.3.13
01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     Testing web browser installation...
01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     Platform: Linux-5.19.17-Unraid-x86_64-with-glibc2.31
01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     Chrome / Chromium path: /usr/bin/chromium
01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     Chrome / Chromium major version: 120
01/11/2024
07:35:52 PM
2024-01-12 00:35:52 INFO     Launching web browser...
01/11/2024
07:35:52 PM
version_main cannot be converted to an integer
01/11/2024
07:35:53 PM
2024-01-12 00:35:53 INFO     FlareSolverr User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
01/11/2024
07:35:53 PM
2024-01-12 00:35:53 INFO     Test successful!
01/11/2024
07:35:53 PM
2024-01-12 00:35:53 INFO     Serving on http://0.0.0.0:8191
01/11/2024
07:35:59 PM
2024-01-12 00:35:59 INFO     Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://www.fanfiction.net/s/14145272/1/In-Your-Wildest-Dreams', 'maxTimeout': 65000, 'cookies': [], 'postData': None}
01/11/2024
07:35:59 PM
version_main cannot be converted to an integer
01/11/2024
07:35:59 PM
2024-01-12 00:35:59 INFO     Challenge detected. Title found: Just a moment...
01/11/2024
07:37:04 PM
2024-01-12 00:37:04 ERROR    Error: Error solving the challenge. Timeout after 65.0 seconds.
01/11/2024
07:37:04 PM
2024-01-12 00:37:04 INFO     Response in 65.723 s
01/11/2024
07:37:04 PM
2024-01-12 00:37:04 INFO     172.17.0.1 POST http://192.168.1.161:8191/v1 500 Internal Server Error

Screenshots

No response

MrTyton commented 3 months ago

Do we know what's different about ffnet?

On Tue, Feb 20, 2024, 6:58 PM ilike2burnthing @.***> wrote:

v3.3.15 https://github.com/FlareSolverr/FlareSolverr/releases/v3.3.15

Thanks @tadasgedgaudas https://github.com/tadasgedgaudas

Resolves looping challenge issue with most sites. Known exceptions:

  • fanfiction.net
  • airbusan.com

Further PRs are welcome.

— Reply to this email directly, view it on GitHub https://github.com/FlareSolverr/FlareSolverr/issues/1036#issuecomment-1955416338, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMW5GY2FGQHEBTCINY7ZNLYUU2BHAVCNFSM6AAAAABBXLAM52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGQYTMMZTHA . You are receiving this because you authored the thread.Message ID: @.***>

ilike2burnthing commented 3 months ago

Nope. Annoyingly, both it and airbusan.com were working with the PR up until a few hours ago (reverting the last commit doesn't make a difference).

juanfrilla commented 3 months ago

with this new release i still cannot get the cookies from here: url = "https://www.icj-cij.org/sites/default/files/case-related/187/187-20231215-ord-01-00-en.pdf" I'll wait for new releases

AakashDev-Draup commented 3 months ago

Another exception is indeed url=https://www.indeed.com/jobs?q=microsoft&l=&vjk=f14139b482bebdbf

ilike2burnthing commented 3 months ago

I'm not getting a Cloudflare challenge for indeed.com, in browser or with FlareSolverr, so I can't test it.

Gallardo26 commented 3 months ago

Just updated. Keep getting challenge not detected. image

Not sure why going to the link in the challenge shows forbidden. (https://m.happymh.com/v2.0/apis/manga/read?code=wuliandianfeng&cid=3863544) The URL: https://m.happymh.com/reads/wuliandianfeng/4035214

iliyaZelenko commented 3 months ago

Error solving the challenge. Timeout after 120.0 seconds http://localhost:8191/v1 500 Internal Server Error

image
FFace32 commented 3 months ago

I've recently found this framework: https://github.com/omkarcloud/botasaurus/tree/master

This code works perfectly fine even with a proxy

from botasaurus import *
from botasaurus.create_stealth_driver import create_stealth_driver

def get_start_url(data):
    return data

@browser(
    proxy="REDACTED",
    user_agent=bt.UserAgent.REAL,
    window_size=bt.WindowSize.REAL,
    max_retry=5,
    create_driver=create_stealth_driver(
        start_url=get_start_url,
        raise_exception=True,
    ),
)
def scrape_heading_task(driver: AntiDetectDriver, data):
    driver.prompt()
    heading = driver.text("h1")
    return heading

scrape_heading_task(
    [
        "https://m.happymh.com/reads/wuliandianfeng/4035214",
        "https://www.indeed.com/jobs?q=microsoft&l=&vjk=f14139b482bebdbf",
        "https://www.icj-cij.org/sites/default/files/case-related/187/187-20231215-ord-01-00-en.pdf",
        "https://www.fanfiction.net/s/14316251/2/Xia",
        "https://en.airbusan.com/content/individual/?",
    ]
)

except for airbusan where I get a

Sorry, you have been blocked
You are unable to access airbusan.com

page. (works fine without a proxy, or maybe even with a better one, didn't test)

I've managed to get it working even in headless mode (you just have to set the proper user agent), and even replaced undetected_chromedriver with it in this project, but my implementation is not yet very robust and starts failing after 2-3 requests.

Maybe someone has time to check it out and adapt undetected_chromedriver with methods from it.

juanfrilla commented 3 months ago

I've recently found this framework: https://github.com/omkarcloud/botasaurus/tree/master

This code works perfectly fine even with a proxy

from botasaurus import *
from botasaurus.create_stealth_driver import create_stealth_driver

def get_start_url(data):
    return data

@browser(
    proxy="REDACTED",
    user_agent=bt.UserAgent.REAL,
    window_size=bt.WindowSize.REAL,
    max_retry=5,
    create_driver=create_stealth_driver(
        start_url=get_start_url,
        raise_exception=True,
    ),
)
def scrape_heading_task(driver: AntiDetectDriver, data):
    driver.prompt()
    heading = driver.text("h1")
    return heading

scrape_heading_task(
    [
        "https://m.happymh.com/reads/wuliandianfeng/4035214",
        "https://www.indeed.com/jobs?q=microsoft&l=&vjk=f14139b482bebdbf",
        "https://www.icj-cij.org/sites/default/files/case-related/187/187-20231215-ord-01-00-en.pdf",
        "https://www.fanfiction.net/s/14316251/2/Xia",
        "https://en.airbusan.com/content/individual/?",
    ]
)

except for airbusan where I get a

Sorry, you have been blocked
You are unable to access airbusan.com

page. (works fine without a proxy, or maybe even with a better one, didn't test)

I've managed to get it working even in headless mode (you just have to set the proper user agent), and even replaced undetected_chromedriver with it in this project, but my implementation is not yet very robust and starts failing after 2-3 requests.

Maybe someone has time to check it out and adapt undetected_chromedriver with methods from it.

It does not work with headless=True

FFace32 commented 3 months ago

I've recently found this framework: https://github.com/omkarcloud/botasaurus/tree/master This code works perfectly fine even with a proxy

from botasaurus import *
from botasaurus.create_stealth_driver import create_stealth_driver

def get_start_url(data):
    return data

@browser(
    proxy="REDACTED",
    user_agent=bt.UserAgent.REAL,
    window_size=bt.WindowSize.REAL,
    max_retry=5,
    create_driver=create_stealth_driver(
        start_url=get_start_url,
        raise_exception=True,
    ),
)
def scrape_heading_task(driver: AntiDetectDriver, data):
    driver.prompt()
    heading = driver.text("h1")
    return heading

scrape_heading_task(
    [
        "https://m.happymh.com/reads/wuliandianfeng/4035214",
        "https://www.indeed.com/jobs?q=microsoft&l=&vjk=f14139b482bebdbf",
        "https://www.icj-cij.org/sites/default/files/case-related/187/187-20231215-ord-01-00-en.pdf",
        "https://www.fanfiction.net/s/14316251/2/Xia",
        "https://en.airbusan.com/content/individual/?",
    ]
)

except for airbusan where I get a

Sorry, you have been blocked
You are unable to access airbusan.com

page. (works fine without a proxy, or maybe even with a better one, didn't test) I've managed to get it working even in headless mode (you just have to set the proper user agent), and even replaced undetected_chromedriver with it in this project, but my implementation is not yet very robust and starts failing after 2-3 requests. Maybe someone has time to check it out and adapt undetected_chromedriver with methods from it.

It does not work with headless=True

I've managed to get it working even in headless mode (you just have to set the proper user agent)

AakashDev-Draup commented 3 months ago

I'm not getting a Cloudflare challenge for indeed.com, in browser or with FlareSolverr, so I can't test it.

I am also not getting the challenge locally. But getting in the AWS EC2 instance where we are running the Flaresolverr container. Any recommendations on how to handle this? @ilike2burnthing

ilike2burnthing commented 3 months ago

Try using a proxy.

ParvusLacertus commented 3 months ago

Changing the server from waitress to paste resolved any timeouts I had previously; https://github.com/ParvusLacertus/FlareSolverr/commit/805444bd788dee340131cf770c8441ad39bd79fc

Although I have next to no idea why, and whether it would help in this case.

ilike2burnthing commented 3 months ago

I installed the paste module, used the Python request template from the readme, but with the list of URLs here - https://github.com/FlareSolverr/FlareSolverr/issues/1036#issuecomment-1955416338, and tried both your commit and PR. Not seeing any improvement.

Was there something else you were doing, or different URLs that aren't working with v3.3.15?

ParvusLacertus commented 3 months ago

I installed the paste module, used the Python request template from the readme, but with the list of URLs here - #1036 (comment), and tried both your commit and PR. Not seeing any improvement.

Was there something else you were doing, or different URLs that aren't working with v3.3.15?

Oh, the URLs were different; Trying with the ones listed in the issue - I get only timeouts as well. I'm only a hobbyist, so my apologies if speaking nonsense - but could this be due to the response the sites send to for the challenge request (which flaresolverr perhaps currently can't see?) ? I manually resolved a few challenges on both the affected URLs and ones which work fine; The affected ones have no Nel header in the response; And also seem to log the challenge resolution differently - based on origin - like https://www.fanfiction.net/eye/1/1/58292627/14316251/ shows me either my IPv4 or IPv6 address, country, and some //old story comment; and deleting cookies for the site did not bring me to the challenge again, it was still considered passed.

ilike2burnthing commented 3 months ago

Could be, I'm not sure. PRs and ideas are always welcome. Thanks.

erdemyunsel commented 3 months ago

Hello,

Same issue here. My url is https://otokocbornova.sahibinden.com/ but i cant pass the cloudflare challange.

332123342 commented 3 months ago

@ilike2burnthing flaresolverr:v3.3.16 still not working with "https://fortnite.gg/player-count?page=1"

308097160-2031fe54-cbef-45d5-af2f-cdb99d77d210 
2024-02-29 06:41:45 INFO     ReqId 140184289924864 Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://fortnite.gg/player-count?page=1', 'maxTimeout': 60000}
2024-02-29 06:41:45 DEBUG    ReqId 140184289924864 Launching web browser...
version_main cannot be converted to an integer
2024-02-29 06:41:45 DEBUG    ReqId 140184289924864 Started executable: `/app/chromedriver` in a child process with pid: 1281
2024-02-29 06:41:46 DEBUG    ReqId 140184289924864 New instance of webdriver has been created to perform the request
2024-02-29 06:41:46 DEBUG    ReqId 140184264746752 Navigating to... https://fortnite.gg/player-count?page=1
2024-02-29 06:41:48 INFO     ReqId 140184264746752 Challenge detected. Selector found: #cf-please-wait
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for title (attempt 1): Just a moment...
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for title (attempt 1): DDoS-Guard
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 1): #cf-challenge-running
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 1): .ray_id
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 1): .attack-box
2024-02-29 06:41:48 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 1): #cf-please-wait
2024-02-29 06:41:49 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:41:49 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:41:49 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:41:49 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:41:49 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for title (attempt 2): Just a moment...
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for title (attempt 2): DDoS-Guard
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 2): #cf-challenge-running
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 2): .ray_id
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 2): .attack-box
2024-02-29 06:41:51 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 2): #cf-please-wait
2024-02-29 06:41:52 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:41:52 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:41:52 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:41:52 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:41:53 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for title (attempt 3): Just a moment...
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for title (attempt 3): DDoS-Guard
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 3): #cf-challenge-running
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 3): .ray_id
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 3): .attack-box
2024-02-29 06:41:55 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 3): #cf-please-wait
2024-02-29 06:41:56 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:41:56 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:41:56 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:41:56 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:41:56 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for title (attempt 4): Just a moment...
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for title (attempt 4): DDoS-Guard
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 4): #cf-challenge-running
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 4): .ray_id
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 4): .attack-box
2024-02-29 06:41:58 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 4): #cf-please-wait
2024-02-29 06:41:59 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:41:59 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:41:59 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:41:59 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:41:59 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for title (attempt 5): Just a moment...
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for title (attempt 5): DDoS-Guard
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 5): #cf-challenge-running
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 5): .ray_id
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 5): .attack-box
2024-02-29 06:42:01 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 5): #cf-please-wait
2024-02-29 06:42:02 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:02 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:02 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:02 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:02 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for title (attempt 6): Just a moment...
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for title (attempt 6): DDoS-Guard
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 6): #cf-challenge-running
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 6): .ray_id
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 6): .attack-box
2024-02-29 06:42:04 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 6): #cf-please-wait
2024-02-29 06:42:05 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:05 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:05 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:05 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:05 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for title (attempt 7): Just a moment...
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for title (attempt 7): DDoS-Guard
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 7): #cf-challenge-running
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 7): .ray_id
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 7): .attack-box
2024-02-29 06:42:07 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 7): #cf-please-wait
2024-02-29 06:42:08 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:08 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:08 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:08 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:08 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for title (attempt 8): Just a moment...
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for title (attempt 8): DDoS-Guard
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 8): #cf-challenge-running
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 8): .ray_id
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 8): .attack-box
2024-02-29 06:42:10 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 8): #cf-please-wait
2024-02-29 06:42:11 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:11 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:11 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:11 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:11 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:13 DEBUG    ReqId 140184264746752 Waiting for title (attempt 9): Just a moment...
2024-02-29 06:42:13 DEBUG    ReqId 140184264746752 Waiting for title (attempt 9): DDoS-Guard
2024-02-29 06:42:13 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 9): #cf-challenge-running
2024-02-29 06:42:13 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 9): .ray_id
2024-02-29 06:42:13 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 9): .attack-box
2024-02-29 06:42:14 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 9): #cf-please-wait
2024-02-29 06:42:15 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:15 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:15 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:15 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:15 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for title (attempt 10): Just a moment...
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for title (attempt 10): DDoS-Guard
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 10): #cf-challenge-running
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 10): .ray_id
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 10): .attack-box
2024-02-29 06:42:17 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 10): #cf-please-wait
2024-02-29 06:42:18 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:18 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:18 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:18 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:18 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for title (attempt 11): Just a moment...
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for title (attempt 11): DDoS-Guard
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 11): #cf-challenge-running
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 11): .ray_id
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 11): .attack-box
2024-02-29 06:42:20 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 11): #cf-please-wait
2024-02-29 06:42:21 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:21 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:22 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox found and clicked!
2024-02-29 06:42:22 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:22 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for title (attempt 12): Just a moment...
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for title (attempt 12): DDoS-Guard
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 12): #cf-challenge-running
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 12): .ray_id
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 12): .attack-box
2024-02-29 06:42:24 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 12): #cf-please-wait
2024-02-29 06:42:25 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:25 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:25 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:25 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:25 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for title (attempt 13): Just a moment...
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for title (attempt 13): DDoS-Guard
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 13): #cf-challenge-running
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 13): .ray_id
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 13): .attack-box
2024-02-29 06:42:27 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 13): #cf-please-wait
2024-02-29 06:42:28 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:28 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:28 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:28 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:28 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for title (attempt 14): Just a moment...
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for title (attempt 14): DDoS-Guard
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 14): #cf-challenge-running
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 14): .ray_id
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 14): .attack-box
2024-02-29 06:42:30 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 14): #cf-please-wait
2024-02-29 06:42:31 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:31 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:31 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:31 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:32 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for title (attempt 15): Just a moment...
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for title (attempt 15): DDoS-Guard
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 15): #cf-challenge-running
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 15): .ray_id
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 15): .attack-box
2024-02-29 06:42:34 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 15): #cf-please-wait
2024-02-29 06:42:35 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:35 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:35 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:35 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:35 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for title (attempt 16): Just a moment...
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for title (attempt 16): DDoS-Guard
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 16): #cf-challenge-running
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 16): .ray_id
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 16): .attack-box
2024-02-29 06:42:37 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 16): #cf-please-wait
2024-02-29 06:42:38 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:38 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:38 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:38 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:38 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for title (attempt 17): Just a moment...
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for title (attempt 17): DDoS-Guard
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 17): #cf-challenge-running
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 17): .ray_id
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 17): .attack-box
2024-02-29 06:42:40 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 17): #cf-please-wait
2024-02-29 06:42:41 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:41 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:41 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:41 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:41 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for title (attempt 18): Just a moment...
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for title (attempt 18): DDoS-Guard
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 18): #cf-challenge-running
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 18): .ray_id
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 18): .attack-box
2024-02-29 06:42:43 DEBUG    ReqId 140184264746752 Waiting for selector (attempt 18): #cf-please-wait
2024-02-29 06:42:44 DEBUG    ReqId 140184264746752 Timeout waiting for selector
2024-02-29 06:42:44 DEBUG    ReqId 140184264746752 Try to find the Cloudflare verify checkbox...
2024-02-29 06:42:44 DEBUG    ReqId 140184264746752 Cloudflare verify checkbox not found on the page.
2024-02-29 06:42:44 DEBUG    ReqId 140184264746752 Try to find the Cloudflare 'Verify you are human' button...
2024-02-29 06:42:44 DEBUG    ReqId 140184264746752 The Cloudflare 'Verify you are human' button not found on the page.
2024-02-29 06:42:46 DEBUG    ReqId 140184289924864 A used instance of webdriver has been destroyed
2024-02-29 06:42:46 ERROR    ReqId 140184289924864 Error: Error solving the challenge. Timeout after 60.0 seconds.
2024-02-29 06:42:46 DEBUG    ReqId 140184289924864 Response => POST /v1 body: {'status': 'error', 'message': 'Error: Error solving the challenge. Timeout after 60.0 seconds.', 'startTimestamp': 1709188905900, 'endTimestamp': 1709188966603, 'version': '3.3.16'}
2024-02-29 06:42:46 INFO     ReqId 140184289924864 Response in 60.703 s
2024-02-29 06:42:46 INFO     ReqId 140184289924864 172.17.0.1 POST http://localhost:8191/v1 500 Internal Server Error
apo208 commented 3 months ago

Just tried with latest version, still occurs with following on debug.

Details

flaresolverr | 2024-02-29 12:19:05 INFO ReqId 139692888872704 Challenge detected. Title found: Just a moment... flaresolverr | 2024-02-29 12:19:05 DEBUG ReqId 139692888872704 Waiting for title (attempt 1): Just a moment... flaresolverr | 2024-02-29 12:19:06 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:06 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:06 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:06 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:06 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:08 DEBUG ReqId 139692888872704 Waiting for title (attempt 2): Just a moment... flaresolverr | 2024-02-29 12:19:09 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:09 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:10 DEBUG ReqId 139692888872704 Cloudflare verify checkbox found and clicked! flaresolverr | 2024-02-29 12:19:10 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:10 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:12 DEBUG ReqId 139692888872704 Waiting for title (attempt 3): Just a moment... flaresolverr | 2024-02-29 12:19:13 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:13 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:14 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:14 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:14 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:16 DEBUG ReqId 139692888872704 Waiting for title (attempt 4): Just a moment... flaresolverr | 2024-02-29 12:19:17 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:17 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:17 DEBUG ReqId 139692888872704 Cloudflare verify checkbox found and clicked! flaresolverr | 2024-02-29 12:19:17 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:18 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:20 DEBUG ReqId 139692888872704 Waiting for title (attempt 5): Just a moment... flaresolverr | 2024-02-29 12:19:21 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:21 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:21 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:21 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:21 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:23 DEBUG ReqId 139692888872704 Waiting for title (attempt 6): Just a moment... flaresolverr | 2024-02-29 12:19:24 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:24 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:25 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:25 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:25 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:27 DEBUG ReqId 139692888872704 Waiting for title (attempt 7): Just a moment... flaresolverr | 2024-02-29 12:19:28 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:28 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:28 DEBUG ReqId 139692888872704 Cloudflare verify checkbox found and clicked! flaresolverr | 2024-02-29 12:19:28 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:28 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:30 DEBUG ReqId 139692888872704 Waiting for title (attempt 8): Just a moment... flaresolverr | 2024-02-29 12:19:31 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:31 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:31 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:31 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:31 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:33 DEBUG ReqId 139692888872704 Waiting for title (attempt 9): Just a moment... flaresolverr | 2024-02-29 12:19:34 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:34 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:34 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:34 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:34 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:36 DEBUG ReqId 139692888872704 Waiting for title (attempt 10): Just a moment... flaresolverr | 2024-02-29 12:19:37 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:37 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:37 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:37 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:38 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:40 DEBUG ReqId 139692888872704 Waiting for title (attempt 11): Just a moment... flaresolverr | 2024-02-29 12:19:41 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:41 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:41 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:41 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:41 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:43 DEBUG ReqId 139692888872704 Waiting for title (attempt 12): Just a moment... flaresolverr | 2024-02-29 12:19:44 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:44 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:44 DEBUG ReqId 139692888872704 Cloudflare verify checkbox found and clicked! flaresolverr | 2024-02-29 12:19:44 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:44 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:46 DEBUG ReqId 139692888872704 Waiting for title (attempt 13): Just a moment... flaresolverr | 2024-02-29 12:19:47 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:47 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:47 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:47 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:48 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:50 DEBUG ReqId 139692888872704 Waiting for title (attempt 14): Just a moment... flaresolverr | 2024-02-29 12:19:51 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:51 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:51 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:51 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:51 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:53 DEBUG ReqId 139692888872704 Waiting for title (attempt 15): Just a moment... flaresolverr | 2024-02-29 12:19:54 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:54 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:54 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:54 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:54 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:56 DEBUG ReqId 139692888872704 Waiting for title (attempt 16): Just a moment... flaresolverr | 2024-02-29 12:19:57 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:19:57 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:19:57 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:19:57 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:19:57 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:19:59 DEBUG ReqId 139692888872704 Waiting for title (attempt 17): Just a moment... flaresolverr | 2024-02-29 12:20:00 DEBUG ReqId 139692888872704 Timeout waiting for selector flaresolverr | 2024-02-29 12:20:00 DEBUG ReqId 139692888872704 Try to find the Cloudflare verify checkbox... flaresolverr | 2024-02-29 12:20:00 DEBUG ReqId 139692888872704 Cloudflare verify checkbox not found on the page. flaresolverr | 2024-02-29 12:20:00 DEBUG ReqId 139692888872704 Try to find the Cloudflare 'Verify you are human' button... flaresolverr | 2024-02-29 12:20:00 DEBUG ReqId 139692888872704 The Cloudflare 'Verify you are human' button not found on the page. flaresolverr | 2024-02-29 12:20:02 DEBUG ReqId 139692888872704 Waiting for title (attempt 18): Just a moment... flaresolverr | 2024-02-29 12:20:02 DEBUG ReqId 139692922443520 A used instance of webdriver has been destroyed flaresolverr | 2024-02-29 12:20:02 ERROR ReqId 139692922443520 Error: Error solving the challenge. Timeout after 60.0 seconds.

ilike2burnthing commented 3 months ago

v3.3.16 is not a fix for this issue. When one is available I'll post here.

antanilol commented 3 months ago

Hi, I have the same problem for itorrents.org

andtoliom commented 2 months ago

Flaresolverr stopped to work since the new Chrome v123, won't solve any challenge, just will be looping. Using FlareSolverr 3.3.16.

ilike2burnthing commented 2 months ago

Bar a few exceptions, it's largely working fine.

dreammis commented 2 months ago

today I scrap the annas site also loop, all work well before today. I update the 3.3.16, can't work also. site: https://annas-archive.org/slow_download/68b5eed3bb48e093f98842202cbd8842/0/0

HellxGodLike commented 2 months ago

2024-03-25 21:16:40 INFO Challenge detected. Title found: Just a moment... 2024-03-25 21:18:33 ERROR Error: Error solving the challenge. Timeout after 120.0 seconds. 2024-03-25 21:18:33 INFO Response in 122.014 s 2024-03-25 21:18:33 INFO 127.0.0.1 POST http://localhost:8191/v1 500 Internal Server Error

I don't know if everyone facing this issue or not. I am newest version of flaresolver available

Vrozaksen commented 2 months ago

Yep, same issue

ipaqi commented 2 months ago

Can one of the guys working on this item describe what they believe is going on with sites like ff.net that still have issues? I might have some time this weekend to futz around and help a bit if I knew where to start looking

ilike2burnthing commented 2 months ago

Something is being detected by Cloudflare, causing the challenge to loop, but no idea what. A previous challenge loop for YGGtorrent was resolved by introducing the LANG ENV and setting it to an English language code, but that's not the case here.

FFace32 commented 2 months ago

Can one of the guys working on this item describe what they believe is going on with sites like ff.net that still have issues? I might have some time this weekend to futz around and help a bit if I knew where to start looking

I'd like to note that this https://github.com/FlareSolverr/FlareSolverr/issues/1036#issuecomment-1956828758 no longer works with proxies. (At least not with data center ones and at least on one site which I'm scraping, not included in the list on this issue.)

What I've noticed is that with botasaurus, if you open the website and manually click the cloud flare checkbox, you pass it.

juanfrilla commented 2 months ago

From what I've heard cloudflare detects that you are running through a docker container, I don't know if this could be the reason why cloudflare detect us.

FFace32 commented 2 months ago

From what I've heard cloudflare detects that you are running through a docker container, I don't know if this could be the reason why cloudflare detect us.

The issue I talked about earlier happens outside of a container.

marios88 commented 2 months ago

From what I've heard cloudflare detects that you are running through a docker container, I don't know if this could be the reason why cloudflare detect us.

No, i tested it, this happens even without docker and headless

xulei324 commented 2 months ago

Mozilla/5.0 (Linux; Android 13; SM-G530BT) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36

I confirm, I added the user-agent and it worked again

https://github.com/FlareSolverr/FlareSolverr/blob/df06d13cf8f9e4ea71a22057af12e570ff3e98d4/src/utils.py#L132

Just add after line before: options.add_argument('--user-agent=Mozilla/5.0 (Linux; Android 13; SM-G530BT) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36')

I deploy from source code and update utils.py. But not working to me. 

root@localhost:~/FlareSolverr# python3 src/flaresolverr.py
2024-03-29 11:07:16 INFO     FlareSolverr 3.3.16
2024-03-29 11:07:16 INFO     Testing web browser installation...
2024-03-29 11:07:16 INFO     Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.31
2024-03-29 11:07:16 INFO     Chrome / Chromium path: /bin/google-chrome-stable
2024-03-29 11:07:16 INFO     Chrome / Chromium major version: 123
2024-03-29 11:07:16 INFO     Launching web browser...
2024-03-29 11:07:18 INFO     FlareSolverr User-Agent: Mozilla/5.0 (Linux; Android 13; SM-G530BT) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36
2024-03-29 11:07:18 INFO     Test successful!
2024-03-29 11:07:18 INFO     Serving on http://0.0.0.0:8191
2024-03-29 11:07:26 INFO     Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://www.aei.org/press/', 'maxTimeout': 60000}
2024-03-29 11:08:27 ERROR    Error: Error solving the challenge. Timeout after 60.0 seconds.
2024-03-29 11:08:27 INFO     Response in 61.052 s
Maiikoo commented 2 months ago

update for flaresolverr soon ? not working with yggtorrent

bournezjc commented 2 months ago

It seems that Selenium + undected_chromedriver does not pass CDP detection. Probably you could check this website (https://www.browserscan.net/en/bot-detection) to see the bot detection result. I tried to turn off the dev-tools in new window, but still failing to pass the CDP detection. Any help?

ilike2burnthing commented 2 months ago

Probably similar to https://github.com/FlareSolverr/FlareSolverr/issues/1000

PrzemekSkw commented 2 months ago

Hello @ilike2burnthing will be some fix for that? Regards.

juanfrilla commented 2 months ago

Hello @ilike2burnthing will be some fix for that? Regards.

I think yes, but I guess they don't even know why cloudflare is detecting flaresolverr, when they discover why, they'll update. Last cloudflare update is detecting everything, things are going hard to bypass cloudflare. Even though, It has being said more on the top that PR's are welcome

PrzemekSkw commented 2 months ago

Hello @ilike2burnthing will be some fix for that? Regards.

I think yes, but I guess they don't even know why cloudflare is detecting flaresolverr, when they discover why, they'll update. Last cloudflare update is detecting everything, things are going hard to bypass cloudflare. Even though, It has being said more on the top that PR's are welcome

Thanks for Your answer. I will wait for fix then :) Regards.

qq846511277 commented 1 month ago

https://platesmania.com/jp/gallery.php?gal=jp&ctype=2 This URL challenge won't pass either

skillix commented 1 month ago

Problem solved by adding the language environment variable.

https://github.com/FlareSolverr/FlareSolverr#environment-variables

PrzemekSkw commented 1 month ago

Problem solved by adding the language environment variable.

https://github.com/FlareSolverr/FlareSolverr#environment-variables

Hi, it don't fix for me. I try to add container variable like pl_PL or en_GB but nothing change. Regards.

skillix commented 1 month ago

Problem solved by adding the language environment variable. https://github.com/FlareSolverr/FlareSolverr#environment-variables

Hi, it don't fix for me. I try to add container variable like pl_PL or en_GB but nothing change. Regards.

I thought I had already made this change, but I noticed that it was not taken into account, by changing the log level in debug, I noticed that it remained in info and therefore the variable for the language n It wasn't taken into account either. For me on Windows, I had to add them to my session user and not to the system for this to be taken into account. I don't know how to do it with your system, but the log level test allows you to be sure that the variable in language is correctly applied or not

ilike2burnthing commented 1 month ago

Are you specifically referring to YGGtorrent, or is there another site this is required for?

skillix commented 1 month ago

This is the only indexer I use with cloudflare. But it's worth checking with those who have the problem on other sites.

ilike2burnthing commented 1 month ago

The LANG ENV was specifically added for YGGtorrent, no other sites have required this. The same sites fail with or without it for me.

s1lverkin commented 1 month ago

How do I verify if the LANG flag is working fine? Not seeing any info in the debug logs about it, despite checking the variable directly under docker container

ilike2burnthing commented 1 month ago

Enable debug logging, send a request to FlareSolverr for google.com, the returned HTML should be in the language you set.

lucos11 commented 1 month ago

Got the same issue as well. google is always returning in French even if LANG is set to EN. Here is the return of docker exec flaresolverr locale

LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_CTYPE="en_US.UTF8" LC_NUMERIC="en_US.UTF8" LC_TIME="en_US.UTF8" LC_COLLATE="en_US.UTF8" LC_MONETARY="en_US.UTF8" LC_MESSAGES="en_US.UTF8" LC_PAPER="en_US.UTF8" LC_NAME="en_US.UTF8" LC_ADDRESS="en_US.UTF8" LC_TELEPHONE="en_US.UTF8" LC_MEASUREMENT="en_US.UTF8" LC_IDENTIFICATION="en_US.UTF8" LC_ALL=en_US.UTF8 locale: Cannot set LC_CTYPE to default locale: No such file or directory locale: Cannot set LC_MESSAGES to default locale: No such file or directory locale: Cannot set LC_ALL to default locale: No such file or directory

But if I try visiting https://www.google.com/ncr (no country redirect), which sets a browser cookie to prevent redirect to the local Google country service. then the page is returned in English. It may explain why we are facing this issue but don't know how to solve it

ilike2burnthing commented 1 month ago

I tried installing locales and locales-all, setting LANG, LANGUAGE, and LC_ALL, but when checking with https://www.localeplanet.com/support/browser.html and https://gtranslate.io/detect-browser-language there is no change. I'll try a few more things later, but it's just limited to Linux/Docker.