Firecrawl is blocked by Cloudflare

[VakhoGeoLab]

Problem Description Firecrawl is a nice tool but it is stopped by the free CloudFlare bot protection service.

Proposed Feature If Firecrawl could bypass cloudflare, it could be unstoppable service.

[JakobStadlhuber]

@VakhoGeoLab do you use the self hosted or the cloud version?

[VakhoGeoLab]

I’ve tried both. Selfhosted from my computer and also used playground for testing and cloudflare bot protection works.

On Mon, Aug 5, 2024 at 2:49 AM Jakob Stadlhuber @.***> wrote:

@VakhoGeoLab https://github.com/VakhoGeoLab do you use the self hosted or the cloud version?

— Reply to this email directly, view it on GitHub https://github.com/mendableai/firecrawl/issues/495#issuecomment-2267846093, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD3UF37AOS2XPN5AV6WOPL3ZP2VWFAVCNFSM6AAAAABL6HTSJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRXHA2DMMBZGM . You are receiving this because you were mentioned.Message ID: @.***>

[nickscamara]

ccing @tomkosm here

[nickscamara]

@VakhoGeoLab which URLs are getting blocked?

[VakhoGeoLab]

I'm testing https://coinmania.ge a crypto exchange. I'm building the bot for checking crypto prices.

[iuliaturc]

It also gets blocked by Cloudflare when trying to scrape any subpages of https://jax.readthedocs.io/en/latest/

[nickscamara]

@tomkosm can you look into those as soon as you get a chance? tks

[tomkosm]

It also gets blocked by Cloudflare when trying to scrape any subpages of https://jax.readthedocs.io/en/latest/

Hey, I am not able to replicate this, it seems to work for me now. Cloudflare has different triggers for the high protection modes, try again and let me know if you encounter this issue again, also please provide the exact link and the error you got.

[tomkosm]

I'm testing https://coinmania.ge a crypto exchange. I'm building the bot for checking crypto prices.

This website requires cloudflare captcha solving to access, working on it but cant guarantee that we will be able to support it.