Open mitscheluk opened 1 year ago
scrape_website("Why did Elon rebrand twitter?", "https://www.axios.com/2023/08/03/twitters-x-rebrand-explained")
Gives:
Scraping website... CONTENTTTTTT: Just a moment...[www.axios.comChecking](https://file+.vscode-resource.vscode-cdn.net/Users/paul/Downloads/Chainlit-OpenAI-Functions-main/www.axios.comChecking) if the site connection is secure[www.axios.com](https://file+.vscode-resource.vscode-cdn.net/Users/paul/Downloads/Chainlit-OpenAI-Functions-main/www.axios.com) needs to review the security of your connection before proceeding.Connection is secureProceeding...Enable JavaScript and cookies to continueRay ID: 7f9b1f2eeb82cf6dPerformance & security by Cloudflare 'Just a moment...[www.axios.comChecking](https://file+.vscode-resource.vscode-cdn.net/Users/paul/Downloads/Chainlit-OpenAI-Functions-main/www.axios.comChecking) if the site connection is secure[www.axios.com](https://file+.vscode-resource.vscode-cdn.net/Users/paul/Downloads/Chainlit-OpenAI-Functions-main/www.axios.com) needs to review the security of your connection before proceeding.Connection is secureProceeding...Enable JavaScript and cookies to continueRay ID: 7f9b1f2eeb82cf6dPerformance & security by Cloudflare'
It's because browserless is not the best at bypassing Cloudflare anti browser protection.....
any work around here? im getting the same outcome
so everything seems to work (via streamlit) however - browserless isnt called at all for me - meaning the scraper isnt scraping anything. I do get serps and based on those I get outputs.