search

techwithtim / AI-Web-Scraper

An AI web scraper using ollama, brightdata, selenium and other libraries.
200 stars 81 forks source link

The code just fails with a python TypeError #4

Open mpsOxygen opened 3 weeks ago

mpsOxygen commented 3 weeks ago

This is the error that it gives when I try to scrape a website.

TypeError: a bytes-like object is required, not 'str'
Traceback:

File "folder/aiwebscrapper/ai/lib64/python3.12/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
             ^^^^^^
File "folder/aiwebscrapper/ai/lib64/python3.12/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 590, in code_to_exec
    exec(code, module.__dict__)
File "folder/aiwebscrapper/main.py", line 20, in <module>
    dom_content = scrape_website(url)
                  ^^^^^^^^^^^^^^^^^^^
File "folder/aiwebscrapper/scrape.py", line 14, in scrape_website
    sbr_connection = ChromiumRemoteConnection(SBR_WEBDRIVER, "goog", "chrome")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "folder/aiwebscrapper/ai/lib64/python3.12/site-packages/selenium/webdriver/chromium/remote_connection.py", line 30, in __init__
    super().__init__(remote_server_addr, keep_alive, ignore_proxy)
File "folder/aiwebscrapper/ai/lib64/python3.12/site-packages/selenium/webdriver/remote/remote_connection.py", line 274, in __init__
    if n_url.path in remote_add.netloc:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BrianMosbeux commented 16 hours ago

You should share your code...

I think it is the SBR_WEBDRIVER variable that is causing the problem in scrape.py. Maybe you did not set up the .env file propery?