ClericPy / ichrome

Chrome controller for Humans, based on Chrome Devtools Protocol(CDP) and python3.7+.
https://pypi.org/project/ichrome/
MIT License
227 stars 29 forks source link

Is it possible to run multiple AsyncChromeDaemon's at the same time? #131

Closed juanfrilla closed 1 year ago

juanfrilla commented 1 year ago

I'm running multiple like this:

        async with AsyncChromeDaemon(
            headless=True,
            proxy=proxy,
            chrome_path=chrome_path,
            user_agent=self._headers["User-Agent"],
        ) as browser:
            async with browser.incognito_tab(proxyServer=proxy) as tab:
                await tab.goto(self.origin + "/document/" + doc_id, timeout=60000)
                result = await tab.wait_tag("#head", max_wait_time=60)
                html = await tab.html
                cookies = await tab.get_cookies_dict()

But I'm receiving this error:

"/Users/juanfranciscomartinrodriguez/modulos/scrapy/latam/jurisprudenciapy3/jurisprudenciapy3/spiders/jCostaRicaJoderJudicial.py", line 588, in navigate_ichrome
    async with AsyncChromeDaemon(
  File "/Users/juanfranciscomartinrodriguez/miniconda3/envs/p38/lib/python3.8/site-packages/ichrome/daemon.py", line 938, in __aenter__
    return await self._init_coro
  File "/Users/juanfranciscomartinrodriguez/miniconda3/envs/p38/lib/python3.8/site-packages/ichrome/daemon.py", line 807, in _init_chrome_daemon
    await self.launch_chrome()
  File "/Users/juanfranciscomartinrodriguez/miniconda3/envs/p38/lib/python3.8/site-packages/ichrome/daemon.py", line 836, in launch_chrome
    raise ChromeRuntimeError(error)
ichrome.exceptions.ChromeRuntimeError: launch_chrome failed for proc not ok
ClericPy commented 1 year ago

differ the port with port arg. because each port is singleton

ClericPy commented 1 year ago

actually, incognito_tab is Environmentally isolated, so you can learn to use asyncio.create_task for a real coroutine usage

juanfrilla commented 1 year ago

thanks @ClericPy , so If I put a different port for every AsyncChromeDaemon it should work

ClericPy commented 1 year ago

yes. host+port is singleton.

juanfrilla commented 1 year ago

Thanks, I created a method :

    def find_free_port(self):
        # Create a temporary socket to find a free port.
        temp_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        temp_socket.bind(('localhost', 0))  # Bind to any available local address and port 0.
        _, port = temp_socket.getsockname()  # Get the assigned port.
        temp_socket.close()  # Close the temporary socket.
        return port

And then:

         async with AsyncChromeDaemon(
            headless=True,
            proxy=proxy,
            port = self.find_free_port(),
            chrome_path=chrome_path,
            user_agent=self._headers["User-Agent"],
        ) as browser:
            async with browser.incognito_tab(proxyServer=proxy) as tab:
                await tab.goto(self.origin + "/document/" + doc_id, timeout=60000)
                result = await tab.wait_tag("#head", max_wait_time=60)
                html = await tab.html
                cookies = await tab.get_cookies_dict()