search

kaliiiiiiiiii / Selenium-Driverless

undetected Selenium without usage of chromedriver
https://kaliiiiiiiiii.github.io/Selenium-Driverless/
Other
616 stars 70 forks source link

ubuntu server 22.04 unable to load driver #156

Closed shaivilp closed 9 months ago

shaivilp commented 9 months ago

I made a flask api which includes the selenium-driverless package to scrape some data and it works flawlessly on my windows 11 machine but will open the webpage on my ubuntu 22.04 server im not even sure if it supports server versions of ubnutu

im using python 3.11.7 with ubuntu 22.04

async def getDroptime(name: str):
    async with webdriver.Chrome() as driver:
        await driver.get(f"https://namemc.com/search?q={name}")
        await driver.sleep(0.5)
        await driver.wait_for_cdp("Page.domContentEventFired", timeout=15)

        startElement = await driver.find_element(By.ID, "availability-time")
        endElement = await driver.find_element(By.ID, "availability-time2")

        start = await startElement.get_dom_attribute("datetime")
        end = await endElement.get_dom_attribute("datetime")

        startUnix = int(dateutil.parser.isoparse(start).timestamp())
        endUnix = int(dateutil.parser.isoparse(end).timestamp())

        data = {
            "startUnix": startUnix,
            "endUnix": endUnix
        }

        return data

the code freezes up at async with webdriver.Chrome() as driver:

kaliiiiiiiiii commented 9 months ago

use debug=True to start the driver & fix the errors. That's likely not ab issue with driverless, closing therefore

shaivilp commented 9 months ago

Are you able to help me figure out the issue?

here is my code

from selenium_driverless import webdriver
from selenium_driverless.types.by import By
import asyncio
import dateutil.parser

async def getDroptime(name: str):
    async with webdriver.Chrome() as driver:

        await driver.get(f"https://namemc.com/search?q={name}")
        await driver.sleep(0.5)
        await driver.wait_for_cdp("Page.domContentEventFired", timeout=15)

        startElement = await driver.find_element(By.ID, "availability-time")
        endElement = await driver.find_element(By.ID, "availability-time2")

        start = await startElement.get_dom_attribute("datetime")
        end = await endElement.get_dom_attribute("datetime")

        startUnix = int(dateutil.parser.isoparse(start).timestamp())
        endUnix = int(dateutil.parser.isoparse(end).timestamp())

        data = {
            "startUnix": startUnix,
            "endUnix": endUnix
        }

        return data

name = input("What name do you want to scrape? ")
print(asyncio.run(getDroptime(name)))

here are the logs 

root@mail:~/namemc-api# python3.11 test.py
What name do you want to scrape? G8Z
[0127/191050.794883:INFO:policy_logger.cc(144)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping mandatory platform policies because no policy file was found at: /etc/opt/chrome/policies/managed
[0127/191050.794914:INFO:policy_logger.cc(144)] :components/policy/core/common/config_dir_policy_loader.cc(118) Skipping recommended platform policies because no policy file was found at: /etc/opt/chrome/policies/recommended

DevTools listening on ws://127.0.0.1:56835/devtools/browser/a9199b6c-589b-476f-b817-2c155cf5cb0a
[0127/191050.799058:WARNING:bluez_dbus_manager.cc(248)] Floss manager not present, cannot set Floss enable/disable.
[0127/191050.826804:WARNING:sandbox_linux.cc(418)] InitializeSandbox() called with multiple threads in process gpu-process.
created driver
trying to load page
[0127/191050.893569:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'browsing-topics'.", source:  (0)
[0127/191050.893583:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.", source:  (0)
trying to get droptime
[0127/191051.116828:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'browsing-topics'.", source:  (0)
[0127/191051.116845:INFO:CONSOLE(0)] "Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.", source:  (0)
[0127/191051.512696:INFO:CONSOLE(1)] "Request for the Private Access Token challenge.", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.512889:INFO:CONSOLE(1)] "The next request for the Private Access Token challenge may return a 401 and show a warning in console.", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.524585:INFO:CONSOLE(1)] "console.groupEnd", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.685955:INFO:CONSOLE(1)] "idxk", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686010:INFO:CONSOLE(1)] "idxk", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686270:INFO:CONSOLE(1)] "[object HTMLAnchorElement]", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686293:INFO:CONSOLE(1)] "[object HTMLAnchorElement]", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686566:INFO:CONSOLE(1)] "%c%d: 1", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686597:INFO:CONSOLE(1)] "%c%d: 1", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686789:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686872:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.686980:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687066:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687171:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687250:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687357:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687437:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687549:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687629:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687736:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687822:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.687935:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.688009:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent ", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.688116:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.688195:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.688303:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.688420:INFO:CONSOLE(1)] "%c%d font-size:0;color:transparent NaN", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.689384:INFO:CONSOLE(1)] "console.groupEnd", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/orchestrate/chl_api/v1?ray=84c55ba95b3e178c (1)
[0127/191051.794296:ERROR:object_proxy.cc(576)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
[0127/191051.794314:WARNING:property.cc(144)] DaemonVersion: GetAndBlock: failed.
[0127/191051.794441:ERROR:object_proxy.cc(576)] Failed to call method: org.freedesktop.UPower.GetDisplayDevice: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
[0127/191051.794527:ERROR:object_proxy.cc(576)] Failed to call method: org.freedesktop.UPower.EnumerateDevices: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
[0127/191054.863844:INFO:CONSOLE(0)] "The resource https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/cmg/1/exg3SQHB3yvLDwq1Mc50ItJdmMy5fMYnz1bGr6wYFEE%3D was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.", source: https://challenges.cloudflare.com/cdn-cgi/challenge-platform/h/g/turnstile/if/ov2/av0/rcv0/0/i6477/0x4AAAAAAADnPIDROrmt1Wwj/light/normal (0)
Traceback (most recent call last):
  File "/usr/lib/python3.11/asyncio/tasks.py", line 500, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cdp_socket/scripts/abstract.py", line 23, in __anext__
    return await self._fut
           ^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/namemc-api/test.py", line 42, in <module>
    print(asyncio.run(getDroptime(name)))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/namemc-api/test.py", line 23, in getDroptime
    await driver.wait_for_cdp("Page.domContentEventFired", timeout=15)
  File "/usr/local/lib/python3.11/dist-packages/selenium_driverless/webdriver.py", line 1304, in wait_for_cdp
    return await target.wait_for_cdp(event=event, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/selenium_driverless/types/target.py", line 761, in wait_for_cdp
    return await self.socket.wait_for(event, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/cdp_socket/socket.py", line 112, in wait_for
    raise e
  File "/usr/local/lib/python3.11/dist-packages/cdp_socket/socket.py", line 107, in wait_for
    res = await asyncio.wait_for(_iter.__anext__(), timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/asyncio/tasks.py", line 502, in wait_for
    raise exceptions.TimeoutError() from exc
TimeoutError
kaliiiiiiiiii commented 9 months ago

mas smth to do with your desktop manager. Can't say more tho