Error with VPN & error with selenium-profiles

ruslanx3m commented 1 year ago

updated ... 1.3.2 broke all , not work for me

from selenium_driverless.sync import webdriver
from selenium_profiles.webdriver import Chrome
import json

def load_profile():
    try:
        with open("data.json", "r") as f:
            profile = json.load(f)
            return profile
    except Exception as e:
        print(e)
        return None

def test_driverless():
    options = webdriver.ChromeOptions()
    profile = load_profile()
    driver = Chrome(profile=profile, options=options, driverless_options=True)
    driver.get('http://nowsecure.nl#relax')
    driver.wait_for_cdp(event="Page.loadEventFired", timeout=5)

if __name__ == "__main__":
    test_driverless()

zzzzzzzzzz

httpx.RemoteProtocolError: Server disconnected without sending a response.

kaliiiiiiiiii commented 1 year ago

@ruslanx3m please provide

the last selenium-driverless version that worked
the selenium-driverless version that doesn't work
the current version of cdp-socket
Did chrome start (window open?)

ruslanx3m commented 1 year ago

temporarily returned to 1.2.6 .. there will be time to check and write here

kaliiiiiiiiii commented 1 year ago

I think pip install selenium-driverless==1.3.2.1 might fix it again

ruslanx3m commented 1 year ago

hi, installed 1.3.2.1

  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions    
    yield
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request      
    resp = await self._pool.handle_async_request(req)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\connection_pool.py", line 262, in handle_async_request
    raise exc
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\http_proxy.py", line 192, in handle_async_request     
    return await self._connection.handle_async_request(proxy_request)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\connection.py", line 96, in handle_async_request      
    return await self._connection.handle_async_request(request)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\http11.py", line 121, in handle_async_request
    raise exc
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\http11.py", line 164, in _receive_response_headers    
    event = await self._receive_event(timeout=timeout)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpcore\_async\http11.py", line 214, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "d:\!Programing\!Test\App\tokens+\async_tokens.py", line 18, in <module>
    asyncio.run(main())
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
    return future.result()
  File "d:\!Programing\!Test\App\tokens+\async_tokens.py", line 7, in main
    async with webdriver.Chrome(options=options) as driver:
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\webdriver.py", line 134, in __aenter__
    await self.start_session()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\selenium_driverless\webdriver.py", line 220, in start_session
    self._base = await CDPSocket(port=port, host=host, loop=self._loop)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\socket.py", line 186, in start_session
    return await asyncio.wait_for(self._connect(), timeout=timeout)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\asyncio\tasks.py", line 445, in wait_for
    return fut.result()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\socket.py", line 189, in _connect
    ws_url = await get_websock_url(self._port, self._host_, timeout=None)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\utils\conn.py", line 25, in get_websock_url
    _json = await get_json(host, timeout=timeout)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\utils\conn.py", line 16, in get_json
    res = await get_http(f"http://{host}/json", timeout=timeout)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\cdp_socket\utils\conn.py", line 8, in get_http
    result = await client.get(url=url, timeout=timeout)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1757, in get
    return await self.request(
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_transports\default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python310\lib\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

cdp-socket - 1.0.10

from selenium_driverless import webdriver
import asyncio

async def main():
    options = webdriver.ChromeOptions()
    async with webdriver.Chrome(options=options) as driver:
        await driver.get('http://nowsecure.nl#relax')
        # wait for redirected page
        await driver.wait_for_cdp(event="Page.loadEventFired", timeout=5)

        title = await driver.title
        url = await driver.current_url
        source = await driver.page_source
        print(title)

asyncio.run(main())

chrome start - window open and disconnected

ruslanx3m commented 1 year ago

hi, reopened, if I use vpn (Psiphon 3) what is using localhost:port ... got disconnected 1.2.6 - is working fine with vpn 1.3.2 - not! , localhost or port conflict

start_session - here is conflict or websockets lib

I cant use proxy (selenium_profiles) if I start task using asyncio.run -> got error: Cannot run the event loop while another loop is running .. so I only can use 1.2.6 version in sync with profiles

Cannot run the event loop while another loop is running
Cancelling tasks...
Exception ignored in atexit callback: <bound method Injector.stop of <selenium_injector.scripts.injector.Injector object at 0x0000027B51A60AD0>>Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_injector\scripts\injector.py", line 194, in stop     
    self.socket.stop()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_injector\scripts\sync_websocket.py", line 179, in stop
    self.ws_server.ws_server.close()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\server.py", line 750, in close
    self.close_task = self.get_loop().create_task(
                      ^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\server.py", line 832, in get_loop
    return self.server.get_loop()
           ^^^^^^^^^^^
AttributeError: 'WebSocketServer' object has no attribute 'server'
Exception ignored in atexit callback: <bound method Injector.stop of <selenium_injector.scripts.injector.Injector object at 0x0000027B4F843010>>Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_injector\scripts\injector.py", line 194, in stop
    self.socket.stop()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_injector\scripts\sync_websocket.py", line 179, in stop
    self.ws_server.ws_server.close()
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\server.py", line 750, in close
    self.close_task = self.get_loop().create_task(
                      ^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\server.py", line 832, in get_loop
    return self.server.get_loop()
           ^^^^^^^^^^^
AttributeError: 'WebSocketServer' object has no attribute 'server'

kaliiiiiiiiii commented 1 year ago

@ruslanx3m seems to me like the issue occurs since I removed the PyCDP dependency and use CDP-Socket instead. maybe it's a bug in httpx

Can you try the following script?

from cdp_socket.utils.utils import launch_chrome, random_port
from cdp_socket.utils.conn import get_websock_url

from cdp_socket.socket import SingleCDPSocket

import os
import asyncio

async def main():
    PORT = random_port()
    process = launch_chrome(PORT)

    websock_url = await get_websock_url(PORT, timeout=5)
    async with SingleCDPSocket(websock_url, timeout=5) as sock:
        targets = await sock.exec("Target.getTargets")
        print(targets)

    os.kill(process.pid, 15)

asyncio.run(main())

I assume that about the same error will occur. Also, provide what your cdp-socket version is when you test it

ruslanx3m commented 1 year ago

no vpn:

{'targetInfos': [{'targetId': '91A0D0B5166C4D9954848720C419540C', 'type': 'page', 'title': 'New Tab', 'url': 'chrome://newtab/', 'attached': True, 'canAccessOpener': False, 'browserContextId': 'A15CDBB3296007900FDACBA05CC275BD'}, {'targetId': 'C63F4FE0EC6392298FECD93926FCD965', 'type': 'iframe', 'title': 
'chrome-untrusted://new-tab-page/one-google-bar?paramsencoded=', 'url': 'chrome-untrusted://new-tab-page/one-google-bar?paramsencoded=', 'attached': False, 'canAccessOpener': False, 'browserContextId': 'A15CDBB3296007900FDACBA05CC275BD'}]}

with vpn:

Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions    
    yield
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request      
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\connection_pool.py", line 262, in handle_async_request
    raise exc
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\http_proxy.py", line 192, in handle_async_request     
    return await self._connection.handle_async_request(proxy_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\connection.py", line 96, in handle_async_request      
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\http11.py", line 121, in handle_async_request
    raise exc
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\http11.py", line 99, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\http11.py", line 164, in _receive_response_headers     
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpcore\_async\http11.py", line 214, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "d:\test\test_1.py", line 22, in <module>
    asyncio.run(main())
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "d:\test\test_1.py", line 14, in main
    websock_url = await get_websock_url(PORT, timeout=5)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\utils\conn.py", line 25, in get_websock_url
    _json = await get_json(host, timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\utils\conn.py", line 16, in get_json
    res = await get_http(f"http://{host}/json", timeout=timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\utils\conn.py", line 8, in get_http
    result = await client.get(url=url, timeout=timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1757, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_transports\default.py", line 352, in handle_async_request       
    with map_httpcore_exceptions():
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions     
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

cdp-socket v1.0.10

kaliiiiiiiiii commented 1 year ago

@ruslanx3m

how do you specify your chrome to use your vpn? if you do it globally, can you set a bypass for localhost connections?
Does selenium-driverless (sync and async) without the vpn?
You could try setting up a proxy instead of a global vpn
You could try using 127.0.0.1 instead of localhost, maybe the vpn directs localhost to a wrong ip-address

ruslanx3m commented 1 year ago

just use Psiphon 3 free-vpn, that use proxy windows setting
if use selenium-driverless only (sync and async) without the vpn is working fine, vpn work only 1.2.6 (PyCDP), proxy only sync.
how ? in async selenium-profiles proxy not working, vpn too, in sync work but no reason
selenium-profiles not work in async I cant set any proxy

so I can use only "selenium-driverless" 1.2.6 + "selenium-profiles" in sync only if I need proxy or vpn async selenium-driverless without proxy or vpn is useless

C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_driverless\sync\webdriver.py:13: RuntimeWarning: coroutine 'Chrome.start_session' 
was never awaited
  self.start_session()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\webdriver.py:156: RuntimeWarning: coroutine 'Chrome.get' was never awaited
  self.get("chrome://version/")  # wait browser to start
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\profiles.py:153: RuntimeWarning: coroutine 'Chrome.set_window_rect' was never awaited
  self._driver.set_window_rect(0,0,emulation["width"], emulation["height"])
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\webdriver.py:169: RuntimeWarning: coroutine 'Chrome.execute_cdp_cmd' was 
never awaited
  self.profiles.cdp_handler.apply(cdp_profile=profile["cdp"])
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py:13: RuntimeWarning: coroutine 'cdp_handler.evaluate_on_new_document' was never awaited
  cdp_handler.evaluate_on_new_document(read('js/undetected/navigator_webdriver.js'))
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py:15: RuntimeWarning: coroutine 'Chrome.execute_cdp_cmd' was never awaited
  driver.execute_cdp_cmd('Emulation.setIdleOverride', {'isUserActive': True, 'isScreenUnlocked': True})
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py:17: RuntimeWarning: coroutine 'Chrome.execute_cdp_cmd' was never awaited
  cdp_handler.remove_evaluate_on_document(1)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
User not connected

        ...
        from selenium_profiles import webdriver
        from selenium_driverless.webdriver import ChromeOptions
        ...
        async with webdriver.Chrome(            
            profile=profile,
            options=options,
            chrome_binary=chrome_binary,
            driverless_options=True,
            injector_options=True,
        ) as driver:

            # go to site
            await driver.get('http://nowsecure.nl#relax')
            ...

kaliiiiiiiiii commented 1 year ago

@ruslanx3m you should be able to use a proxy (without credentials) for selenium-driverless (sync and async with:

PROXY = "23.23.23.23:3128" # IP:PORT or HOST:PORT
chrome_options.add_argument(f'--proxy-server={PROXY}')

resource

ruslanx3m commented 1 year ago

but all my proxies is with authentication ok ... vpn I can set 127.0.0.1:1008, i'll try.. but I need profiles set custom profiles, ... can you update selenium-profiles to work in async?

now I can use 4 options 1. use sync 1.2.6 with profiles + proxy with auth + vpn

use async 1.2.6 without profiles and only vpn .. or proxy without auth
use async 1.3.2.1 without profiles,without vpn .. only proxy without auth
use sync 1.3.2.1 with profiles, proxy .. no vpn (not checked I need async)

option 1 is suitable for me, but is not async.

kaliiiiiiiiii commented 1 year ago

@ruslanx3m

ok ... vpn I can set 127.0.0.1:1008 yeah that's what I meant

Do you still get an error with the following script WITHOUT VPN ?

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles
from selenium_driverless.webdriver import ChromeOptions
from selenium_driverless.types.by import By

profile = profiles.Windows()  # or .Android
options = ChromeOptions()
# options.add_argument("--headless=new")
driver = Chrome(profile, options=options, driverless_options=True)

# get url
driver.get('https://nowsecure.nl#relax')  # test fingerprint

driver.quit()  # Execute on the End!

ruslanx3m commented 1 year ago

ok ... vpn I can set 127.0.0.1:1008 yeah that's what I meant but only 1.2.6 sync, async .. upper versions disconnected .. if vpn used

I need vpn, proxy with auth, profiles and ASYNC. .. how can this be achieved?

PS D:\test> python -u "d:\test\test_1.py"
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py:13: RuntimeWarning: coroutine 'cdp_handler.evaluate_on_new_document' was never awaited
  cdp_handler.evaluate_on_new_document(read('js/undetected/navigator_webdriver.js'))
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "d:\test\test_1.py", line 9, in <module>
    driver = Chrome(profile, options=options, driverless_options=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\webdriver.py", line 173, in __init__
    undetected.exec_cdp(self, cdp_handler=self.profiles.cdp_handler)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py", line 17, in exec_cdp
    cdp_handler.remove_evaluate_on_document(1)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\profiles.py", line 209, in remove_evaluate_on_document
    return self._driver.execute_cdp_cmd("Page.removeScriptToEvaluateOnNewDocument", {"identifier": str(identifier)})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_driverless\sync\webdriver.py", line 37, in syncified
    return self._loop.run_until_complete(item(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_driverless\webdriver.py", line 1246, in execute_cdp_cmd       
    result = await socket.exec(method=cmd, params=cmd_args, timeout=self._script_timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\socket.py", line 70, in exec
    return await asyncio.wait_for(self._responses[_id], timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
cdp_socket.exceptions.CDPError: {'code': -32000, 'message': 'Script not found'}
Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x00000192F59C3C60>
Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1338, in close_connection
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1354, in close_transport
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 761, in call_soon
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x00000192F59C3AC0>
Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1338, in close_connection
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1354, in close_transport
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 761, in call_soon
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

ruslanx3m commented 1 year ago

update not resolved the issue problem is here in cdp_socket httpx or asyncio or else ... changet all ip local ip, 127.0.0.1 ... same error

kaliiiiiiiiii commented 1 year ago

update not resolved the issue problem is here in cdp_socket httpx or asyncio or else ... changet all ip local ip, 127.0.0.1 ... same error

@ruslanx3m I think that answers a lot of questions. I think I'll be switching to aiohttp if that resolves the issue.

you can try replacing the function body of get_http with:

async with aiohttp.ClientSession() as session:
        async with session.get('http://httpbin.org/get') as resp:
            return await resp.text()

and changing json.loads(res.text) to json.loads(res)

and of course pip install aiohttp and import aiohttp

kaliiiiiiiiii commented 1 year ago

ok ... vpn I can set 127.0.0.1:1008 yeah that's what I meant but only 1.2.6 sync, async .. upper versions disconnected .. if vpn used

I need vpn, proxy with auth, profiles and ASYNC. .. how can this be achieved?

PS D:\test> python -u "d:\test\test_1.py"
C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py:13: RuntimeWarning: coroutine 'cdp_handler.evaluate_on_new_document' was never awaited
  cdp_handler.evaluate_on_new_document(read('js/undetected/navigator_webdriver.js'))
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "d:\test\test_1.py", line 9, in <module>
    driver = Chrome(profile, options=options, driverless_options=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\webdriver.py", line 173, in __init__
    undetected.exec_cdp(self, cdp_handler=self.profiles.cdp_handler)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\undetected.py", line 17, in exec_cdp
    cdp_handler.remove_evaluate_on_document(1)
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_profiles\scripts\profiles.py", line 209, in remove_evaluate_on_document
    return self._driver.execute_cdp_cmd("Page.removeScriptToEvaluateOnNewDocument", {"identifier": str(identifier)})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_driverless\sync\webdriver.py", line 37, in syncified
    return self._loop.run_until_complete(item(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium_driverless\webdriver.py", line 1246, in execute_cdp_cmd       
    result = await socket.exec(method=cmd, params=cmd_args, timeout=self._script_timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\cdp_socket\socket.py", line 70, in exec
    return await asyncio.wait_for(self._responses[_id], timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
cdp_socket.exceptions.CDPError: {'code': -32000, 'message': 'Script not found'}
Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x00000192F59C3C60>
Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1338, in close_connection
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1354, in close_transport
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 761, in call_soon
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed
Exception ignored in: <coroutine object WebSocketCommonProtocol.close_connection at 0x00000192F59C3AC0>
Traceback (most recent call last):
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1338, in close_connection
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\site-packages\websockets\legacy\protocol.py", line 1354, in close_transport
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\proactor_events.py", line 109, in close
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 761, in call_soon
  File "C:\Users\Ruslan\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 519, in _check_closed
RuntimeError: Event loop is closed

ohh I see the issue for that as well. Selenium-profiles tries to configure undetectability with use of chromedriver.

I suppose that changing if not uc_driver to if not (uc_driver or driverless_options) at /webdriver.py#L171 should solve this issue as well. I'll include it in the next release.

ruslanx3m commented 1 year ago

conn.py code updated and working with vpn :)

import json
import asyncio
import aiohttp

async def get_http(url: str, timeout:float or None=10):
    async with aiohttp.ClientSession() as session:
        async with session.get(url, timeout=timeout) as resp:
            return resp

async def get_json(host: str, timeout:float or None=10):
    res = None
    while not res:
        try:
            async with aiohttp.ClientSession() as session:
                res = await session.get(f"http://{host}/json", timeout=timeout)
                return await res.json()
        except aiohttp.ClientError:
            pass

async def get_websock_url(port: int, host: str = "localhost", timeout: float or None = 10):
    host = f"{host}:{port}"
    try:
        _json = await get_json(host, timeout=timeout)
    except asyncio.TimeoutError:
        raise asyncio.TimeoutError(f"No response from Chrome within {timeout} seconds, assuming it crashed")
    return _json[0]['webSocketDebuggerUrl']

ruslanx3m commented 1 year ago

resolved. u can update conn.py is working now! waiting for selenium-profiles update...

kaliiiiiiiiii commented 1 year ago

@ruslanx3m feel free to report if pip install selenium-profiles==2.2.10 now works:)

ruslanx3m commented 1 year ago

with patch:

import nest_asyncio nest_asyncio.apply()

RuntimeWarning: coroutine 'Chrome.start_session' was never awaited  self.start_session()
RuntimeWarning: coroutine 'Chrome.get' was never awaited  self.get("chrome://version/")  # wait browser to start
RuntimeWarning: coroutine 'Chrome.set_window_rect' was never awaited  self._driver.set_window_rect(0,0,emulation["width"], mulation["height"])
RuntimeWarning: coroutine 'Chrome.execute_cdp_cmd' was never awaited  self.profiles.cdp_handler.apply(cdp_profile=profile["cdp"])

without patch: Cannot run the event loop while another loop is running

not work in async

can you provide an example of using driverless with profiles in async ? may I do something wrong...

kaliiiiiiiiii commented 1 year ago

@ruslanx3m Hmm the following script works just fine for me:

from selenium_profiles.webdriver import Chrome
from selenium_profiles.profiles import profiles
from selenium_driverless.webdriver import ChromeOptions
from selenium_driverless.types.by import By

profile = profiles.Windows()  # or .Android
options = ChromeOptions()
# options.add_argument("--headless=new")
driver = Chrome(profile=profile, options=options, driverless_options=True)

# get url
driver.get('https://nowsecure.nl#relax')  # test fingerprint

driver.quit()  # Execute on the End!

ruslanx3m commented 1 year ago

your code is sync.

I use async, I use asyncio

kaliiiiiiiiii commented 1 year ago

your code is sync.

I use async, I use asyncio

oh sorry, didn't read your question corrrectly. Don't think selenium-profiles works async currently

kaliiiiiiiiii / Selenium-Driverless

Error with VPN & error with selenium-profiles #26