search

SeleniumHQ / selenium

A browser automation framework and ecosystem.
https://selenium.dev
Apache License 2.0
30.51k stars 8.15k forks source link

[🐛 Bug]: page.add_script_to_evaluate_on_new_document is not working on a target session while other page.* commands works - Chromedriver 123 - Selenium 4.19.0 - Python 3.11.6 #13758

Closed HermesNWO closed 6 months ago

HermesNWO commented 6 months ago

What happened?

I try to hook TargetCreated event and deploy some commands on the target once it start.

I have struggled to understand the CDPs session protocol and it seems like there's a internal bug that need to be solved...

On the initial BIDI session the page.add_script_to_evaluate_on_new_document work as expected.

When doing page.add_script_to_evaluate_on_new_document on a new session it doesn't work while an other command like page.navigate are working...

Keep in mind I have modified the BIDI source code so we can specify a target session id when creating new BIDI session.

See also overflow : https://stackoverflow.com/questions/78240333/chrome-selenium-cdp-bidi-api-next-commands-sended-to-target-session-have-no-ef

How can we reproduce the issue?

async with self.driver.bidi_connection() as connection:
    self.cdp_bidi_main_session, self.cdp_bidi_main_devtools, self.cdp_cdp = connection.session, connection.devtools, connection.cdp
    await self.cdp_bidi_main_session.execute(self.cdp_bidi_main_devtools.page.enable())
    await self.__inject_javascript_object_via_bidi(self.cdp_bidi_main_session, self.cdp_bidi_main_devtools)

# later in the event hooked
if event.target_info.type_ == "page":
    async with self.driver.bidi_connection_with_target(event.target_info.target_id, self.target_attached_counter) as connection:
        cdp_bidi_main_session, cdp_bidi_main_devtools, cdp_cdp = connection.session, connection.devtools, connection.cdp
        await cdp_bidi_main_session.execute(cdp_bidi_main_devtools.page.enable())
        # below we confirm page is enabled for the session and working
        await cdp_bidi_main_session.execute(cdp_bidi_main_devtools.page.navigate(url="https://github.com/aslushnikov/getting-started-with-cdp/blob/master/README.md"))
        # below we inject javascript but it doesn't work
        await cdp_bidi_main_session.execute(cdp_bidi_main_devtools.page.add_script_to_evaluate_on_new_document(source="Object.defineProperty(navigator, 'deviceMemory', {get: () => 32})"))

Relevant log output

DEBUG:POST http://localhost:57009/session {'capabilities': {'firstMatch': [{}], 'alwaysMatch': {'browserName': 'chrome', 'pageLoadStrategy': <PageLoadStrategy.normal: 'normal'>, 'goog:chromeOptions': {'prefs': {'credentials_enable_service': False, 'profile.password_manager_enabled': False, 'webrtc.ip_handling_policy': 'disable_non_proxied_udp', 'webrtc.multiple_routes_enabled': False, 'webrtc.nonproxied_udp_enabled': False, 'intl.accept_languages': 'fr-FR', 'intl.timezone': 'Europe/Paris'}, 'excludeSwitches': ['enable-automation'], 'extensions': [], 'args': ['--proxy-server=socks5://127.0.0.1:35001', '--enable-logging=true', '--log-level=3', '--no-sandbox', '--ignore-certificate-errors', '--disable-popup-blocking', '--blink-settings=imagesEnabled=true', '--window-position=0,0', '--window-size=1920,1080', '--remote-debugging-port=9222', '--remote-allow-origins=http://localhost:9222']}}}}
DEBUG:Remote response: status=200 | data={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"123.0.6312.86","chrome":{"chromedriverVersion":"123.0.6312.86 (9b72c47a053648d405376c5cf07999ed626728da-refs/branch-heads/6312@{#698})","userDataDir":"/tmp/.org.chromium.Chromium.qt4ryg"},"fedcm:accounts":true,"goog:chromeOptions":{"debuggerAddress":"localhost:9222"},"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platformName":"linux","proxy":{},"setWindowRect":true,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"014682449876e57b90c8129f88bc3c23"}} | headers=HTTPHeaderDict({'Content-Length': '846', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Emulation.setGeolocationOverride', 'params': {'latitude': 48.8582, 'longitude': 2.3387, 'accuracy': 1000}}
DEBUG:Remote response: status=200 | data={"value":{}} | headers=HTTPHeaderDict({'Content-Length': '12', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Emulation.setLocaleOverride', 'params': {'locale': 'fr-FR'}}
DEBUG:Remote response: status=200 | data={"value":{}} | headers=HTTPHeaderDict({'Content-Length': '12', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Emulation.setTimezoneOverride', 'params': {'timezoneId': 'Europe/Paris'}}
DEBUG:Remote response: status=200 | data={"value":{}} | headers=HTTPHeaderDict({'Content-Length': '12', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Page.getFrameTree', 'params': {}}
DEBUG:Remote response: status=200 | data={"value":{"frameTree":{"frame":{"adFrameStatus":{"adFrameType":"none"},"crossOriginIsolatedContextType":"NotIsolated","domainAndRegistry":"","gatedAPIFeatures":[],"id":"6E6B318BC3D5B078D047A69CD864B480","loaderId":"76A0F3ADC1B18FB32210E4E8D4B208A8","mimeType":"text/plain","secureContextType":"InsecureScheme","securityOrigin":"://","url":"data:,"}}}} | headers=HTTPHeaderDict({'Content-Length': '351', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Page.getFrameTree', 'params': {}}
DEBUG:Remote response: status=200 | data={"value":{"frameTree":{"frame":{"adFrameStatus":{"adFrameType":"none"},"crossOriginIsolatedContextType":"NotIsolated","domainAndRegistry":"","gatedAPIFeatures":[],"id":"6E6B318BC3D5B078D047A69CD864B480","loaderId":"76A0F3ADC1B18FB32210E4E8D4B208A8","mimeType":"text/plain","secureContextType":"InsecureScheme","securityOrigin":"://","url":"data:,"}}}} | headers=HTTPHeaderDict({'Content-Length': '351', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/goog/cdp/execute {'cmd': 'Page.getFrameTree', 'params': {}}
DEBUG:Remote response: status=200 | data={"value":{"frameTree":{"frame":{"adFrameStatus":{"adFrameType":"none"},"crossOriginIsolatedContextType":"NotIsolated","domainAndRegistry":"","gatedAPIFeatures":[],"id":"6E6B318BC3D5B078D047A69CD864B480","loaderId":"76A0F3ADC1B18FB32210E4E8D4B208A8","mimeType":"text/plain","secureContextType":"InsecureScheme","securityOrigin":"://","url":"data:,"}}}} | headers=HTTPHeaderDict({'Content-Length': '351', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
6E6B318BC3D5B078D047A69CD864B480
{"method": "Target.getTargets", "params": {}, "id": 0}
{"method": "Target.attachToTarget", "params": {"targetId": "6E6B318BC3D5B078D047A69CD864B480", "flatten": true}, "id": 1}
{"method": "Page.enable", "id": 0, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
{"method": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "Object.defineProperty(navigator, 'deviceMemory', {get: () => 32})"}, "id": 1, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
{"method": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "Object.defineProperty(navigator, 'webdriver', {get: () => false})"}, "id": 2, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
{"method": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "Object.defineProperty(navigator, 'languages', {get: () => ['en-US', 'en','fr-FR', 'fr']})"}, "id": 3, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
{"method": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "Object.defineProperty(navigator, 'language', {get: () => ''fr-FR'})"}, "id": 4, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
{"method": "Target.setDiscoverTargets", "params": {"discover": true}, "id": 5, "sessionId": "51BF3CB3C42CCD88BA6BAC2B0BCDD098"}
mitm targets thread started
DEBUG:POST http://localhost:57009/session/014682449876e57b90c8129f88bc3c23/url {'url': 'https://whoer.net/'}
DEBUG:Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
DEBUG:Finished Request
TargetCreated(target_info=TargetInfo(target_id=TargetID('8D846E40848ECB116C7D191AE2B029B9'), type_='page', title='New Tab', url='chrome://newtab/', attached=False, can_access_opener=False, opener_id=None, opener_frame_id=None, browser_context_id=BrowserContextID('B4F4214DC121C02A627F52083DC0DC6B'), subtype=None))
{"method": "Target.attachToTarget", "params": {"targetId": "8D846E40848ECB116C7D191AE2B029B9", "flatten": true}, "id": 2}
{"method": "Page.enable", "id": 0, "sessionId": "84D92EBE2E097838058D8415854E2BC8"}
{"method": "Page.navigate", "params": {"url": "https://github.com/aslushnikov/getting-started-with-cdp/blob/master/README.md"}, "id": 1, "sessionId": "84D92EBE2E097838058D8415854E2BC8"}
{"method": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "Object.defineProperty(navigator, 'deviceMemory', {get: () => 32})"}, "id": 2, "sessionId": "84D92EBE2E097838058D8415854E2BC8"}
TargetCreated(target_info=TargetInfo(target_id=TargetID('08B0BF16857A287717FB3365F87E9AFD'), type_='iframe', title='', url='', attached=False, can_access_opener=False, opener_id=None, opener_frame_id=None, browser_context_id=BrowserContextID('B4F4214DC121C02A627F52083DC0DC6B'), subtype=None))

Operating System

Linux

Selenium version

4.19.0

What are the browser(s) and version(s) where you see this issue?

Chromedriver 123

What are the browser driver(s) and version(s) where you see this issue?

Chromedriver 123 - 122

Are you using Selenium Grid?

No response

github-actions[bot] commented 6 months ago

@HermesNWO, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

diemol commented 6 months ago

Was this working before, or are you trying this for the first time? I believe our BiDi and CDP support in Python need work.

@AutomatedTester do you know if this should be working?

HermesNWO commented 6 months ago

I have also tried with Selenium 4.16.0 and Chromedriver 122 and then updated to see if any chance of work but nothing... And this is the first time I implement BIDI for Target.* related, and it doesn't work...

HermesNWO commented 6 months ago

Okay I have found the answer.

Sended commands are destroyed once the attached session does.

So you have to keep a opened flattened session reference in a external Thread to keep your commands data.

https://stackoverflow.com/a/78249646/11173029

github-actions[bot] commented 5 months ago

This issue has been automatically locked since there has not been any recent activity since it was closed. Please open a new issue for related bugs.