[Nodriver] Repeated `KeyError` from `cdp.util.parse_json_event` on `DOM.scrollableFlagUpdated`

[michaellee94]

I'm logging these exceptions from cdp.util.parse_json_event when I visit certain websites.

2024-11-10 23:51:43,649 INFO uc.connection KeyError: ('DOM.scrollableFlagUpdated',)  during parsing of json from event : {'method': 'DOM.scrollableFlagUpdated', 'params': {'nodeId': 1044, 'isScrollable': True}}
Traceback (most recent call last):
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/core/connection.py", line 649, in listener_loop
    event = cdp.util.parse_json_event(message)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/cdp/util.py", line 19, in parse_json_event
    return _event_parsers[json["method"]].from_json(json["params"])
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'DOM.scrollableFlagUpdated'
2024-11-10 23:51:43,937 INFO uc.connection KeyError: ('DOM.scrollableFlagUpdated',)  during parsing of json from event : {'method': 'DOM.scrollableFlagUpdated', 'params': {'nodeId': 1617, 'isScrollable': False}}
Traceback (most recent call last):
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/core/connection.py", line 649, in listener_loop
    event = cdp.util.parse_json_event(message)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/cdp/util.py", line 19, in parse_json_event
    return _event_parsers[json["method"]].from_json(json["params"])
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'DOM.scrollableFlagUpdated'
2024-11-10 23:51:45,216 INFO uc.connection KeyError: ('DOM.scrollableFlagUpdated',)  during parsing of json from event : {'method': 'DOM.scrollableFlagUpdated', 'params': {'nodeId': 1917, 'isScrollable': False}}
Traceback (most recent call last):
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/core/connection.py", line 649, in listener_loop
    event = cdp.util.parse_json_event(message)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/.local/lib/python3.12/site-packages/nodriver/cdp/util.py", line 19, in parse_json_event
    return _event_parsers[json["method"]].from_json(json["params"])
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^

Will attach a minimal example

[michaellee94]

This is a working example (minus the credentials):

import asyncio
import logging
import os
import sys

import nodriver as uc

ACCOUNT_ID = "foo"
PASSWORD = "bar"

USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"

async def login(driver: uc.Browser):
    tab = await driver.get("https://betanysports.eu/login", new_tab=True)

    username_field = None
    while not username_field:
        username_field = await tab.query_selector("[id='usernameDesk']")
        password_field = await tab.query_selector("[id='passwordDesk']")
        login_button = await tab.find("LOGIN")
        await asyncio.sleep(0.1)

    await username_field.send_keys(ACCOUNT_ID)
    await asyncio.sleep(0.1)
    await password_field.send_keys(PASSWORD)
    await asyncio.sleep(0.1)
    await login_button.click()

    return tab

async def test_fn(driver: uc.Browser):
    tab = await login(driver)

    fighting_tab = None
    while not fighting_tab:
        await asyncio.sleep(0.1)
        fighting_tab = await tab.query_selector("[id='img_Fighting']")
    await fighting_tab.click()
    await asyncio.sleep(0.5)
    ufc = await tab.query_selector("label[for='gl_Fighting_UFC_G']")
    await ufc.click()

    while True:
        await asyncio.sleep(1)

def handle_exception(exc_type, exc_value, exc_traceback):
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return

    logger.error(
        "Uncaught exception",
        exc_info=(exc_type, exc_value, exc_traceback),
    )

async def main():
    driver = await uc.start(
        browser_args=[
            "--headless=new",
            f"--user-agent={USER_AGENT}",
            "--window-size=1920x1080",
        ]
    )
    await test_fn(driver)
    driver.stop()

if __name__ == "__main__":
    logging.basicConfig(
        filename="nodriver_bug.log",
        level=logging.INFO,
        format="%(asctime)s %(levelname)s %(name)s %(message)s",
    )
    logger = logging.getLogger(__name__)
    sys.excepthook = handle_exception

    uc.loop().run_until_complete(main())

[michaellee94]

Here is a fix:

https://github.com/michaellee94/nodriver/commit/602a3eda5e9ce1070fd98b3a27db7160a90ca7ef

I can't open a PR on https://github.com/ultrafunkamsterdam/nodriver, unfortunately. If a contributor to the repo would be willing to take a look, I would appreciate it!

[therealpurplemana]

Here is a fix:

michaellee94/nodriver@602a3ed

I can't open a PR on https://github.com/ultrafunkamsterdam/nodriver, unfortunately. If a contributor to the repo would be willing to take a look, I would appreciate it!

Could you please PR this into here? https://github.com/stephanlensky/zendriver. This is a friendlier to PR fork including a number of fixes.

[michaellee94]

@therealpurplemana Can do!

[therealpurplemana]

@therealpurplemana Can do!

I think it's already fixed. I tried to PR it myself but didn't see the DomScrollable code.

[michaellee94]

@therealpurplemana Can do!

I think it's already fixed. I tried to PR it myself but didn't see the DomScrollable code.

@therealpurplemana It's not yet implemented in zendriver, so I opened a PR here.

[mdmintz]

Here is a fix:

michaellee94/nodriver@602a3ed

I can't open a PR on https://github.com/ultrafunkamsterdam/nodriver, unfortunately. If a contributor to the repo would be willing to take a look, I would appreciate it!

Thank you. I added that to MyCDP, which is used by my nodriver fork ("CDP Mode") in https://github.com/seleniumbase/SeleniumBase.