Login Error: waiting for locator("select#landingPageOptions")

[NelsonDane]

The past few days this issue has been popping up for me:

Error logging in to Schwab: Frame.select_option: Timeout 30000ms exceeded.
Call log:
waiting for locator("select#landingPageOptions")

Traceback (most recent call last):
  File "/Users/ndane/GitHub/auto-rsa/schwabAPI.py", line 35, in schwab_init
    schwab.login(
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/schwab_api/authentication.py", line 103, in login
    return asyncio.run(self._async_login())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndane/.pyenv/versions/3.12.2/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/ndane/.pyenv/versions/3.12.2/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndane/.pyenv/versions/3.12.2/lib/python3.12/asyncio/base_events.py", line 685, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/schwab_api/authentication.py", line 145, in _async_login
    await self.page.frame(name=login_frame).select_option("select#landingPageOptions", index=3)
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/playwright/async_api/_generated.py", line 5335, in select_option
    await self._impl_obj.select_option(
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/playwright/_impl/_frame.py", line 679, in select_option
    return await self._channel.send("selectOption", params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 59, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ndane/GitHub/auto-rsa/.venv/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call
    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.TimeoutError: Frame.select_option: Timeout 30000ms exceeded.
Call log:
waiting for locator("select#landingPageOptions")

This occurs whether saving the session cache or not.

[cnMuggle]

I saw the same time out error around last week. My schwab-api code is old around 2/June/2024, my python version is also older. It's good to know that 3.12 also has this issue.

I just try re-login later and it will work sometimes, but definitely annoying (This just happened today), I haven't figure out any root cause yet. It feels like schwab site issue? Or will this be un-stable network connection, does anyone on wired network having this problem or not? It should not be pc performance issue? I have 16GB ram and AMD Ryzen 7 7840HS. I previously saw this timeout on a 4GB ram Linux machine sometimes, so I thought it's RAM limited at that time. In the closed attempt (https://github.com/itsjafer/schwab-api/pull/41), I simply try increase the timeout but there will be another time out jump out elsewhere. Sorry, I didn't spend enough time to look into this issue on Jan/2024. it's not pain enough for me.

Appreciate any feedback or possible solution as well

File "C:\git/schwab-api_2June2024\schwab_api\authentication.py", line 82, in _async_login await self.page.frame(name=login_frame).select_option("select#landingPageOptions", index=3) File "C:\virtualEnv_Windows\lib\site-packages\playwright\async_api_generated.py", line 5656, in select_option await self._impl_obj.select_option( File "C:\virtualEnv_Windows\lib\site-packages\playwright_impl_frame.py", line 675, in select_option return await self._channel.send("selectOption", params) File "C:\virtualEnv_Windows\lib\site-packages\playwright_impl_connection.py", line 62, in send return await self._connection.wrap_api_call( File "C:\virtualEnv_Windows\lib\site-packages\playwright_impl_connection.py", line 492, in wrap_api_call return await cb() File "C:\virtualEnv_Windows\lib\site-packages\playwright_impl_connection.py", line 100, in inner_send result = next(iter(done)).result() playwright._impl._errors.TimeoutError: Timeout 30000ms exceeded. Exception ignored in: <function BaseSubprocessTransport.del at 0x000001F7D3A57D30> Traceback (most recent call last): File "C:\Programs\Python\Python39\Lib\asyncio\base_subprocess.py", line 126, in del File "C:\Programs\Python\Python39\Lib\asyncio\base_subprocess.py", line 104, in close File "C:\Programs\Python\Python39\Lib\asyncio\proactor_events.py", line 108, in close File "C:\Programs\Python\Python39\Lib\asyncio\base_events.py", line 751, in call_soon File "C:\Programs\Python\Python39\Lib\asyncio\base_events.py", line 515, in _check_closed RuntimeError: Event loop is closed Exception ignored in: <function _ProactorBasePipeTransport.del at 0x000001F7D3A795E0> Traceback (most recent call last): File "C:\Programs\Python\Python39\Lib\asyncio\proactor_events.py", line 116, in del File "C:\Programs\Python\Python39\Lib\asyncio\proactor_events.py", line 108, in close File "C:\Programs\Python\Python39\Lib\asyncio\base_events.py", line 751, in call_soon File "C:\Programs\Python\Python39\Lib\asyncio\base_events.py", line 515, in _check_closed

[davidacrook]

This also has been happening to me the last few days. 90% fails. Same error: "waiting for locator("select#landingPageOptions")" I looked here to see if anyone else had the same problem. Schwab must have made a change.

[NelsonDane]

I did a little debugging myself and here's what I found:

The script uses this to login on the home page (schwab.com) But when opened using Playwright it doesn't load in: This leads to the timeout error.

Here's some things I noticed while troubleshooting:

1. In my normal browser (Brave), schwab.com redirects to schwab.com/client-home, but it doesn't do this in Playwright (not sure if this is important or not). Going directly to /client-home didn't change anything.
2. Commenting out await stealth_async(self.page) in authentication.py causes the above iframe to show up, but login fails: (Which makes sense since it's probably blocking the automation)
3. Everything loads and shows up correctly accessing this login link https://client.schwab.com/Areas/Access/Login, but of course some selectors would need to be changed for a successful login. Maybe we want to switch to this page instead?

Hopefully this is helpful and a fix is found soon :)

[NelsonDane]

I found a fix that worked for me and created a PR. Would love to know if this also works for you @cnMuggle @davidacrook

[davidacrook]

Thanks Nelson! I get farther now, but then get a 30 second timeout error on this line:
await self.page.wait_for_url(re.compile(r"app/trade"), wait_until="domcontentloaded") # Making it more robust than specifying an exact url which may change.

[davidacrook]

Testing Nelson's change on Windows and WSL, 5 tries each: Windows (Python 3.12) - 0 successes, 5 failures. Linux (Python 3.10) - 2 successes, 3 failures. The successes take 17 seconds and it used to take 10 seconds, so something changed on Schwab's side. @NelsonDane, what is your operating environment that this works successfully in?

[NelsonDane]

@davidacrook Thanks for the reply and for testing. I've tested on: (both headless and not)

1. MacOS 14.5 Apple Silicon (Python 3.12.2) - Success
2. Linux (Docker - Python 3.12-slim image) Intel Server - Success

However, when I tested it this morning on a Windows 11 VM (Python 3.12.2) I found that it failed to the same timeout error as previously. The iframe also didn't load and "strobed" like in the image I sent above (when ran non-headless).

Also this morning, I found that commenting the await stealth_async(self.page) caused it to work on both MacOS and Windows without giving the "call customer support" error as seen above, which is very strange. So there is something that schwab.com does not like being blocked by that function on Windows. I can do more testing today and mess with the stealth config more in my Windows VM and see if I can get it any further.

[NelsonDane]

Doing more digging, the playwright-stealth module seems to be problematic https://github.com/AtuboDad/playwright_stealth/issues, with some users saying newer versions of playwright broke it. This issue gave some options to disable, but none of them would work in my Windows VM (but did on MacOS/Linux).

Perhaps this module is doing more harm than good? It's at least something to look into more.

[itsjafer]

Don't have a ton of bandwidth to test myself, but can we try freezing to playwright 1.4.4 (before the breaking changes) or even 1.27.1 and see if that resolves the issue for now?

[NelsonDane]

Checking today, it seems that the iframe is no longer blocked, and it's working without any changes needed for me now. Strange, perhaps Schwab updated/fixed something on their side

EDIT 7/10/2024: Still being blocked, but working with https://github.com/itsjafer/schwab-api/pull/64

[cnMuggle]

I am able to login with @NelsonDane's change in both a Linux and Windows machine consistently. Just commenting out "await stealth_async(self.page, config)" doesn't work for me. I do see under Linux that without commenting "await stealth_async(self.page, config)", the login box is greyed out. Windows machine (not WSL) playwright version is 1.40.0 with python 3.9.13, Linux machine has playwright 1.45.0 with python 3.12 using suggested method or commenting out "await stealth_async(self.page, config)" shows below in screenshot

minor description, It's strange that my Windows machine can login sometimes without the change https://github.com/itsjafer/schwab-api/pull/64 (cannot login last Friday, but works Monday today, while the same code cannot work on a Linux machine until I applied the NelsonDane's change).

[NelsonDane]

I'm getting the same thing as @cnMuggle, currently getting the first image, but then using https://github.com/itsjafer/schwab-api/pull/64 allows it to login properly.

[itsjafer]

release 0.4.3 contains the fix. Thanks @NelsonDane for the PR!