search

Voyz / ibeam

IBeam is an authentication and maintenance tool used for the Interactive Brokers Client Portal Web API Gateway.
Apache License 2.0
573 stars 113 forks source link

Google Messages 2FA not receiving QR Code URL #193

Closed kaykhan closed 4 months ago

kaykhan commented 5 months ago

Hi friends,

I am attempting to handle 2FA via the google messages handler - https://github.com/Voyz/ibeam/wiki/Two-Factor-Authentication#google-messages-handler

I have a docker-compose file and env.list setup.

version: "2.1"
services:
  ibeam:
    image: voyz/ibeam
    container_name: ibeam
    env_file:
      - env.list
    ports:
      - 5000:5000
      - 5001:5001
    network_mode: bridge # Required due to clientportal.gw IP whitelist
    restart: 'no' # Prevents IBEAM_MAX_FAILED_AUTH from being exceeded
IBEAM_ACCOUNT=REDACTED
IBEAM_PASSWORD=REDACTED
IBEAM_TWO_FA_HANDLER=GOOGLE_MSG
IBEAM_TWO_FA_SELECT_TARGET=One Time Passcode

I run docker compose up

[+] Running 1/1 ✔ Container ibeam Recreated 0.1s Attaching to ibeam ibeam | 2024-06-18 13:55:33,379|I| ############ Starting IBeam version 0.5.1 ############ ibeam | 2024-06-18 13:55:33,380|I| Secrets source: env ibeam | 2024-06-18 13:55:33,381|I| Health server started at port=5001 ibeam | 2024-06-18 13:55:33,381|I| Configuration: ibeam | {'UNDEFINED': <object object at 0x7d5792e602d0>, 'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 2, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'HEALTH_SERVER_PORT': 5001, 'SECRETS_SOURCE': 'env', 'GCP_SECRETS_URL': None, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'NAME@@password', 'SUBMIT_EL': 'CSS_SELECTOR@@.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'TAG_NAME@@Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'CLASS_NAME@@ibkey-promo-skip', 'AUTHENTICATION_STRATEGY': 'B', 'MAX_STATUS_CHECK_RETRIES': 15, 'MAX_REAUTHENTICATE_RETRIES': 3, 'UI_SCALING': 1.0, 'TWO_FA_EL_ID': 'ID@@twofactbase', 'TWO_FA_NOTIFICATION_EL': 'CLASS_NAME@@login-step-notification', 'TWO_FA_INPUT_EL_ID': 'ID@@chlginput', 'TWO_FA_HANDLER': 'GOOGLE_MSG', 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'ID@@sf_select', 'TWO_FA_SELECT_TARGET': 'One Time Passcode', 'CUSTOM_TWO_FA_HANDLER': 'custom_two_fa_handler.CustomTwoFaHandler'} ibeam | 2024-06-18 13:55:33,381|I| Gateway not found, starting new one... ibeam | 2024-06-18 13:55:33,381|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command. ibeam | 2024-06-18 13:55:33,381|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml'] ibeam | running ibeam | runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* ibeam | config file : root/conf.yaml ibeam | 2024-06-18 13:55:33,386|I| Gateway started with pids: [13] ibeam | 2024-06-18 13:55:33,387|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused> ibeam | 2024-06-18 13:55:33,387|I| Gateway connection established ibeam | 2024-06-18 13:55:33,388|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused> ibeam | 2024-06-18 13:55:33,388|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) ibeam | 2024-06-18 13:55:33,388|I| Authentication strategy: "B" ibeam | 2024-06-18 13:55:33,388|I| No active sessions, logging in... ibeam | 2024-06-18 13:55:33,388|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | WARNING: An illegal reflective access operation has occurred ibeam | WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int) ibeam | WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil ibeam | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations ibeam | WARNING: All illegal access operations will be denied in a future release ibeam | -> mount demo on /demo ibeam | Java Version: 11.0.20 ibeam | **** ibeam | version: a27ed42161ad96c53e715ca5c5e3e3fa4cff5262 Mon, 24 Apr 2023 15:41:53 -0400 ibeam | **** ibeam | This is the Client Portal Gateway ibeam | for any issues, please contact api@ibkr.com ibeam | and include a copy of your logs ibeam | **** ibeam | https://www.interactivebrokers.com/api/doc.html ibeam | **** ibeam | Open https://localhost:5000 to login ibeam | App demo is available after you login under: https://localhost:5000/demo#/ ibeam | 2024-06-18 13:55:40,777|I| Gateway auth webpage loaded ibeam | 2024-06-18 13:55:40,777|I| Login attempt number 1 ibeam | 2024-06-18 13:55:46,082|I| Submitting the form ibeam | 2024-06-18 13:56:01,698|E| Timeout reached when waiting for authentication. The website seems to not be loaded correctly. Consider increasing IBEAM_PAGE_LOAD_TIMEOUT. ibeam | Website URL: https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | ibeam | Exception: ibeam | File "/srv/ibeam/ibeam_starter.py", line 167, in ibeam | success, shutdown, status = client.start_and_authenticate() ibeam | File "/srv/ibeam/src/gateway_client.py", line 53, in start_and_authenticate ibeam | success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating ibeam | return self._authentication_strategy_B(status, request_retries) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B ibeam | return self._log_in(status) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in ibeam | success, shutdown = self.login_handler.login() ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 454, in login ibeam | self.handle_timeout_exception(e, targets, driver, website_version, self.route_auth, self.base_url, self.outputs_dir) ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 435, in login ibeam | self.attempt(targets, wait_and_identify_trigger, driver) ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 359, in attempt ibeam | trigger, target = self.step_login(targets, wait_and_identify_trigger, driver, self.secrets_handler.account, self.secrets_handler.password, self.secrets_handler.key, self.presubmit_buffer) ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 180, in step_login ibeam | trigger, target = wait_and_identify_trigger( ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 67, in _wait_and_identify_trigger ibeam | trigger = WebDriverWait(driver, timeout).until(any_of(expected_conditions)) ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until ibeam | raise TimeoutException(message, screen, stacktrace) ibeam | ibeam | <class 'selenium.common.exceptions.TimeoutException'> Message: ibeam | ibeam | 2024-06-18 13:56:01,698|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7d5790c55010> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="206636e1dec0444a415a09daac53c10d")> ibeam | 2024-06-18 13:56:01,732|I| Logging in failed ibeam | 2024-06-18 13:56:01,734|I| Starting maintenance with interval 60 seconds ibeam | 2024-06-18 13:57:01,735|I| Maintenance ibeam | 2024-06-18 13:57:01,810|I| Attempt number 2 ibeam | 2024-06-18 13:57:02,094|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable ibeam | 2024-06-18 13:57:02,094|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) ibeam | 2024-06-18 13:57:02,094|I| Authentication strategy: "B" ibeam | 2024-06-18 13:57:02,094|I| No active sessions, logging in... ibeam | 2024-06-18 13:57:02,094|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | 2024-06-18 13:57:08,781|I| Gateway auth webpage loaded ibeam | 2024-06-18 13:57:08,782|I| Login attempt number 1 ibeam | 2024-06-18 13:57:14,090|I| Submitting the form ibeam | 2024-06-18 13:57:29,640|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7d5790bf0250> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="15b1a403ee88b0a9f27b278e66534add")> ibeam | 2024-06-18 13:57:29,640|E| Timeout reached when waiting for authentication. The website seems to not be loaded correctly. Consider increasing IBEAM_PAGE_LOAD_TIMEOUT. ibeam | Website URL: https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | ibeam | Exception: ibeam | File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap ibeam | self._bootstrap_inner() ibeam | File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner ibeam | self.run() ibeam | File "/usr/local/lib/python3.11/threading.py", line 975, in run ibeam | self._target(self._args, self._kwargs) ibeam | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker ibeam | work_item.run() ibeam | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run ibeam | result = self.fn(*self.args, *self.kwargs) ibeam | File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job ibeam | retval = job.func(job.args, job.kwargs) ibeam | File "/srv/ibeam/src/gateway_client.py", line 84, in _maintenance ibeam | success, shutdown, status = self.start_and_authenticate(request_retries=self.request_retries) ibeam | File "/srv/ibeam/src/gateway_client.py", line 53, in start_and_authenticate ibeam | success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating ibeam | return self._authentication_strategy_B(status, request_retries) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B ibeam | return self._log_in(status) ibeam | File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in ibeam | success, shutdown = self.login_handler.login() ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 454, in login ibeam | self.handle_timeout_exception(e, targets, driver, website_version, self.route_auth, self.base_url, self.outputs_dir) ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 435, in login ibeam | self.attempt(targets, wait_and_identify_trigger, driver) ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 359, in attempt ibeam | trigger, target = self.step_login(targets, wait_and_identify_trigger, driver, self.secrets_handler.account, self.secrets_handler.password, self.secrets_handler.key, self.presubmit_buffer) ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 180, in step_login ibeam | trigger, target = wait_and_identify_trigger( ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/srv/ibeam/src/handlers/login_handler.py", line 67, in _wait_and_identify_trigger ibeam | trigger = WebDriverWait(driver, timeout).until(any_of(*expected_conditions)) ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until ibeam | raise TimeoutException(message, screen, stacktrace) ibeam | ibeam | <class 'selenium.common.exceptions.TimeoutException'> Message: ibeam | ibeam | 2024-06-18 13:57:29,672|I| Logging in failed ^CGracefully stopping... (press Ctrl+C again to force) [+] Stopping 1/1 ✔ Container ibeam Stopped 10.2s canceled

According to the wiki i am expecting to see the qr code url but i never get it. What is happening here?

kaykhan commented 5 months ago

Following on from this, i... for some reason decided to setup the IB Key Mobile Authenticator.

Once you set that up you cannot revert back to receiving SMS codes.

What are my options now... :facepalm

Voyz commented 5 months ago

@kaykhan glad you've figured it out, yeah, once you go to IB Key you cannot go back 😢

Did you consider OAuth? Are you an institutional account? https://ibkrcampus.com/ibkr-api-page/cpapi-v1/#oauth-introduction

kaykhan commented 5 months ago

@kaykhan glad you've figured it out, yeah, once you go to IB Key you cannot go back 😢

Did you consider OAuth? Are you an institutional account? https://ibkrcampus.com/ibkr-api-page/cpapi-v1/#oauth-introduction

Ive not considered OAuth, im on an individual account. (just to confirm oauth is only possible on institutional accounts?)

So im guessing i need to create a brand new IBKR account if i want to continue on this journey? (creating a secondary user on the same account won't work? - how do they authenticate...)

kaykhan commented 5 months ago

And it looks like my original issue would have been resolved - https://github.com/Voyz/ibeam/issues/188 by using the 0.5.4-rc2 image. Damn why did i enable IB Key :(

Voyz commented 4 months ago

That's indeed misfortunate @kaykhan :( Once enabled, you're stuck with it it seems. Opening a new account could potentially be a fix - however IBKR indicate they want to phase out SMS 2FA at some point.

OAuth indeed is only for institutional accounts for now - although IBKR say they want to make it available for all.

kaykhan commented 4 months ago

That's indeed misfortunate @kaykhan :( Once enabled, you're stuck with it it seems. Opening a new account could potentially be a fix - however IBKR indicate they want to phase out SMS 2FA at some point.

OAuth indeed is only for institutional accounts for now - although IBKR say they want to make it available for all.

thanks for confirming that. I will close this issue.