search

Voyz / ibeam

IBeam is an authentication and maintenance tool used for the Interactive Brokers Client Portal Web API Gateway.
Apache License 2.0
582 stars 115 forks source link

Authentication runs in a loop. 'Gateway session active but not authenticated' #152

Open dabrazhe opened 1 year ago

dabrazhe commented 1 year ago

Describe the bug The service keeps authenticating over again See logs

2023-07-11 13:23:02,236|I| ############ Starting IBeam version 0.4.6 ############
2023-07-11 13:23:02,264|I| Secrets source: env
2023-07-11 13:23:02,276|I| Health server started at port=5001
2023-07-11 13:23:02,276|I| Environment variable configuration:
{'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'IBEAM_HEALTH_SERVER_PORT': 5001, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'password', 'SUBMIT_EL': 'button.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 2, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'ibkey-promo-skip', 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_NOTIFICATION_EL': 'login-step-notification', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key'}
2023-07-11 13:23:02,277|I| Gateway not found, starting new one...
2023-07-11 13:23:02,277|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command.
2023-07-11 13:23:02,277|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running  
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 config file  : root/conf.yaml
2023-07-11 13:23:03,310|I| Gateway started with pid: 14
2023-07-11 13:23:03,349|I| Cannot ping Gateway. Retrying for another 20 seconds
2023-07-11 13:23:04,351|I| Cannot ping Gateway. Retrying for another 19 seconds
2023-07-11 13:23:05,353|I| Cannot ping Gateway. Retrying for another 18 seconds
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2023-07-11 13:23:06,355|I| Cannot ping Gateway. Retrying for another 17 seconds
 -> mount demo on /demo
Java Version: 11.0.18
****************************************************
version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400
****************************************************
This is the Client Portal Gateway
for any issues, please contact api@ibkr.com
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo#/
2023-07-11 13:23:07,357|I| Cannot ping Gateway. Retrying for another 16 seconds
2023-07-11 13:23:09,883|I| Gateway connection established
2023-07-11 13:23:10,327|I| No active sessions, logging in...
2023-07-11 13:23:10,328|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-11 13:23:24,075|I| Gateway auth webpage loaded
2023-07-11 13:23:24,076|I| Login attempt number 1
2023-07-11 13:23:29,382|I| Submitting the form
2023-07-11 13:23:30,518|I| Webpage displayed "Client login succeeds"
2023-07-11 13:23:32,518|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7ff51f7c5710> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="2d8b3814bcca3a5462cc61b02ad8b650")>
2023-07-11 13:23:32,852|I| Authentication process succeeded
2023-07-11 13:23:35,961|E| Gateway session active but not authenticated
2023-07-11 13:23:35,982|I| Waiting 15 seconds to reauthenticate before restarting.
2023-07-11 13:23:50,983|I| Logging out and restarting the Gateway
2023-07-11 13:23:51,087|I| Gateway logout successful
2023-07-11 13:23:51,150|I| Gateway session found but not authenticated, authenticating...
2023-07-11 13:23:51,150|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-11 13:23:58,688|I| Gateway auth webpage loaded
2023-07-11 13:23:58,688|I| Login attempt number 1
2023-07-11 13:24:03,995|I| Submitting the form
2023-07-11 13:24:05,098|I| Webpage displayed "Client login succeeds"
2023-07-11 13:24:07,099|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7ff51f7e9d50> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="7c6cc168602cedd44c23b3d5ba6da3fb")>
2023-07-11 13:24:07,129|I| Authentication process succeeded
2023-07-11 13:24:10,190|E| Gateway session active but not authenticated
2023-07-11 13:24:10,206|I| Waiting 15 seconds to reauthenticate before restarting.

Environment IBeam version: the latest Docker image or standalone: Docker

Additional context The login info is correct, can login on the IB web site.

vlpraveen commented 1 year ago

I suggest try the new version https://github.com/Voyz/ibeam/issues/147. For many this has solved the authentication loop. Please do share your results.

automiamo commented 1 year ago

voyz/ibeam:0.5.0-rc2 is running now on my real account and it works istead of voyz/ibeam

daneltiebreak commented 1 year ago

works with --env IBEAM_AUTHENTICATION_STRATEGY=B

Voyz commented 1 year ago

Hey everyone 👋 As much as I'd like to suggest a fix, this authentication loop doesn't seem to be something that is in our control. Ever since I remember IBKR servers will just sporadically refuse to allow you to authenticate successfully over a large period of time despite successfully completing the authentication flow, and cause IBeam to get into this loop.

Remember that what happens is:

  1. IBeam notices we don't have an authenticated session, ie. the /tickle response reports authenticated:false. There seem to be a number of reasons for this, such as: fresh startup, conflicting session, IBKR servers restart, etc.
  2. We attempt to reauthenticate. In Strategy A it means a full re-login, in Strategy B it means attempting to reauthenticate for a number of times before killing the Gateway and reattempting the login. (see #147 for more info on strategies)
  3. We do this until we succeed, indicated by the login website displaying Client login succeeds. The interpretation we go with is that IBKR server has accepted our authentication flow and our session should be authenticated, yet:
  4. /tickle response still reports authenticated:false

image

And so we're back where we started and we end up with the famous authentication loop.

What we've observed so far is:

  1. There isn't any obvious fix for this. Different users suggested a variety of methods, involving calling various endpoints or adding sleep periods in order to break the loop. There's been reports of these working for some users, unfortunately, none is a definitive winner. I implement most of the ideas that are suggested in order to try them out, however there hasn't been a significant breakthrough yet.
  2. IBKR doesn't seem to want to acknowledge that it is happening and provide support to solve this. We've been letting them know for a couple of years already. We shouldn't assume malice or negligence on their side, however also we shouldn't exclude the possibility that they dislike the fact that some users automate the authentication which was intended to be executed manually, and hence refuse to address this issue. The best I can do is to ask every user to submit tickets describing this issue.
  3. There is no clear pattern on when and why this happens. Sometimes it works, sometimes it breaks for hours. There's a good amount of reports that start along the lines of 'It's been working fine for months and then one day...'.
  4. Changing your server's location (eg. using VPN) or manually wrangling conf.yaml file to call a different server may help.

Honestly, this, along with other IBKR-side issues (eg. 'Invalid username and password' error) is what keeps me from upgrading IBeam to version 1.0.0, despite it being used in production by various users. I think everyone here should be aware that until the service is reliable, we're using this system acknowledging that there are and will be long and unexpected drops in connectivity to the broker. There's a good chance that for you it will work for months on end without interruption. There's a good chance that for you it will break every other day.

Best we can do is to:

  1. Speak to IBKR about this as much as possible and hope one day they'll help. Every support ticket counts, so let them know if you haven't, but especially if you're a whale, or an institution, or have some connections: please try pulling some ropes to get more attention to this.
  2. Try out various methods of getting over this issue and hope one of them will actually solve it. Feel free to experiment - add timeouts, call different endpoints, wrangle servers, change your IP, etc. - and report back here.

I've turned this response into an entry in the Troubleshooting Wiki.

dabrazhe commented 1 year ago

I tried again with the 0.5.0-rc3 version. The authentication is still in the loop. Does this work at all for other people and how can I work around it ?


2023-07-16 19:42:12 2023-07-16 17:42:12,382|I| ############ Starting IBeam version 0.5.0-rc3 ############
2023-07-16 19:42:12 2023-07-16 17:42:12,387|I| Secrets source: env
2023-07-16 19:42:12 2023-07-16 17:42:12,391|I| Health server started at port=5001
2023-07-16 19:42:12 2023-07-16 17:42:12,391|I| Configuration:
2023-07-16 19:42:12 {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 2, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'HEALTH_SERVER_PORT': 5001, 'SECRETS_SOURCE': 'env', 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'NAME@@password', 'SUBMIT_EL': 'CSS_SELECTOR@@.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'TAG_NAME@@Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'CLASS_NAME@@ibkey-promo-skip', 'AUTHENTICATION_STRATEGY': 'A', 'MAX_STATUS_CHECK_RETRIES': 15, 'MAX_REAUTHENTICATE_RETRIES': 3, 'UI_SCALING': 1.0, 'TWO_FA_EL_ID': 'ID@@twofactbase', 'TWO_FA_NOTIFICATION_EL': 'CLASS_NAME@@login-step-notification', 'TWO_FA_INPUT_EL_ID': 'ID@@chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'ID@@sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key', 'CUSTOM_TWO_FA_HANDLER': 'custom_two_fa_handler.CustomTwoFaHandler'}
2023-07-16 19:42:12 2023-07-16 17:42:12,392|I| Gateway not found, starting new one...
2023-07-16 19:42:12 2023-07-16 17:42:12,392|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command.
2023-07-16 19:42:12 2023-07-16 17:42:12,392|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
2023-07-16 19:42:12 running  
2023-07-16 19:42:12  runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
2023-07-16 19:42:12  config file  : root/conf.yaml
2023-07-16 19:42:12 2023-07-16 17:42:12,411|I| Gateway started with pids: [13]
2023-07-16 19:42:12 2023-07-16 17:42:12,412|I| Cannot ping Gateway. Retrying for another 20 seconds
2023-07-16 19:42:13 WARNING: An illegal reflective access operation has occurred
2023-07-16 19:42:13 WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
2023-07-16 19:42:13 WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
2023-07-16 19:42:13 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2023-07-16 19:42:13 WARNING: All illegal access operations will be denied in a future release
2023-07-16 19:42:13 2023-07-16 17:42:13,414|I| Cannot ping Gateway. Retrying for another 19 seconds
2023-07-16 19:42:13  -> mount demo on /demo
2023-07-16 19:42:13 Java Version: 11.0.18
2023-07-16 19:42:13 ****************************************************
2023-07-16 19:42:13 version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400
2023-07-16 19:42:13 ****************************************************
2023-07-16 19:42:13 This is the Client Portal Gateway
2023-07-16 19:42:13 for any issues, please contact api@ibkr.com
2023-07-16 19:42:13 and include a copy of your logs
2023-07-16 19:42:13 ****************************************************
2023-07-16 19:42:13 https://www.interactivebrokers.com/api/doc.html
2023-07-16 19:42:13 ****************************************************
2023-07-16 19:42:13 Open https://localhost:5000 to login
2023-07-16 19:42:13 App demo is available after you login under: https://localhost:5000/demo#/
2023-07-16 19:42:15 2023-07-16 17:42:15,210|I| Gateway connection established
2023-07-16 19:42:15 2023-07-16 17:42:15,330|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2023-07-16 19:42:15 2023-07-16 17:42:15,331|I| Authentication strategy: "A"
2023-07-16 19:42:15 2023-07-16 17:42:15,331|I| No active sessions, logging in...
2023-07-16 19:42:15 2023-07-16 17:42:15,331|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-16 19:42:23 2023-07-16 17:42:23,304|I| Gateway auth webpage loaded
2023-07-16 19:42:23 2023-07-16 17:42:23,304|I| Login attempt number 1
2023-07-16 19:42:28 2023-07-16 17:42:28,487|I| Submitting the form
2023-07-16 19:42:29 2023-07-16 17:42:29,472|I| Webpage displayed "Client login succeeds"
2023-07-16 19:42:30 2023-07-16 17:42:30,474|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffffbbc20810> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="c3b3974673ee26d4c7723692e7acdbf8")>
2023-07-16 19:42:30 2023-07-16 17:42:30,554|I| Logging in succeeded
2023-07-16 19:42:33 2023-07-16 17:42:33,781|E| Logging in succeeded, but active session is still not authenticated
2023-07-16 19:42:33 2023-07-16 17:42:33,825|I| Waiting 15 seconds to reauthenticate before restarting.
2023-07-16 19:42:48 2023-07-16 17:42:48,828|I| Logging out and reattempting full authentication
2023-07-16 19:42:49 2023-07-16 17:42:49,006|I| Gateway logout successful
2023-07-16 19:42:49 2023-07-16 17:42:49,115|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='a90c36ac2c420a87edd53a5022700358', server_name=None, server_version=None, expires=600000)
2023-07-16 19:42:49 2023-07-16 17:42:49,116|I| Authentication strategy: "A"
2023-07-16 19:42:49 2023-07-16 17:42:49,116|I| Competing Gateway session found, restarting and logging in...
2023-07-16 19:42:49 2023-07-16 17:42:49,180|I| Gateway logout successful
2023-07-16 19:42:49 2023-07-16 17:42:49,180|I| Gateway session found but not authenticated, logging in...
2023-07-16 19:42:49 2023-07-16 17:42:49,180|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-16 19:42:56 2023-07-16 17:42:56,416|I| Gateway auth webpage loaded
2023-07-16 19:42:56 2023-07-16 17:42:56,417|I| Login attempt number 1
2023-07-16 19:43:01 2023-07-16 17:43:01,566|I| Submitting the form
2023-07-16 19:43:02 2023-07-16 17:43:02,507|I| Webpage displayed "Client login succeeds"
2023-07-16 19:43:03 2023-07-16 17:43:03,509|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffffba82b5d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="ec8365269e764ebdf32b3a180cbacf4b")>
2023-07-16 19:43:03 2023-07-16 17:43:03,566|I| Logging in succeeded
2023-07-16 19:43:06 2023-07-16 17:43:06,686|E| Logging in succeeded, but active session is still not authenticated
2023-07-16 19:43:06 2023-07-16 17:43:06,701|I| Waiting 15 seconds to reauthenticate before restarting.
2023-07-16 19:43:21 2023-07-16 17:43:21,707|I| Logging out and reattempting full authentication
2023-07-16 19:43:21 2023-07-16 17:43:21,921|I| Gateway logout successful
2023-07-16 19:43:22 2023-07-16 17:43:22,051|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='34fa16dad99c99e99525272d709fad06', server_name=None, server_version=None, expires=600000)
2023-07-16 19:43:22 2023-07-16 17:43:22,051|I| Authentication strategy: "A"
2023-07-16 19:43:22 2023-07-16 17:43:22,051|I| Competing Gateway session found, restarting and logging in...
2023-07-16 19:43:22 2023-07-16 17:43:22,127|I| Gateway logout successful
2023-07-16 19:43:22 2023-07-16 17:43:22,127|I| Gateway session found but not authenticated, logging in...
2023-07-16 19:43:22 2023-07-16 17:43:22,128|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on```
dabrazhe commented 1 year ago

The loop is gone and authentication had succeeded upon using IBEAM_AUTHENTICATION_STRATEGY=B 


2023-07-17 14:22:28 2023-07-17 12:22:28,023|I| ############ Starting IBeam version 0.5.0-rc3 ############
2023-07-17 14:22:28 2023-07-17 12:22:28,028|I| Secrets source: env
2023-07-17 14:22:28 2023-07-17 12:22:28,032|I| Health server started at port=5001
2023-07-17 14:22:28 2023-07-17 12:22:28,032|I| Configuration:
2023-07-17 14:22:28 {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 2, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'HEALTH_SERVER_PORT': 5001, 'SECRETS_SOURCE': 'env', 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'NAME@@password', 'SUBMIT_EL': 'CSS_SELECTOR@@.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'TAG_NAME@@Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'CLASS_NAME@@ibkey-promo-skip', 'AUTHENTICATION_STRATEGY': 'B', 'MAX_STATUS_CHECK_RETRIES': 15, 'MAX_REAUTHENTICATE_RETRIES': 3, 'UI_SCALING': 1.0, 'TWO_FA_EL_ID': 'ID@@twofactbase', 'TWO_FA_NOTIFICATION_EL': 'CLASS_NAME@@login-step-notification', 'TWO_FA_INPUT_EL_ID': 'ID@@chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'ID@@sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key', 'CUSTOM_TWO_FA_HANDLER': 'custom_two_fa_handler.CustomTwoFaHandler'}
2023-07-17 14:22:28 2023-07-17 12:22:28,033|I| Gateway not found, starting new one...
2023-07-17 14:22:28 2023-07-17 12:22:28,033|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command.
2023-07-17 14:22:28 2023-07-17 12:22:28,033|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
2023-07-17 14:22:28 running  
2023-07-17 14:22:28  runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
2023-07-17 14:22:28  config file  : root/conf.yaml
2023-07-17 14:22:28 2023-07-17 12:22:28,047|I| Gateway started with pids: [13]
2023-07-17 14:22:28 2023-07-17 12:22:28,049|I| Cannot ping Gateway. Retrying for another 20 seconds
2023-07-17 14:22:28 WARNING: An illegal reflective access operation has occurred
2023-07-17 14:22:28 WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
2023-07-17 14:22:28 WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
2023-07-17 14:22:28 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2023-07-17 14:22:28 WARNING: All illegal access operations will be denied in a future release
2023-07-17 14:22:28  -> mount demo on /demo
2023-07-17 14:22:28 Java Version: 11.0.18
2023-07-17 14:22:28 ****************************************************
2023-07-17 14:22:28 version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400
2023-07-17 14:22:28 ****************************************************
2023-07-17 14:22:28 This is the Client Portal Gateway
2023-07-17 14:22:28 for any issues, please contact api@ibkr.com
2023-07-17 14:22:28 and include a copy of your logs
2023-07-17 14:22:28 ****************************************************
2023-07-17 14:22:28 https://www.interactivebrokers.com/api/doc.html
2023-07-17 14:22:28 ****************************************************
2023-07-17 14:22:28 Open https://localhost:5000 to login
2023-07-17 14:22:28 App demo is available after you login under: https://localhost:5000/demo#/
2023-07-17 14:22:29 2023-07-17 12:22:29,874|I| Gateway connection established
2023-07-17 14:22:30 2023-07-17 12:22:30,135|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2023-07-17 14:22:30 2023-07-17 12:22:30,135|I| Authentication strategy: "B"
2023-07-17 14:22:30 2023-07-17 12:22:30,135|I| No active sessions, logging in...
2023-07-17 14:22:30 2023-07-17 12:22:30,135|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-17 14:22:37 2023-07-17 12:22:37,710|I| Gateway auth webpage loaded
2023-07-17 14:22:37 2023-07-17 12:22:37,710|I| Login attempt number 1
2023-07-17 14:22:42 2023-07-17 12:22:42,882|I| Submitting the form
2023-07-17 14:22:43 2023-07-17 12:22:43,811|I| Webpage displayed "Client login succeeds"
2023-07-17 14:22:44 2023-07-17 12:22:44,814|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0xffff865bb210> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="7cbf961896b352acb275d6782b030731")>
2023-07-17 14:22:44 2023-07-17 12:22:44,879|I| Logging in succeeded
2023-07-17 14:22:45 2023-07-17 12:22:45,966|I| Repeating status check attempts another 14 times
2023-07-17 14:22:46 2023-07-17 12:22:46,550|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='eb04a61a5f8e19ada84f06d4120adfc4', server_name='JifZ02126', server_version='Build 10.20.1i, Jun 27, 2023 5:27:43 PM', expires=600000)
2023-07-17 14:22:46 2023-07-17 12:22:46,550|I| Gateway running and authenticated, session id: eb04a61a5f8e19ada84f06d4120adfc4, server name: JifZ02126
2023-07-17 14:22:46 2023-07-17 12:22:46,558|I| Starting maintenance with interval 60 seconds
2023-07-17 14:23:46 2023-07-17 12:23:46,569|I| Maintenance
2023-07-17 14:23:46 2023-07-17 12:23:46,670|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='c75acdc18d444e7bfb003cdbc790350b', server_name=None, server_version=None, expires=537699)
2023-07-17 14:23:46 2023-07-17 12:23:46,670|I| Authentication strategy: "B"
2023-07-17 14:23:46 2023-07-17 12:23:46,670|I| Competing or disconnected Gateway session found, logging out and reauthenticating...
2023-07-17 14:23:46 2023-07-17 12:23:46,853|I| Gateway logout successful
2023-07-17 14:23:48 2023-07-17 12:23:48,084|I| Repeating status check attempts another 14 times
2023-07-17 14:23:52 2023-07-17 12:23:52,988|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='d1ed2cb37a3b33cacccffe9c6743c202', server_name='JifZ11005', server_version='Build 10.23.2b, Jul 11, 2023 5:11:18 PM', expires=594149)
2023-07-17 14:23:52 2023-07-17 12:23:52,989|I| Gateway running and authenticated, session id: d1ed2cb37a3b33cacccffe9c6743c202, server name: JifZ11005
2023-07-17 14:24:46 2023-07-17 12:24:46,583|I| Maintenance
2023-07-17 14:24:46 2023-07-17 12:24:46,728|I| Gateway running and authenticated, session id: d1ed2cb37a3b33cacccffe9c6743c202, server name: JifZ30016
2023-07-17 14:25:46 2023-07-17 12:25:46,573|I| Maintenance```
dabrazhe commented 1 year ago

I created a script to request the tickle endpoint every minute and run it overnight and day. About 7% of tickle requests fail, which looks like it drops the auth session, below

@Voyz Is this expected, and how to remediate it?

2023-07-20 10:16:45,196|I| Maintenance
2023-07-20 10:16:45,379|I| Attempt number 2
2023-07-20 10:16:45,541|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable
2023-07-20 10:16:45,541|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2023-07-20 10:16:45,542|I| Authentication strategy: "B"
2023-07-20 10:16:45,542|I| No active sessions, logging in...
2023-07-20 10:16:45,542|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-20 10:16:53,118|I| Gateway auth webpage loaded
2023-07-20 10:16:53,118|I| Login attempt number 1
2023-07-20 10:16:58,456|I| Submitting the form
2023-07-20 10:17:00,117|I| Webpage displayed "Client login succeeds"
2023-07-20 10:17:01,117|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7ff80db9d9d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="7cbc9f7f84bbd276dce8c5d15396a274")>
2023-07-20 10:17:01,171|I| Logging in succeeded
2023-07-20 10:17:01,309|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='0100edd39588f1129f6720fa13d1fee1', server_name='JifZ33099', server_version='Build 10.23.2b, Jul 11, 2023 5:11:18 PM', expires=598309)
2023-07-20 10:17:01,309|I| Gateway running and authenticated, session id: 0100edd39588f1129f6720fa13d1fee1, server name: JifZ33099
Voyz commented 1 year ago

@dabrazhe have a look at my reply above in this post in case you missed it: https://github.com/Voyz/ibeam/issues/152#issuecomment-1636123686 In short, as far as I know there's little we can do about the authentication loop.

dabrazhe commented 1 year ago

Alas, the issue is back again.

Tried with a newly opened IBKR paper account and IBeam won't log in with any of the strategies. The Local GW instance logs in right away.

dabrazhe commented 1 year ago

This is the message I got from the IBKR Client Services @Voyz

====

While you may see a message in response stating "Client login succeeds" after logging in to the Client Portal Gateway, this is not always the full story. In order to confirm authentication, you will typically need to make a call to the /iserver/auth/status endpoint in order to confirm authentication status. If the response message includes "authenticated":false, then you have not been authenticated properly. Sometimes this is just a matter of logging in to the client portal gateway once again from the localhost page.

While we are working towards a more consistent result, it can sometimes be beneficial to try calling the /logout endpoint, then proceeding through the login procedure once again. This will clear out any stale or competing brokerage sessions that may be halting the login process for a fully authenticated brokerage session.

Voyz commented 1 year ago

@dabrazhe thanks for reporting what you heard from IBKR. Let me address these points:

While you may see a message in response stating "Client login succeeds" after logging in to the Client Portal Gateway, this is not always the full story. In order to confirm authentication, you will typically need to make a call to the /iserver/auth/status endpoint in order to confirm authentication status.

Yupp, we do that to confirm.

If the response message includes "authenticated":false, then you have not been authenticated properly.

We do that too, and interpret it in the same way.

Sometimes this is just a matter of logging in to the client portal gateway once again from the localhost page.

In Strategy A we do that exactly, in Strategy B we first try to reauthenticate before re-logging in.

While we are working towards a more consistent result, it can sometimes be beneficial to try calling the /logout endpoint, then proceeding through the login procedure once again.

We do this in Strategy A, to no avail. In Strategy B we reauthenticate instead, then if that fails repeatedly we kill the Gateway and log in. Note that in both cases we check for competing sessions, and logout if we find one.

Unless we've gotten something wrong or are missing something, none of these suggestions help unfortunately.

bfoz commented 1 year ago

We do this in Strategy A, to no avail. In Strategy B we reauthenticate instead, then if that fails repeatedly we kill the Gateway and log in. Note that in both cases we check for competing sessions, and logout if we find one.

Would it be worth it to try a logout even if a competing session isn't found?

bfoz commented 1 year ago

For my use case, and I suspect for many others', I don't actually need the gateway connected 24/7. I only need it connected when I need it to be. Is there a way to use that detail to limit the impact of this problem?

For example, would it be worthwhile to add something like "quiet hours"? That is, if the connection drops during those hours, don't bother looping until the "quiet" period ends. That wouldn't fix the problem, but would at least limit the notification storm on my phone.

Alternatively, don't immediately re-authenticate if the connection drops after the API hasn't been used within some timeout. Instead, hold off on re-authenticating until an API call comes in. Again, this isn't a true fix, and would probably require the addition of a proxy server, but it would limit the amount of looping.

Obviously these are both bad ideas, but it seems like we're out of good ideas. Maybe it's time to try some bad ideas until IB sees fit to join the modern era.

Voyz commented 1 year ago

@bfoz thanks for all these suggestions and questions 👍

Would it be worth it to try a logout even if a competing session isn't found?

Yes, we do that in Strategy A, but it doesn't seem to alleviate the problem 😔

I don't actually need the gateway connected 24/7. I only need it connected when I need it to be. Is there a way to use that detail to limit the impact of this problem?

I wouldn't know what impact would this have on the authentication loop problem, but that's an interesting hypothesis.

For example, would it be worthwhile to add something like "quiet hours"? That is, if the connection drops during those hours, don't bother looping until the "quiet" period ends.

That sounds like you want to create an scheduler for either your IBeam container or VM (or wherever you're running it on) - something that would start and stop the service during the 'quiet hours'. Cloud providers have some variations of pipelines that allow to do this. On Google Cloud Platform, for instance, you could use Cloud Scheduler or Instance Schedule. Or do I misunderstand something? I remember some users reporting that restarting IBeam container daily seemed to help.

Alternatively, don't immediately re-authenticate if the connection drops after the API hasn't been used within some timeout. Instead, hold off on re-authenticating until an API call comes in. Again, this isn't a true fix, and would probably require the addition of a proxy server, but it would limit the amount of looping.

Yupp, you'd need a proxy that would intercept the calls to the Gateway and administer IBeam accordingly. Actually my own setup used a proxy like this before (though for completely different reasons) - let me know if you'd find it useful and I could see if I can extract some code out of it to make it public.

Obviously these are both bad ideas, but it seems like we're out of good ideas.

Hahah, sorry this is just a fantastic quote, they should put it in a film someday. Or is it a quote from a film?

Anyway, I don't think these are bad ideas. However, I think they require a peripheral setup that is independent from IBeam itself. Nevertheless, if you find that doing so helps with the authentication loop, then definitely report back and let us know, as this could be very useful to know.

jussirantala commented 1 year ago

I think this problem is in the GW and not at api.ibkr.com. TradingView's IBKR integration (which uses api.ibkr.com) never has this issue. It doesn't let you log in at busy times though. I haven't been able to authenticate for 20 hours now with iBeam and I've tried everything. Contacted IBKR also but they stopped replying.

The GW is java app so you can decompile it easily to readable code (I did it once, you just drag the .jar file into the decompiler). Maybe somebody could fix it or add some debugger stuff. Also I'm not sure if the GW is really needed if you copy the certs etc. The cert pass is "mywebapi". You could probably connect directly to api.ibkr.com. The GW doesn't seem to do much if you check the code...

They seem to have at least three different APIs but only one is documented (3rd party): api.ibkr.com/v1/tv (TradingView) api.ibkr.com/tradingapi/v1 (3rd Party Web API) api.ibkr.com/v1/api (Client Portal Web API)

jussirantala commented 1 year ago

I sent these .har files for them today, let's see.

Could you please try endpoint "/logout" and endpoint "/reauthenticate" then "/tickle" 4 to 5 times to achieve the Authenticated: true.

In case the issue still persists, can you share .Har logs with us.

- Reproduce the steps where you experienced the issues using a browser to generate relevant log entries.
- You can also use the Swagger-UI for this testing purpose: [LINK../endpoints](https://interactivebrokers.github.io/cpwebapi/endpoints)
- Info on how to generate har logs: [LINK../3512](https://ibkr.info/article/3512)
- You can send the logs to api@interactivebrokers.com Kind regards,
Voyz commented 1 year ago

hey @jussirantala thanks for sharing your info 👍

As for decompiling the Gateway or going around it - interesting observations, though make sure you're not breaking IBKR's ToS by doing so.

Could you please try endpoint "/logout" and endpoint "/reauthenticate" then "/tickle" 4 to 5 times to achieve the Authenticated: true.

We tried that, to no avail unfortunately. Strategy A does exactly that, Strategy B will only logout if it sees a conflicting session.

jussirantala commented 1 year ago

It's weird that I'm not having any problems with the CPWEBAPI itself. Only if I use with IBeam. Do you think they can somehow detect it's inside docker container? Or maybe this is something that could be fixed with some docker configs or something.

CaptainTrunky commented 1 year ago

It's weird that I'm not having any problems with the CPWEBAPI itself. Only if I use with IBeam. Do you think they can somehow detect it's inside docker container? Or maybe this is something that could be fixed with some docker configs or something.

Are sure that your local client is exactly the same as the one inside IBeam? Apart from, it should be more or less impossible to detect that something runs inside a container. (I'm not able to login for about 1 month already, both with and without Docker)

jussirantala commented 1 year ago

@CaptainTrunky Looks like it's not the same.

My version: ibgroup.web.core.iblink.router.clientportal.gw-20230424154245 Ibeam version: ibgroup.web.core.iblink.router.clientportal.gw-20230320144412

@Voyz Seems master is using old GW? I downloaded the .jar file and checked manifest file with decompiler.

jussirantala commented 1 year ago

...I'm not able to login for about 1 month already, both with and without Docker

@CaptainTrunky Could you please send your GW logs to api@interactivebrokers.com ?

Voyz commented 1 year ago

@jussirantala I appreciate your thorough effort trying to figure this out 👍

The Gateway version inside of IBeam gets updated sporadically and doesn't follow the release cycle of IBKR. In the past I, and other users, tried seeing if updating it would solve any issues, but it never produced any observable improvements. That being said, if you believe it could be the version that is the problem, we can update it no problem - let me know.

As for why it works for you outside of IBeam - there could be a number of reasons. For example, if you're deploying the IBeam instance on a cloud provider, it has been observed that just communicating from a different IP can bring different results. A VPN has been shown to solve some problems. As another example, some users' country of registration affected their ability to connect. I'm bringing these up to illustrate that there could be some diverse reasons for why you could be observing different results with your standalone Gateway. Again, the recommended way would be to try to figure these reasons out with IBKR support.

jussirantala commented 1 year ago

Weird. IBeam can authenticate locally now but still not on my server. I deployed a new server which is located in the same country I am instead of New York but no luck... Maybe there's some information about the operating system in the headers (see HttpProxy.java) 🤔

Also I noticed it's sending your network adapter MAC address as an URL param when using the SSO (see SsoService.java and TstTokenUtils.java).

Docker start assigning always the same mac 02:42:ac:11:00:02 for the first container and then is increasing by one each mac for each different container. https://stackoverflow.com/questions/42946453/how-does-the-docker-assign-mac-addresses-to-containers

🤔

@Voyz Btw is it possible that you would publish 0.5.0 image? I've been trying to build it but I'm having some issues... Might be related to M1 processor.

EDIT: Tried to change my container MAC to something random but didn't help. EDIT: I'm continuing with ib-gateway-docker and ib-insync. Already wrote a REST API with python.

bfoz commented 1 year ago

Sorry for the slow reply.

Would it be worth it to try a logout even if a competing session isn't found?

Yes, we do that in Strategy A, but it doesn't seem to alleviate the problem 😔

Is it worth it to try doing the same in Strategy B?

For example, would it be worthwhile to add something like "quiet hours"? That is, if the connection drops during those hours, don't bother looping until the "quiet" period ends.

That sounds like you want to create an scheduler for either your IBeam container or VM (or wherever you're running it on) - something that would start and stop the service during the 'quiet hours'. Cloud providers have some variations of pipelines that allow to do this. On Google Cloud Platform, for instance, you could use Cloud Scheduler or Instance Schedule. Or do I misunderstand something? I remember some users reporting that restarting IBeam container daily seemed to help.

Actually, I'm thinking of the opposite of that. The scheduled start/stop solution potentially stops a perfectly connected and functioning gateway, which seems less than ideal. So instead, I'm imaging a mechanism that does nothing during the "quiet hours", and only does the authentication dance If the connection goes "bad" during the "non-quiet hours".

Alternatively, don't immediately re-authenticate if the connection drops after the API hasn't been used within some timeout. Instead, hold off on re-authenticating until an API call comes in. Again, this isn't a true fix, and would probably require the addition of a proxy server, but it would limit the amount of looping.

Yupp, you'd need a proxy that would intercept the calls to the Gateway and administer IBeam accordingly. Actually my own setup used a proxy like this before (though for completely different reasons) - let me know if you'd find it useful and I could see if I can extract some code out of it to make it public.

Creating a proxy seems like a heavy lift, especially if you've already tried it and decided that it didn't work. But if you ever feel like posting the code somewhere I'd be happy to take a look at it.

Obviously these are both bad ideas, but it seems like we're out of good ideas.

Hahah, sorry this is just a fantastic quote, they should put it in a film someday. Or is it a quote from a film?

It probably is a quote, but I have no idea what it's from if it is. If you ever use it in a film, mention me in the credits 🤣

Anyway, I don't think these are bad ideas. However, I think they require a peripheral setup that is independent from IBeam itself. Nevertheless, if you find that doing so helps with the authentication loop, then definitely report back and let us know, as this could be very useful to know.

The "quiet hours" idea might need some support, since it would require disabling the authentication loop at certain times.

CaptainTrunky commented 1 year ago

...I'm not able to login for about 1 month already, both with and without Docker

@CaptainTrunky Could you please send your GW logs to api@interactivebrokers.com ?

I've tried to do this, but I wasn't able to open an issue - support services were down for a while - ticket creation fails with an unknown issue.

Voyz commented 1 year ago

@jussirantala

Btw is it possible that you would publish 0.5.0 image? I've been trying to build it but I'm having some issues... Might be related to M1 processor.

Sure. I was travelling for the past few weeks so didn't want to roll it out yet. But I should have some time to do it now.

Voyz commented 1 year ago

@bfoz

Is it worth it to try doing the same in Strategy B?

My initial thoughts would be that no, I wouldn't do it. The reasoning here, is that Strategy B attempts to perform authentication in a different way to Strategy A, in order to see if that will be more reliable. I don't think making them very similar is the right way to go. If there was some strong indication that /logout would help then I'd change that opinion.

Actually, I'm thinking of the opposite of that. The scheduled start/stop solution potentially stops a perfectly connected and functioning gateway, which seems less than ideal. So instead, I'm imaging a mechanism that does nothing during the "quiet hours", and only does the authentication dance If the connection goes "bad" during the "non-quiet hours". ... The "quiet hours" idea might need some support, since it would require disabling the authentication loop at certain times.

I'm not sure if I'm following you here, sorry 😔 Currently we don't do anything during the 'quiet hours' and only 'do the authentication dance' if the connection goes bad. If the connection is fine we don't do anything. Could you open a separate issue and describe in detail how would you see this work?

Creating a proxy seems like a heavy lift, especially if you've already tried it and decided that it didn't work.

I created the proxy for a completely different task, unrelated to the authentication.

CaptainTrunky commented 1 year ago

FYI: Strategy B with voyz/ibeam-0.5rc4 seems to work for me as well.

greybox1 commented 1 year ago

Does this issue affect more paper trade accounts? Was working fine until today.

I also noticed seems to not like 2FA accounts. Seems to loop as well. Is this an IBK issue?

Getting running
runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* config file : root/conf.yaml -> mount demo on /demo Java Version: 11.0.18

version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400

This is the Client Portal Gateway for any issues, please contact api@ibkr.com and include a copy of your logs

https://www.interactivebrokers.com/api/doc.html

Open https://localhost:5007 to login App demo is available after you login under: https://localhost:5007/demo#/ running
runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* config file : root/conf.yaml -> mount demo on /demo Java Version: 11.0.18

version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400

This is the Client Portal Gateway for any issues, please contact api@ibkr.com and include a copy of your logs

https://www.interactivebrokers.com/api/doc.html

Open https://localhost:5007 to login App demo is available after you login under: https://localhost:5007/demo#/ running
runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* config file : root/conf.yaml -> mount demo on /demo Java Version: 11.0.18

version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400

This is the Client Portal Gateway for any issues, please contact api@ibkr.com and include a copy of your logs

https://www.interactivebrokers.com/api/doc.html

Open https://localhost:5007 to login App demo is available after you login under: https://localhost:5007/demo#/ 2023-09-25 14:29:57,749|I| ############ Starting IBeam version 0.4.6 ############ 2023-09-25 14:29:57,750|I| Custom conf.yaml found and will be used by the Gateway 2023-09-25 14:29:57,751|I| Secrets source: env 2023-09-25 14:29:57,752|I| Health server started at port=5001 2023-09-25 14:29:57,752|I| Environment variable configuration: {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'IBEAM_HEALTH_SERVER_PORT': 5001, 'GATEWAY_BASE_URL': 'https://localhost:5007', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'password', 'SUBMIT_EL': 'button.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'ibkey-promo-skip', 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_NOTIFICATION_EL': 'login-step-notification', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key'} 2023-09-25 14:29:57,753|I| Gateway not found, starting new one... 2023-09-25 14:29:57,753|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command. 2023-09-25 14:29:57,753|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml'] WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2023-09-25 14:29:58,754|I| Gateway started with pid: 13 2023-09-25 14:29:59,148|I| Gateway connection established 2023-09-25 14:29:59,812|I| No active sessions, logging in... 2023-09-25 14:29:59,812|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:30:07,447|I| Gateway auth webpage loaded 2023-09-25 14:30:07,447|I| Login attempt number 1 2023-09-25 14:30:12,654|I| Submitting the form 2023-09-25 14:30:13,441|I| Webpage displayed "Client login succeeds" 2023-09-25 14:30:15,442|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61e069490> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="58e3da780d0646242ff8fad0390bbf26")> 2023-09-25 14:30:15,483|I| Authentication process succeeded 2023-09-25 14:30:18,737|E| Gateway session active but not authenticated 2023-09-25 14:30:18,758|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:30:33,758|I| Logging out and restarting the Gateway 2023-09-25 14:30:34,002|I| Gateway logout successful 2023-09-25 14:30:34,363|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:30:34,363|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:30:41,106|I| Gateway auth webpage loaded 2023-09-25 14:30:41,106|I| Login attempt number 1 2023-09-25 14:30:46,311|I| Submitting the form 2023-09-25 14:30:47,121|I| Webpage displayed "Client login succeeds" 2023-09-25 14:30:49,121|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61e06b5d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="5cbb0f0be8f89a9b3b55cbee5c52535f")> 2023-09-25 14:30:49,154|I| Authentication process succeeded 2023-09-25 14:30:52,367|E| Gateway session active but not authenticated 2023-09-25 14:30:52,375|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:31:07,375|I| Logging out and restarting the Gateway 2023-09-25 14:31:07,739|I| Gateway logout successful 2023-09-25 14:31:07,986|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:31:07,986|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:31:14,459|I| Gateway auth webpage loaded 2023-09-25 14:31:14,459|I| Login attempt number 1 2023-09-25 14:31:19,665|I| Submitting the form 2023-09-25 14:31:20,475|I| Webpage displayed "Client login succeeds" 2023-09-25 14:31:22,475|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61e07be50> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="5f18b38ee6bee40d04f1afe4774ff26e")> 2023-09-25 14:31:22,508|I| Authentication process succeeded 2023-09-25 14:31:25,760|E| Gateway session active but not authenticated 2023-09-25 14:31:25,767|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:31:40,767|I| Logging out and restarting the Gateway 2023-09-25 14:31:41,239|I| Gateway logout successful 2023-09-25 14:31:41,503|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:31:41,503|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:31:48,050|I| Gateway auth webpage loaded 2023-09-25 14:31:48,050|I| Login attempt number 1 2023-09-25 14:31:53,260|I| Submitting the form 2023-09-25 14:31:54,071|I| Webpage displayed "Client login succeeds" 2023-09-25 14:31:56,071|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61e084350> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="592b3c1201bf1a9d01bd14f6c2636009")> 2023-09-25 14:31:56,100|I| Authentication process succeeded 2023-09-25 14:31:59,314|E| Gateway session active but not authenticated 2023-09-25 14:31:59,321|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:32:14,322|I| Logging out and restarting the Gateway 2023-09-25 14:32:14,602|I| Gateway logout successful 2023-09-25 14:32:14,883|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:32:14,883|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:32:21,482|I| Gateway auth webpage loaded 2023-09-25 14:32:21,482|I| Login attempt number 1 2023-09-25 14:32:26,718|I| Submitting the form 2023-09-25 14:32:27,508|I| Webpage displayed "Client login succeeds" 2023-09-25 14:32:29,508|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb620062bd0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="95fa44b850ff9d5f25a9269f0992320e")> 2023-09-25 14:32:29,542|I| Authentication process succeeded 2023-09-25 14:32:32,781|E| Gateway session active but not authenticated 2023-09-25 14:32:32,788|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:32:47,788|I| Logging out and restarting the Gateway 2023-09-25 14:32:48,004|I| Gateway logout successful 2023-09-25 14:32:48,224|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:32:48,224|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:32:55,050|I| Gateway auth webpage loaded 2023-09-25 14:32:55,050|I| Login attempt number 1 2023-09-25 14:33:00,264|I| Submitting the form 2023-09-25 14:33:01,079|I| Webpage displayed "Client login succeeds" 2023-09-25 14:33:03,079|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61f7cc850> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="ee4b48ec180c6cbe0efeb10af48a46f3")> 2023-09-25 14:33:03,120|I| Authentication process succeeded 2023-09-25 14:33:06,319|E| Gateway session active but not authenticated 2023-09-25 14:33:06,326|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:33:21,326|I| Logging out and restarting the Gateway 2023-09-25 14:33:21,567|I| Gateway logout successful 2023-09-25 14:33:21,871|I| Gateway session found but not authenticated, authenticating... 2023-09-25 14:33:21,872|I| Loading auth webpage at https://localhost:5007/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-09-25 14:33:28,324|I| Gateway auth webpage loaded 2023-09-25 14:33:28,324|I| Login attempt number 1 2023-09-25 14:33:33,538|I| Submitting the form 2023-09-25 14:33:34,335|I| Webpage displayed "Client login succeeds" 2023-09-25 14:33:36,335|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7fb61e086e50> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="1ad543111ea3527964827512631229ba")> 2023-09-25 14:33:36,368|I| Authentication process succeeded 2023-09-25 14:33:39,667|E| Gateway session active but not authenticated 2023-09-25 14:33:39,673|I| Waiting 15 seconds to reauthenticate before restarting. 2023-09-25 14:33:54,673|I| Logging out and restarting the Gateway 2023-09-25 14:33:55,054|I| Gateway logout successful 2023-09-25 14:33:55,331|I| Gateway session found but not authenticated, authenticating...

Voyz commented 1 year ago

@greybox1 we haven't noticed whether this would affect paper accounts more than live. And yes, this seems to be an IBKR issue.

jussirantala commented 1 year ago

FYI, in August, I sent all the logs etc which IBKR asked. After that I've asked two times how they are progressing. The first reply was "Still investigating". And they didn't reply at all the second time.

Voyz commented 1 year ago

@bfoz @jussirantala @dabrazhe and others - there's a new attempt at fixing the authentication loop: 0.5.2-rc1. Please give it a try and let us know if it helps with the authentication loop.

It's based on some intel from @greybox1 talking to the IBKR support that casts some light on the auth loop.

According to the support agents, even though you may be authenticated, the authenticated flag may return false for some time. They mentioned 20 seconds, though realistically this could be more. I'm thinking this could be a server overload issue, which could explain why we're seeing it happen so randomly.

I've increased the IBEAM_MAX_STATUS_CHECK_RETRIES from 15 to 120 on this new image, which should allow for 2 minutes of checking for the authenticated flag. If you want to avoid pulling a new image, you can just set that flag to 120 yourself and also report back.

If this doesn't help, the next step would be to stop relying on authenticated: true flag as an indication of our authentication status, but try some other endpoint and see if it returns 401 Unauthorised.

jcity commented 11 months ago

@Voyz I'm running into what I think is a similar issue here. I'm running version 0.5.1 with the IBEAM_MAX_STATUS_CHECK_RETRIES set to 120. It appears the gateway thinks it's authenticated but calls to the iserver endpoints fail.

Here are my startup logs:

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                message                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1704214855613 | 2024-01-02 17:00:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704214855897 | 2024-01-02 17:00:55,897|I| Gateway running and authenticated, session id: 50104f9e12579bf7dd1c5a442f96fbc1, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704214915613 | 2024-01-02 17:01:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704214915750 | 2024-01-02 17:01:55,750|I| Gateway running and authenticated, session id: 50104f9e12579bf7dd1c5a442f96fbc1, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704214975613 | 2024-01-02 17:02:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704214975746 | 2024-01-02 17:02:55,746|I| Gateway running and authenticated, session id: 50104f9e12579bf7dd1c5a442f96fbc1, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704214975981 | 2024-01-02 17:02:55,981|W| Validation result is False when IBeam attempted to extend the SSO token. This could indicate token authentication issues.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1704215035613 | 2024-01-02 17:03:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704215035808 | 2024-01-02 17:03:55,808|I| Attempt number 2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| 1704215036067 | 2024-01-02 17:03:56,067|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 1704215036067 | 2024-01-02 17:03:56,067|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 1704215036067 | 2024-01-02 17:03:56,067|I| Authentication strategy: "B"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 1704215036068 | 2024-01-02 17:03:56,067|I| No active sessions, logging in...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1704215036068 | 2024-01-02 17:03:56,068|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1704215043036 | 2024-01-02 17:04:03,036|I| Gateway auth webpage loaded                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704215043036 | 2024-01-02 17:04:03,036|I| Login attempt number 1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 1704215048409 | 2024-01-02 17:04:08,409|I| Submitting the form                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 1704215049201 | 2024-01-02 17:04:09,201|I| Webpage displayed "Client login succeeds"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1704215050201 | 2024-01-02 17:04:10,201|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f76b37ac110> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="92b6ce46609a280ac485dec06f7c9ebe")>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 1704215050234 | 2024-01-02 17:04:10,234|I| Logging in succeeded                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1704215050363 | 2024-01-02 17:04:10,363|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='6e674375c167b12089c9630a4ce81eda', server_name='JifN14004', server_version='Build 10.26.0b, Dec 20, 2023 12:24:34 PM', expires=598874)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1704215050364 | 2024-01-02 17:04:10,363|I| Gateway running and authenticated, session id: 6e674375c167b12089c9630a4ce81eda, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704215095613 | 2024-01-02 17:04:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704215095737 | 2024-01-02 17:04:55,736|I| Gateway running and authenticated, session id: 6e674375c167b12089c9630a4ce81eda, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704215137140 | 2024-01-02 17:05:37,140|I| ############ Starting IBeam version 0.5.1 ############                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 1704215137141 | 2024-01-02 17:05:37,141|I| Custom conf.yaml found and will be used by the Gateway                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 1704215137143 | 2024-01-02 17:05:37,143|I| Secrets source: env                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 1704215137144 | 2024-01-02 17:05:37,144|I| Health server started at port=5001                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1704215137144 | 2024-01-02 17:05:37,144|I| Configuration:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| 1704215137144 | {'UNDEFINED': <object object at 0x7f08346b8580>, 'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 2, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'HEALTH_SERVER_PORT': 5001, 'SECRETS_SOURCE': 'env', 'GCP_SECRETS_URL': None, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'NAME@@password', 'SUBMIT_EL': 'CSS_SELECTOR@@.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'TAG_NAME@@Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'CLASS_NAME@@ibkey-promo-skip', 'AUTHENTICATION_STRATEGY': 'B', 'MAX_STATUS_CHECK_RETRIES': 120, 'MAX_REAUTHENTICATE_RETRIES': 3, 'UI_SCALING': 1.0, 'TWO_FA_EL_ID': 'ID@@twofactbase', 'TWO_FA_NOTIFICATION_EL': 'CLASS_NAME@@login-step-notification', 'TWO_FA_INPUT_EL_ID': 'ID@@chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'ID@@sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key', 'CUSTOM_TWO_FA_HANDLER': 'custom_two_fa_handler.CustomTwoFaHandler'} |
| 1704215137144 | 2024-01-02 17:05:37,144|I| Gateway not found, starting new one...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| 1704215137145 | 2024-01-02 17:05:37,144|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704215137145 | 2024-01-02 17:05:37,145|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

I'm not sure if this is helpful but, it appears that it detects the SSO Token is invalid but never tries to revalidate it.

When I call the /iserver/auth/status endpoint I get the following response:

{
    "authenticated": false,
    "competing": false,
    "connected": false,
    "MAC": "F4:03:43:CD:BD:C0"
}

When I call an endpoint that requires a valid session, I get a response that says to initiate a session first. For example here is the response from trying to place an order with this endpoint /iserver/account/<account-id>/orders

{
    "error": "Init session first",
    "statusCode": 400
}

I think your suggestion of trying another endpoint to determine auth status could work. I'm happy to give this a try if you want to point me to the spot in the code to make the change and maybe suggest an endpoint to use.

Voyz commented 10 months ago

Hey @jcity thanks for reporting all that back.

I think your suggestion of trying another endpoint to determine auth status could work

To be honest, if you get that "Init session first" response from /iserver/account/<account-id>/orders then this debunks my idea to 'call another endpoint', as obviously this doesn't work and their servers continue to tell us the authentication needs to be true first 😔

As for the log you added, not sure if that's a complete log or if it misses something? These lines are confusing:

| 1704215050363 | 2024-01-02 17:04:10,363|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='6e674375c167b12089c9630a4ce81eda', server_name='JifN14004', server_version='Build 10.26.0b, Dec 20, 2023 12:24:34 PM', expires=598874)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1704215050364 | 2024-01-02 17:04:10,363|I| Gateway running and authenticated, session id: 6e674375c167b12089c9630a4ce81eda, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704215095613 | 2024-01-02 17:04:55,612|I| Maintenance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1704215095737 | 2024-01-02 17:04:55,736|I| Gateway running and authenticated, session id: 6e674375c167b12089c9630a4ce81eda, server name: JifN14004                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1704215137140 | 2024-01-02 17:05:37,140|I| ############ Starting IBeam version 0.5.1 ############

So it confirms being authenticated, works fine, continues working fine, and the suddenly starts a new IBeam instance?

jcity commented 10 months ago

Hmmmm, so this is strange and may not end up being unrelated to the issue being discussed here.

When I run ibeam locally, I'm able to establish a session but when I run as a docker container in AWS Fargate, a session is never able to be established and I continue to the the "Init session first" error

I'm wondering if I've potantially hit the rate limit and they have blocked the IP associated with my Internet Gateway

Voyz commented 10 months ago

Interesting @jcity - could be also that your AWS server is hitting a different IBKR server which causes the issues.

pinkfrog9 commented 10 months ago

@Voyz Does TWS also suffer the auth issue?

Voyz commented 10 months ago

@pinkfrog9 TWS is a completely different interface, and as such I don't think this particular issue occurs, at least not in the exact same form. However, TWS has its own set of issues, and some of them are also connectivity related.

jcity commented 10 months ago

I want to chime in here and close the thread on my end.

I could never get this stable enough where I felt comfortable it would work in production. At no fault to @Voyz and all the contributors -- thank you all. This is the best project I've seen to get something somewhat working.

I ended up switching to the 1st party OAuth integration and that has been relatively stable. Unfortunately, I don't think it's available for personal accounts at this time. You do still have to maintain a session as well but that piece has been relatively stable.

We also intermittently get ReadTimeout and ConnectTimeout errors during periods of high traffic but a retry usually solves it. Unfortunately, IBKR API just isn't as great as you would hope it would be for a company that size.

Voyz commented 8 months ago

I'm going to close this issue due to inactivity. Thanks for your contribution and please feel free to request a reopen if you'd like to continue the discussion 👍

dabrazhe commented 8 months ago

One last thing. I played with various types of IB accounts and noticed that it works best with a paper account coupled with a Live account. The live account does not need to be funded. When I opened a standalone paper account it did not work. IB advised me to open another pair of Live-Paper accounts and the container proved to be quite stable. The server stays connected for weeks, sometimes months until if fails for some reason (I don't see any Voyz logs at that point but able to login to the container). The B strategy works best for me.

Voyz commented 8 months ago

@dabrazhe many thanks for adding all that! Could I ask you to elaborate a tiny bit? This could be very useful to a lot of us here.

I played with various types of IB accounts and noticed that it works best with a paper account coupled with a Live account

When you say 'coupled' - what do you mean? I though that all paper accounts need to have at least one live account?

When I opened a standalone paper account it did not work.

What's a 'standalone paper account'? From what I understand when you open an account with them you get a live account, and you can add a paper credential to it. What would 'standalone' be?

The server stays connected for weeks, sometimes months until if fails for some reason

I'm honestly really happy to read this. Great job! 👏 Is that a live server? If so, how do you handle 2FA?

dabrazhe commented 7 months ago

Hi @Voyz What I was able to grasp from tests and communication with the IB Support: Every Live account can have a linked paper account. Only one paper account is allowed, it’s their tech limitation. If your Live account is approved you can leave it non-funded, and open the linked paper one.

I'm honestly really happy to read this. Great job! 👏

The credit is all yours! Thank you for your work and the hours you put in.

Is that a live server? If so, how do you handle 2FA?

A live account is useless for my purposes because it requires MFA authentication every day. IB won’t budge on this. This would solve my problems because the things I need from the IB API are the margins for my account and real-time pricing subscription. Which the paper acc. does not have. (

When you say 'coupled' - what do you mean? I though that all paper accounts need to have at least one live account?

You can choose to open a paper account without the Live one, but this does not work for me. After back and forth with support, it was clear that Live is needed for the stable API server run.

dabrazhe commented 7 months ago

The container can run for months, but if you are interested it crashed with this kind of errors.

image
Voyz commented 7 months ago

@dabrazhe thanks for reporting back on your interaction with IBKR support. I've added a section to troubleshooting based off of it.

Also thanks for sharing that log. Indeed, Invalid Username and Password is a frequent out-of-nowhere error. Hope it fixed itself 🤞

rileyvdl16 commented 6 months ago

Hi @Voyz and all,

I came across your repo and this particular issue as I've built a very similar tool (not knowing that iBeam was already out there in the wild) that occasionally has a very similar login loop issue. I tried many of the same things in this thread including a full rm -r of the gateway and then re-unzip for a fresh install from an automated shell script thinking that IBKR were specifically looking to block automated logins. While I'm not absolutely convinced that they're looking to block automated logins, I'm not entirely unconvinced either. Irrespective, the entire login process is an absolute exercise in frustration.

The three thing I've done which appear to have helped (but not 100% solved) the login issue in the IBKR gateway conf.yaml file are:

  1. Disabled SSL between my API application and the Java Gateway on local host. This is the listenSsl variable. This is because I regularly saw SSL exceptions thrown on the client gateway logs. Disabling SSL stopped these. Your connection to IBKR should still be secure. This is the proxyRemoteSsl setting
  2. Changed proxyRemoteHost to the nearest set of servers to me. See here for IBKR servers https://ibkrcampus.com/ibkr-api-page/cpapi-v1/#oauth-base-url This bypasses what I can only assume is their primary load balancer at api.ibkr.com and directs your traffic to your chosen server.
  3. Changed ip2loc from US to the country I'm in. Full disclosure, I'm not sure if this had any effect

If I'm not mistaken, most or all of these settings are available in the iBeam docker env list. Hopefully some or all of these help with your login issues.

Voyz commented 6 months ago

Hey @rileyvdl16 many thanks for sharing all that info! And glad to hear you've already found IBeam. Btw. check out IBind if you'd be looking for a Python client for the IBKR Web API.

While I'm not absolutely convinced that they're looking to block automated logins, I'm not entirely unconvinced either.

There is no way of knowing, really. Occam's razon would suggest that it is not deliberate but accidental though.

Disabled SSL between my API application and the Java Gateway on local host.

Many thanks for suggesting this!

If I'm not mistaken, most or all of these settings are available in the iBeam docker env list

Almost - you indeed can modify these, yet not through env vars but by providing a custom conf.yaml file. Thanks for the suggestions 👍

demansou commented 2 weeks ago

Just wanted to comment that I was having problems running on Apple silicon (M2 Pro).

I just got a new Raspberry Pi to self host some things and deployed the docker container just to check things and it's incredibly stable running in Docker using Portainer.