v0.5.0 - development - Githubissues

This issue is tracking the development of IBeam v0.5.0.

You can test the v0.5.0 by pulling the most recent -rc* tag, eg: voyz/ibeam:0.5.0-rc1. Go to IBeam's Docker Hub and check what the most recent v0.5.0 tag is.

Major changes:

IBeam now supports various login logic flows - we call them strategies. These can be selected by setting IBEAM_AUTHENTICATION_STRATEGY env var as A or B. It is possible that this is a temporary feature that will be removed once a clear winning strategy is found.
Currently used login logic is called strategy 'A', and is used by default.
New strategy 'B' was added, based on @migoohao's suggestions in #146. The new logic is described in detail below. This is done in an attempt to simplify the login logic and rely on reauthenticate more than on full re-login. Many thanks for suggesting this logic flow @migoohao 👏 I also want to give kudos to any other users who mentioned using reauthenticate in the past.

New strategy 'B' logic:

See https://github.com/Voyz/ibeam/blob/v0.5.0/ibeam/src/strategy_handler.py and look for try_authenticating and _authentication_strategy_B

Context:

Strategy 'B', just like strategy 'A', is run on every maintenance.

Note that the result of this logic can take one of the four forms:

✅ all good - the Gateway is running and authenticated 🔐 need auth - the Gateway is running but we need to authenticate ⚠️ something's wrong - there's been some issue so we stop this authentication attempt and wait until next maintenance 🚨 shutdown - there has been a critical error during login, so we shut IBeam down

The different authentication actions that can be taken are:

login - perform a "full" login, using the web form.
reauthenticate - perform a reauthentication by calling 'reauthenticate' endpoint. This requires a login to have already happened.
logout - perform a logout by calling the 'logout' endpoint. This, somehow, will not invalidate the session, so we can still reauthenticate afterwards.

The logic is described below:

(this is subject to change)

Check Status. In short, 'checking Status' is a call to tickle endpoint that refreshes the session and provides us with a bunch of information on the Gateway, the current session and our authentication status. Note that Status now contains additional information (see source).
If authenticated and not competing: ✅ all good
if not running: ⚠️ something's wrong - we can't communicate with the Gateway, so we won't attempt authentication.

Above steps are the exact same for strategy 'A' and strategy 'B'. The following is where strategy 'B' differs:

if not session: 🔐 need auth - there is a running Gateway but it has no active session, so we login using the web form
elif not connected or competing: 🔐 need auth - this is an indication of a conflicting session (eg. TWS or Web Portal) taking over the credentials. We logout, and then reauthenticate.
else: 🔐 need auth - the session is present, just needs authentication, so we reauthenticate.

Irrespectively of which of the three cases take place, we end up with carrying out the post authentication double-checks:

Repeatedly check the Status to see if authentication was successful: a) Check Status every second, up to IBEAM_MAX_STATUS_CHECK_RETRIES times (15 by default) b) If all Status checks finish and don't report successful authentication, call reauthenticate once again. c) Repeat this whole process up to IBEAM_MAX_REAUTHENTICATE_RETRIES times (3 by default)
if any of the Status checks reports not running or not session or not connected or competing or not authenticated:⚠️ something's wrong - the Gateway is having some issues and we aren't going to keep on trying. We'll kill its process. It will be restarted on next maintenance, hoping this will solve the issues.
else: ✅ all good - we've verified that the authentication worked! 🥳

If you want to test this version:

Set IBEAM_AUTHENTICATION_STRATEGY to B in order to test the new strategy
Pay attention to how it handles error cases - such as repeated login loops, errors, timeouts, etc.
Report any observations, errors and ideas directly in this issue.
Accept my sincere thanks for helping us all out 🙏

Minor changes:

There has been some tidying up of the `gateway_client.py` file:

moved login/reauth logic into strategy_handler
moved all HTTP/request-related functionality to http_handler
moved all OS process-related functionality to process_utils
moved handling of secrets into secrets_handler

And minor fixes:

reworked the Status class and handling of Gateway/authentication status
added fixes to process starting and killing, ensuring we kill all processes
added more verbosity to 'Gateway running and authenticated' by attaching session_id and server_name
added support for GET and POST requests according to IBKR docs
using word 'logging in' when referring to doing a full webpage login, otherwise 'reauthenticate'
fixed KeyboardInterrupt behaviour by shutting down the health server

Roadmap:

If this new strategy 'B' requires tweaks, we'll work on these as first priority - this is the main goal of the v0.5.0 release.
~If there is time, I'd want to look into the authenticate.py file and rework that logic there. It has grown from a small function into some nasty rotten spaghetti code and also deserves some love.~
~Improve on OOP and testability by unifying all environment variable reading into one place, and pass arguments from then on.~

Here are the logs from my first attempt at using strategy B:

Attaching to ibeam ibeam | WARNING: All illegal access operations will be denied in a future release ibeam | 2023-06-29 16:28:35,793|I| Cannot ping Gateway. Retrying for another 19 seconds ibeam | -> mount demo on /demo ibeam | Java Version: 11.0.18 ibeam | **** ibeam | version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400 ibeam | **** ibeam | This is the Client Portal Gateway ibeam | for any issues, please contact api@ibkr.com ibeam | and include a copy of your logs ibeam | **** ibeam | https://www.interactivebrokers.com/api/doc.html ibeam | **** ibeam | Open https://localhost:5000 to login ibeam | App demo is available after you login under: https://localhost:5000/demo#/ ibeam | 2023-06-29 16:28:37,483|I| Gateway connection established ibeam | 2023-06-29 16:28:37,816|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) ibeam | 2023-06-29 16:28:37,816|I| Authentication strategy: "B" ibeam | 2023-06-29 16:28:37,816|I| No active sessions, logging in... ibeam | 2023-06-29 16:28:37,816|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | 2023-06-29 16:28:47,330|I| Gateway auth webpage loaded ibeam | 2023-06-29 16:28:47,330|I| Login attempt number 1 ibeam | 2023-06-29 16:28:52,689|I| Submitting the form ibeam | 2023-06-29 16:29:03,151|I| Webpage displayed "Client login succeeds" ibeam | 2023-06-29 16:29:05,151|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f19761c73d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="34f3177e6b18ad0f5eebac5774f1ab7e")> ibeam | 2023-06-29 16:29:05,204|I| Logging in succeeded ibeam | 2023-06-29 16:29:06,337|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:29:21,927|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:29:21,928|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='85cab2bd4a8505681028c4664a9dd510', server_name=None, server_version=None, expires=581466) ibeam | 2023-06-29 16:29:21,954|I| Repeated reauthentication attempt number 2 ibeam | 2023-06-29 16:29:23,103|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:29:38,446|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:29:38,446|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='85cab2bd4a8505681028c4664a9dd510', server_name=None, server_version=None, expires=564917) ibeam | 2023-06-29 16:29:38,462|I| Repeated reauthentication attempt number 3 ibeam | 2023-06-29 16:29:39,638|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:29:55,207|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:29:55,207|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='85cab2bd4a8505681028c4664a9dd510', server_name=None, server_version=None, expires=548161) ibeam | 2023-06-29 16:29:55,221|I| Repeated reauthentication attempt number 4 ibeam | 2023-06-29 16:29:56,384|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:30:11,883|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:30:11,883|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='85cab2bd4a8505681028c4664a9dd510', server_name=None, server_version=None, expires=531485) ibeam | 2023-06-29 16:30:11,898|I| Repeated reauthentication attempt number 5 ibeam | 2023-06-29 16:30:13,043|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:30:28,386|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:30:28,386|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='85cab2bd4a8505681028c4664a9dd510', server_name=None, server_version=None, expires=514991) ibeam | 2023-06-29 16:30:28,387|I| Max reauthenticate retries reached after 5 attempts. Consider increasing the retries by setting IBEAM_MAX_REAUTHENTICATE_RETRIES environment variable ibeam | 2023-06-29 16:30:28,387|E| Repeatedly reauthenticating failed 5 times. Killing the Gateway and restarting the authentication process. ibeam | 2023-06-29 16:30:29,394|I| Starting maintenance with interval 60 seconds

After the maintenance interval:

ibeam | 2023-06-29 16:31:32,801|I| Authentication strategy: "B" ibeam | 2023-06-29 16:31:32,801|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on ibeam | 2023-06-29 16:31:39,948|I| Gateway auth webpage loaded ibeam | 2023-06-29 16:31:39,948|I| Login attempt number 1 ibeam | 2023-06-29 16:31:45,301|I| Submitting the form ibeam | 2023-06-29 16:32:00,901|E| Timeout reached searching for website elements, but the website seems to be loaded correctly. It is possible the setup is incorrect. ibeam | Website version: 2 ibeam | DOM elements searched for: {'USER_NAME_EL': 'username', 'PASSWORD_EL': 'password', 'SUBMIT_EL': 'button.btn.btn-lg.btn-primary', 'ERROR_EL': 'xyz-errormessage', 'SUCCESS_EL_TEXT': 'Client login succeeds', 'IBKEY_PROMO_EL_CLASS': 'ibkey-promo-skip', 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_NOTIFICATION_EL': 'login-step-notification', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_SELECT_EL_ID': 'sf_select'}. ibeam | Exception: ibeam | File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap ibeam | self._bootstrap_inner() ibeam | File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner ibeam | self.run() ibeam | File "/usr/local/lib/python3.11/threading.py", line 975, in run ibeam | self._target(*self._args, self._kwargs) ibeam | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker ibeam | work_item.run() ibeam | File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run ibeam | result = self.fn(*self.args, *self.kwargs) ibeam | File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job ibeam | retval = job.func(job.args, job.kwargs) ibeam | File "/srv/ibeam/src/gateway_client.py", line 159, in _maintenance ibeam | success, shutdown, status = self.start_and_authenticate(request_retries=var.REQUEST_RETRIES) ibeam | File "/srv/ibeam/src/gateway_client.py", line 128, in start_and_authenticate ibeam | success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries) ibeam | File "/srv/ibeam/src/strategy_handler.py", line 84, in try_authenticating ibeam | return self._authentication_strategy_B(status, request_retries) ibeam | File "/srv/ibeam/src/strategy_handler.py", line 139, in _authentication_strategy_B ibeam | return self._log_in(status) ibeam | File "/srv/ibeam/src/strategy_handler.py", line 150, in _log_in ibeam | success, shutdown = self.log_in_function() ibeam | File "/srv/ibeam/src/gateway_client.py", line 106, in _log_in ibeam | return log_in(driver_path=self.driver_path, ibeam | File "/srv/ibeam/src/authenticate.py", line 312, in log_in ibeam | trigger = WebDriverWait(driver, var.OAUTH_TIMEOUT).until( ibeam | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ibeam | File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until ibeam | raise TimeoutException(message, screen, stacktrace) ibeam | ibeam | <class 'selenium.common.exceptions.TimeoutException'> Message: ibeam | ibeam | 2023-06-29 16:32:00,901|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f1975f99710> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="70896739a2191be3955fe9fb4ae1a233")> ibeam | 2023-06-29 16:32:00,951|I| Logging in failed ibeam | 2023-06-29 16:32:29,395|I| Maintenance

Sadly, it looks like I'll have to abandon strategy B for now because it appears to be stuck in a loop.

ibeam | 2023-06-29 16:44:46,842|I| Webpage displayed "Client login succeeds" ibeam | 2023-06-29 16:44:48,843|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f1975f88450> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="d4baa27f202d25a12cdc1d975996231f")> ibeam | 2023-06-29 16:44:48,889|I| Logging in succeeded ibeam | 2023-06-29 16:44:50,019|I| Repeating status check attempts another 14 times ibeam | 2023-06-29 16:45:05,283|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable ibeam | 2023-06-29 16:45:05,283|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='e85dcbb1fb41a4a61684937149618ae3', server_name=None, server_version=None, expires=581698)

@Voyz From my side, it has been running the whole day today, and I didn't observe any issues. I will leave it for the next few days and let you know if something goes wrong.

@Voyz 👏 Strategy B worked well in my server last week, good job! 🥳

Do not need my app's help now
No login loop
recover during maintenance quickly

Let's observe it for more weeks.

@migoohao Mine is going into a authentication loop for strategy B in our Debian GNU/Linux 11 (bullseye) Server. Can you please help with your server setup ?

@Voyz 0.5 Strategy B version throws error (attached in the log below) during maintenance. Is this an expected behaviour ? Thanks

Logs Set 1

2023-07-11 09:08:16,411|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:09:15,779|I| Maintenance 2023-07-11 09:09:16,413|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:10:15,779|I| Maintenance 2023-07-11 09:10:16,437|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:11:15,779|I| Maintenance 2023-07-11 09:11:16,408|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:12:15,779|I| Maintenance 2023-07-11 09:12:16,563|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:13:15,779|I| Maintenance 2023-07-11 09:13:30,807|E| Connection timeout after 15 seconds. Consider increasing timeout by setting IBEAM_REQUEST_TIMEOUT environment variable. Error: The read operation timed out 2023-07-11 09:13:30,807|I| Attempt number 2 2023-07-11 09:13:31,053|E| Unrecognised HTTPError Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 119, in _request response = self.url_request(url, method=method) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/ibeam/src/http_handler.py", line 104, in url_request return urllib.request.urlopen(req, context=self.ssl_context, timeout=self.request_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error return self._call_chain(*args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 139, in _request raise RuntimeError('Unrecognised HTTPError') from e RuntimeError: Unrecognised HTTPError 2023-07-11 09:13:31,058|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2023-07-11 09:13:31,058|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 09:13:31,058|I| Authentication strategy: "B" 2023-07-11 09:13:31,058|I| No active sessions, logging in... 2023-07-11 09:13:31,058|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-11 09:14:07,145|E| Timeout reached when waiting for authentication. The website seems to not be loaded correctly. Consider increasing IBEAM_PAGE_LOAD_TIMEOUT. Website URL: https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on IBEAM_PAGE_LOAD_TIMEOUT: 30 Exception: File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap self._bootstrap_inner() File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/local/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker work_item.run() File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job retval = job.func(*job.args, **job.kwargs) File "/srv/ibeam/src/gateway_client.py", line 159, in _maintenance success, shutdown, status = self.start_and_authenticate(request_retries=var.REQUEST_RETRIES) File "/srv/ibeam/src/gateway_client.py", line 128, in start_and_authenticate success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries) File "/srv/ibeam/src/strategy_handler.py", line 84, in try_authenticating return self._authentication_strategy_B(status, request_retries) File "/srv/ibeam/src/strategy_handler.py", line 139, in _authentication_strategy_B return self._log_in(status) File "/srv/ibeam/src/strategy_handler.py", line 150, in _log_in success, shutdown = self.log_in_function() File "/srv/ibeam/src/gateway_client.py", line 106, in _log_in return log_in(driver_path=self.driver_path, File "/srv/ibeam/src/authenticate.py", line 257, in log_in driver.get(base_url + var.ROUTE_AUTH) File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 449, in get self.execute(Command.GET, {"url": url}) File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute self.error_handler.check_response(response) File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response raise exception_class(message, screen, stacktrace) Message: timeout: Timed out receiving message from renderer: -0.004 (Session info: headless chrome=113.0.5672.126) Stacktrace: #0 0x55d647ef9e43 #1 0x55d647c31908 #2 0x55d647c1caec #3 0x55d647c1c87b #4 0x55d647c1b234 #5 0x55d647c1b850 #6 0x55d647c2d9f3 #7 0x55d647c2e982 #8 0x55d647c3bddf #9 0x55d647c3fc91 #10 0x55d647c1be27 #11 0x55d647c3ba94 #12 0x55d647cad55b #13 0x55d647c95773 #14 0x55d647c626de #15 0x55d647c63a72 #16 0x55d647eca7d2 #17 0x55d647eccf7b #18 0x55d647ecc9f6 #19 0x55d647ecd6f5 #20 0x55d647ed347b #21 0x55d647ecdaa6 #22 0x55d647eab074 #23 0x55d647ee6b75 #24 0x55d647ee6d1e #25 0x55d647ef456f #26 0x7f2e6cfb9ea7 start_thread 2023-07-11 09:14:07,146|I| Cleaning up the resources. Display: | Driver: 2023-07-11 09:14:07,196|I| Logging in failed 2023-07-11 09:14:15,779|I| Maintenance 2023-07-11 09:14:16,411|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:15:15,779|I| Maintenance 2023-07-11 09:15:16,428|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:16:15,779|I| Maintenance 2023-07-11 09:16:16,407|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 09:17:15,779|I| Maintenance 2023-07-11 09:17:16,408|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040

Logs Set 2

2023-07-11 15:24:16,280|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:25:15,779|I| Maintenance 2023-07-11 15:25:16,126|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:26:15,779|I| Maintenance 2023-07-11 15:26:16,597|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:27:15,779|I| Maintenance 2023-07-11 15:27:16,160|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:28:15,779|I| Maintenance 2023-07-11 15:28:16,168|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:29:15,779|I| Maintenance 2023-07-11 15:29:16,234|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:30:15,779|I| Maintenance 2023-07-11 15:30:16,166|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:31:15,779|I| Maintenance 2023-07-11 15:31:16,655|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:32:15,779|I| Maintenance 2023-07-11 15:32:16,242|E| Unrecognised HTTPError Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 119, in _request response = self.url_request(url, method=method) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/ibeam/src/http_handler.py", line 104, in url_request return urllib.request.urlopen(req, context=self.ssl_context, timeout=self.request_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error return self._call_chain(*args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 139, in _request raise RuntimeError('Unrecognised HTTPError') from e RuntimeError: Unrecognised HTTPError 2023-07-11 15:32:16,244|I| Attempt number 2 2023-07-11 15:32:17,128|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:33:15,779|I| Maintenance 2023-07-11 15:33:16,221|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:34:15,779|I| Maintenance 2023-07-11 15:34:16,118|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:35:15,779|I| Maintenance 2023-07-11 15:35:16,614|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:36:15,779|I| Maintenance 2023-07-11 15:36:16,138|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:37:15,779|I| Maintenance 2023-07-11 15:37:16,254|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:38:15,779|I| Maintenance 2023-07-11 15:38:16,237|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:39:15,779|I| Maintenance 2023-07-11 15:39:16,679|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:40:15,779|I| Maintenance 2023-07-11 15:40:16,466|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:41:15,779|I| Maintenance 2023-07-11 15:41:16,303|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:42:15,779|I| Maintenance 2023-07-11 15:42:16,155|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:43:15,779|I| Maintenance 2023-07-11 15:43:16,148|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:44:15,779|I| Maintenance 2023-07-11 15:44:16,876|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:45:15,779|I| Maintenance 2023-07-11 15:45:16,623|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:46:15,779|I| Maintenance 2023-07-11 15:46:16,111|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:47:15,779|I| Maintenance 2023-07-11 15:47:16,773|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:48:15,779|I| Maintenance 2023-07-11 15:48:16,108|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:49:15,779|I| Maintenance 2023-07-11 15:49:16,610|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:50:15,779|I| Maintenance 2023-07-11 15:50:16,585|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:51:15,779|I| Maintenance 2023-07-11 15:51:16,147|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:52:15,779|I| Maintenance 2023-07-11 15:52:16,170|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:53:15,779|I| Maintenance 2023-07-11 15:53:16,363|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:54:15,779|I| Maintenance 2023-07-11 15:54:16,268|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:55:15,779|I| Maintenance 2023-07-11 15:55:16,203|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:56:15,779|I| Maintenance 2023-07-11 15:56:16,142|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:57:15,779|I| Maintenance 2023-07-11 15:57:16,186|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:58:15,779|I| Maintenance 2023-07-11 15:58:16,644|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 15:59:15,779|I| Maintenance 2023-07-11 15:59:16,612|I| Gateway running and authenticated, session id: c1567fc89b630802a333c9d207c887fb, server name: JifH16040 2023-07-11 16:00:15,779|I| Maintenance 2023-07-11 16:00:16,890|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:00:16,891|I| Attempt number 2 2023-07-11 16:00:17,993|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:00:17,994|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2023-07-11 16:00:17,994|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:00:17,995|I| Authentication strategy: "B" 2023-07-11 16:00:17,995|I| No active sessions, logging in... 2023-07-11 16:00:17,995|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-11 16:00:30,845|I| Gateway auth webpage loaded 2023-07-11 16:00:30,846|I| Login attempt number 1 2023-07-11 16:00:36,178|I| Submitting the form 2023-07-11 16:00:39,755|I| Webpage displayed "Client login succeeds" 2023-07-11 16:00:41,756|I| Cleaning up the resources. Display: | Driver: 2023-07-11 16:00:41,804|I| Logging in succeeded 2023-07-11 16:00:42,793|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:00:42,793|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:01:15,779|I| Maintenance 2023-07-11 16:01:16,735|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:01:16,735|I| Attempt number 2 2023-07-11 16:01:17,649|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:01:17,649|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2023-07-11 16:01:17,649|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:01:17,649|I| Authentication strategy: "B" 2023-07-11 16:01:17,650|I| No active sessions, logging in... 2023-07-11 16:01:17,650|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-11 16:01:28,367|I| Gateway auth webpage loaded 2023-07-11 16:01:28,368|I| Login attempt number 1 2023-07-11 16:01:33,708|I| Submitting the form 2023-07-11 16:01:38,203|I| Webpage displayed "Client login succeeds" 2023-07-11 16:01:40,204|I| Cleaning up the resources. Display: | Driver: 2023-07-11 16:01:40,261|I| Logging in succeeded 2023-07-11 16:01:41,397|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:01:41,397|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:02:15,779|I| Maintenance 2023-07-11 16:02:17,022|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:02:17,023|I| Attempt number 2 2023-07-11 16:02:18,136|E| IBKR server error: "HTTP Error 500: Internal Server Error". One of reasons for this error is IBKR server restart. 2023-07-11 16:02:18,137|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2023-07-11 16:02:18,137|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:02:18,137|I| Authentication strategy: "B" 2023-07-11 16:02:18,137|I| No active sessions, logging in... 2023-07-11 16:02:18,137|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-11 16:02:29,306|I| Gateway auth webpage loaded 2023-07-11 16:02:29,306|I| Login attempt number 1 2023-07-11 16:02:34,654|I| Submitting the form 2023-07-11 16:02:39,279|I| Webpage displayed "Client login succeeds" 2023-07-11 16:02:41,279|I| Cleaning up the resources. Display: | Driver: 2023-07-11 16:02:41,337|I| Logging in succeeded 2023-07-11 16:02:43,769|I| Repeating status check attempts another 14 times 2023-07-11 16:03:05,637|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable 2023-07-11 16:03:05,638|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='071fcf57fc4426d9e9bc3865596a1179', server_name=None, server_version=None, expires=582710) 2023-07-11 16:03:05,645|I| Repeated reauthentication attempt number 2 2023-07-11 16:03:07,037|I| Repeating status check attempts another 14 times Execution of job "GatewayClient._maintenance (trigger: interval[0:01:00], next run at: 2023-07-11 16:03:15 UTC)" skipped: maximum number of running instances reached (1) 2023-07-11 16:03:27,624|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable 2023-07-11 16:03:27,624|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='071fcf57fc4426d9e9bc3865596a1179', server_name=None, server_version=None, expires=560694) 2023-07-11 16:03:27,630|I| Repeated reauthentication attempt number 3 2023-07-11 16:03:29,003|I| Repeating status check attempts another 14 times 2023-07-11 16:03:49,279|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable 2023-07-11 16:03:49,280|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='071fcf57fc4426d9e9bc3865596a1179', server_name=None, server_version=None, expires=539038) 2023-07-11 16:03:49,286|I| Repeated reauthentication attempt number 4 2023-07-11 16:03:50,648|I| Repeating status check attempts another 14 times 2023-07-11 16:04:11,367|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable 2023-07-11 16:04:11,367|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='071fcf57fc4426d9e9bc3865596a1179', server_name=None, server_version=None, expires=516957) 2023-07-11 16:04:11,374|I| Repeated reauthentication attempt number 5 2023-07-11 16:04:12,745|I| Repeating status check attempts another 14 times Execution of job "GatewayClient._maintenance (trigger: interval[0:01:00], next run at: 2023-07-11 16:04:15 UTC)" skipped: maximum number of running instances reached (1) 2023-07-11 16:04:32,928|I| Max status check retries reached after 15 attempts. Consider increasing the retries by setting IBEAM_MAX_STATUS_CHECK_ATTEMPTS environment variable 2023-07-11 16:04:32,928|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='071fcf57fc4426d9e9bc3865596a1179', server_name=None, server_version=None, expires=495400) 2023-07-11 16:04:32,929|I| Max reauthenticate retries reached after 5 attempts. Consider increasing the retries by setting IBEAM_MAX_REAUTHENTICATE_RETRIES environment variable 2023-07-11 16:04:32,929|E| Repeatedly reauthenticating failed 5 times. Killing the Gateway and restarting the authentication process. 2023-07-11 16:05:15,779|I| Maintenance 2023-07-11 16:05:15,780|I| Gateway not found, starting new one... 2023-07-11 16:05:15,780|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command. 2023-07-11 16:05:15,781|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml'] running runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* config file : root/conf.yaml 2023-07-11 16:05:15,789|I| Gateway started with pids: [497] 2023-07-11 16:05:15,791|I| Cannot ping Gateway. Retrying for another 20 seconds WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2023-07-11 16:05:16,793|I| Cannot ping Gateway. Retrying for another 19 seconds -> mount demo on /demo Java Version: 11.0.18 **************************************************** version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400 **************************************************** This is the Client Portal Gateway for any issues, please contact api@ibkr.com and include a copy of your logs **************************************************** https://www.interactivebrokers.com/api/doc.html **************************************************** Open https://localhost:5000 to login App demo is available after you login under: https://localhost:5000/demo#/ 2023-07-11 16:05:20,137|I| Gateway connection established 2023-07-11 16:05:21,093|I| Attempt number 2 2023-07-11 16:05:21,601|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2023-07-11 16:05:21,601|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2023-07-11 16:05:21,602|I| Authentication strategy: "B" 2023-07-11 16:05:21,602|I| No active sessions, logging in... 2023-07-11 16:05:21,602|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-11 16:05:32,120|I| Gateway auth webpage loaded 2023-07-11 16:05:32,120|I| Login attempt number 1 2023-07-11 16:05:37,457|I| Submitting the form 2023-07-11 16:05:40,586|I| Webpage displayed "Client login succeeds" 2023-07-11 16:05:42,587|I| Cleaning up the resources. Display: | Driver: 2023-07-11 16:05:42,638|I| Logging in succeeded 2023-07-11 16:05:44,000|I| Repeating status check attempts another 14 times 2023-07-11 16:05:49,864|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='8250ac4a22156fcaccb2091ae0b094d7', server_name=None, server_version=None, expires=591582) 2023-07-11 16:05:49,865|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: None 2023-07-11 16:06:15,779|I| Maintenance 2023-07-11 16:06:16,162|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: JifH03089 2023-07-11 16:07:15,779|I| Maintenance 2023-07-11 16:07:16,691|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: JifH03089 2023-07-11 16:08:15,779|I| Maintenance 2023-07-11 16:08:16,145|E| Unrecognised HTTPError Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 119, in _request response = self.url_request(url, method=method) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/ibeam/src/http_handler.py", line 104, in url_request return urllib.request.urlopen(req, context=self.ssl_context, timeout=self.request_timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 525, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 634, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 563, in error return self._call_chain(*args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/local/lib/python3.11/urllib/request.py", line 643, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 503: Service Unavailable The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/srv/ibeam/src/http_handler.py", line 139, in _request raise RuntimeError('Unrecognised HTTPError') from e RuntimeError: Unrecognised HTTPError 2023-07-11 16:08:16,147|I| Attempt number 2 2023-07-11 16:08:16,971|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: JifH03089 2023-07-11 16:09:15,779|I| Maintenance 2023-07-11 16:09:16,219|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: JifH03089 2023-07-11 16:10:15,779|I| Maintenance 2023-07-11 16:10:16,141|I| Gateway running and authenticated, session id: 8250ac4a22156fcaccb2091ae0b094d7, server name: JifH03089 2023-07-11 16:11:15,779|I| Maintenance

@vlpraveen thanks for these logs. I believe the issues you're seeing are caused by IBKR server restarts. The 503 and 500 errors seem to indicate that. If you think these could have a different cause, I'd suggest you create a support ticket with IBKR and speak to them about this.

Regarding the authentication loop, please see: https://github.com/Voyz/ibeam/issues/152#issuecomment-1636123686

Hey friends, I've found time to do a proper revamp and handle the remaining tasks on my roadmap. This involved:

reworking the authenticate.py. Now it is broken down to step-by-step functions, and hopefully has a much better readability and maintainability. It's functionality has been moved to login_handler.py
removing reading environment variables throughout the app. Now all env vars are read in ibeam_starter.py and then everything is passed as arguments
decoupling things from GatewayClient even more. Now we have a bunch of handlers that are responsible for different things
expanding customisability for different versions of website found. Each env var that specifies the DOM element looked for, requires a [TYPE]@@[IDENTIFIER] format, eg.: CSS_SELECTOR@@.btn.btn-lg.btn-primary for the primary submit button. I concluded that such system is necessary when I realised that the 2FA forms have a very different layout for different users. Now it can be customised even more.

Breaking changes:

If you used custom env vars for DOM elements: due to the change in the env var DOM elements processing, you'll need to specify the [TYPE]@@[IDENTIFIER]. This will most likely mean you'll need to add [TYPE]@@ to your existing env vars that specify custom DOM elements if you had any.

I'm pretty happy with the results. Most recent version voyz/ibeam:0.5.0-rc3 is out, give it a shot when you've got a minute.

Strategy B with 0.5.0-rc3 worked on the first try! 🎉

ibeam | 2023-07-16 00:45:31,558|I| Gateway running and authenticated, session id: 96696cdfb672bb6a8617777c6eb6f27e, server name: JifN16024

I have tried 0.5.0-rc3 and it works perfectly. Thanks! It should be the main strategy. With the default strategy from the stable version I had issues that 70% of the time I have being disconnected and reauthenticated.

This thread helped me a lot! I am running the 0.5.0-rc4, with auth strategy B and everything runs smoothly. Plus, a quick thank you for creating ibeam. Simplified a lot of things for me!

IBKR/Gateway runs into returning a 503 Service Unavailable error while placing the order: /iserver/account/{accountId}/orders

All other API endpoints work without any errors or issues. Facing the error only for Order API

Did anyone else face this issue ?

@vlpraveen there's been a separate issue for 503 Service Unavailable: https://github.com/Voyz/ibeam/issues/38. Also, it is being mentioned in other issues too. Seeing that this is only one endpoint, I'd address this with IBKR

Hey guys, 0.5.0 just got released as the latest 🥳 I decided to set strategy B as the default, as there seems to be good indication that it performs better than A.

You can see full release notes here: https://github.com/Voyz/ibeam/releases/tag/v0.5.0

Big thanks to you for helping test this version out and reporting back 👏👏👏

Hey @Voyz et al. I've been trying to understand how to execute the login flow successfully and stumbled over this issue in search for answers to some of the unexpected behaviour I'm seeing. Would you say the following maps to your understanding & implementation of a successful login flow? login-flow

@Voyz I had wanted to share that I'm occasionally seeing 503 when POSTing WhatIf orders. Your analysis in the previous issue thread was spot on and a simple log into my paper trading account to raise the compete flag works to get my automation out of its death spiral of failed message processing.

I've set a monitor to let me know if my message processing is failing from 503s so it's something that I'm catching but I do think that there's some issue on IB's side with user sessions that seems to be causing the gateway to start failing with order placing.

I'm using the latest Docker image in Azure containers. I have seen strategy 'B' in my logs so I assume I've ended up using that.

@rbjorklin thanks for submitting your findings 👏 I'd suggest you start a different issue where we could discuss this

@demansou

Your analysis in the previous issue thread was spot on and a simple log into my paper trading account to raise the compete flag works to get my automation out of its death spiral of failed message processing.

Hold up hold up! Thanks for the credit, but I don't remember mentioning this and this is a great idea. Does doing so get you out of the authentication loop?

Do I understand your solution correctly:

Identify being in an authentication loop
Log in in a separate instance to create a session conflict and have the compete flag on.
Original instance sees the conflicting session and tries to solve it, breaking the loop

Could you comment more on this?

@demansou could you please elaborate when you find a minute?

Yes sorry I've been on vacation.

Ibeam Gateway automates login and session maintenance.
I run my trading automation to get market snapshot and POST 'what if' order to get my margin requirements.
Order should be placed and some response given from server. But HTTP 503 is received.
I rerun automation to double check and get proper responses for each endpoint hit until I place 'what if' order again. HTTP 503
I found the information somewhere (thought it was you) to log into IB manually which will raise the compete flag in my session, so I do that.
I rerun automation and no 503 issue when placing 'what if' order.

@demansou ahh gotcha you're referring to 503 not the auth loop. I just gave it a shot and indeed it seemed to have helped. I'll try it on the reauth loop too, let's see if it works. Thanks for sharing! 👏👏

Update: That also fixes timeout errors. Amazing! I'm gonna test it out on the authentication loop the next time I spot it. Fingers crossed 🤞

Voyz / ibeam

v0.5.0 - development #147

Major changes:

New strategy 'B' logic:

Context:

The logic is described below:

If you want to test this version:

Minor changes:

There has been some tidying up of the `gateway_client.py` file:

And minor fixes:

Roadmap:

Voyz / ibeam

v0.5.0 - development #147

Major changes:

New strategy 'B' logic:

Context:

The logic is described below:

If you want to test this version:

Minor changes:

There has been some tidying up of the gateway_client.py file:

And minor fixes:

Roadmap:

There has been some tidying up of the `gateway_client.py` file: