RE-authentication loop

[exallon72]

I got 15-20 authorization requests without the ibeam stopping. It has restart: "no" set in the docker-compose file. So I don't know why it's restarting the process over and over.

2024-07-12 16:23:40,712|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f4869cdb210> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="99140a4b5d238a6d39b31dc7498d69a6")> 2024-07-12 16:23:40,747|I| Logging in failed 2024-07-12 16:23:53,351|I| Maintenance 2024-07-12 16:24:03,489|E| IBKR service unavailable: "HTTP Error 503: Service Unavailable". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready. 2024-07-12 16:24:03,490|I| Attempt number 2 2024-07-12 16:24:13,568|E| IBKR service unavailable: "HTTP Error 503: Service Unavailable". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready. 2024-07-12 16:24:13,569|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2024-07-12 16:24:13,569|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2024-07-12 16:24:13,570|I| Authentication strategy: "B" 2024-07-12 16:24:13,570|I| No active sessions, logging in... 2024-07-12 16:24:13,570|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2024-07-12 16:24:20,043|I| Gateway auth webpage loaded 2024-07-12 16:24:20,043|I| Login attempt number 1 2024-07-12 16:24:25,190|I| Submitting the form 2024-07-12 16:24:40,313|E| Timeout reached when waiting for authentication. The website seems to not be loaded correctly. Consider increasing IBEAM_PAGE_LOAD_TIMEOUT. Website URL: https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on

Exception: File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap self._bootstrap_inner() File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/local/lib/python3.11/threading.py", line 975, in run self._target(*self._args, self._kwargs) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker work_item.run() File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job retval = job.func(job.args, job.kwargs) File "/srv/ibeam/src/gateway_client.py", line 115, in _maintenance success, shutdown, status = self.start_and_authenticate(request_retries=self.request_retries) File "/srv/ibeam/src/gateway_client.py", line 62, in start_and_authenticate success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries) File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating return self._authentication_strategy_B(status, request_retries) File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B return self._log_in(status) File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in success, shutdown = self.login_handler.login() File "/srv/ibeam/src/handlers/login_handler.py", line 466, in login self.handle_timeout_exception(e, targets, driver, website_version, self.route_auth, self.base_url, self.outputs_dir) File "/srv/ibeam/src/handlers/login_handler.py", line 447, in login self.attempt(targets, wait_and_identify_trigger, driver) 2024-07-12 16:24:40,313|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f4869cc0ad0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="9d3684ab675ebda8c203e04e871d08d2")> File "/srv/ibeam/src/handlers/login_handler.py", line 371, in attempt trigger, target = self.step_login(targets, wait_and_identify_trigger, driver, self.secrets_handler.account, self.secrets_handler.password, self.secrets_handler.key, self.presubmit_buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/ibeam/src/handlers/login_handler.py", line 188, in step_login trigger, target = wait_and_identify_trigger( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/ibeam/src/handlers/login_handler.py", line 67, in _wait_and_identify_trigger trigger = WebDriverWait(driver, timeout).until(any_of(*expected_conditions)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until raise TimeoutException(message, screen, stacktrace)

<class 'selenium.common.exceptions.TimeoutException'> Message:

2024-07-12 16:24:40,340|I| Logging in failed 2024-07-12 16:24:53,351|I| Maintenance 2024-07-12 16:25:03,428|E| IBKR service unavailable: "HTTP Error 503: Service Unavailable". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready. 2024-07-12 16:25:03,428|I| Attempt number 2 2024-07-12 16:25:13,510|E| IBKR service unavailable: "HTTP Error 503: Service Unavailable". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready. 2024-07-12 16:25:13,511|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable 2024-07-12 16:25:13,511|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None) 2024-07-12 16:25:13,511|I| Authentication strategy: "B" 2024-07-12 16:25:13,511|I| No active sessions, logging in... 2024-07-12 16:25:13,512|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on 2024-07-12 16:25:20,086|I| Gateway auth webpage loaded 2024-07-12 16:25:20,086|I| Login attempt number 1 2024-07-12 16:25:25,227|I| Submitting the form 2024-07-12 16:25:38,292|I| Webpage displayed "Client login succeeds" 2024-07-12 16:25:39,292|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f4869bf12d0> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="b80a945da88a64e3dc76176fe54a01d5")> 2024-07-12 16:25:39,324|I| Logging in succeeded 2024-07-12 16:25:49,419|E| IBKR service unavailable: "HTTP Error 503: Service Unavailable". It seems IBKR servers are not ready to handle requests. We may need to wait until the servers are ready.

[exallon72]

I am getting a distinct feeling that it's an intentional block from IB. This seems to happen during night time. If we get into problems during night time and continue posting queries all the time during the downtime it seems like the block will happen.

I am using iBind so I think with some 429/500 error handling in there this could be at least somewhat mitigated.

[Voyz]

hey @exallon72 I'm afraid this is a long-standing issue on IBKR side. Please see the WiKi for more info: https://github.com/Voyz/ibeam/wiki/Troubleshooting#authentication-loop

[exallon72]

When I look at this again. It seems like the main issue here is if the client get disconnected IBKR seems to send a not recongnized page by the login mechanism.

If I stop ibeam and start again it will login as usual.

[Voyz]

Can you think of a solution we could implement to help solve this? IBeam has a built in Gateway restart if it fails to authenticate after several attempts, but I understand that this fails to help your case?