search

Voyz / ibeam

IBeam is an authentication and maintenance tool used for the Interactive Brokers Client Portal Web API Gateway.
Apache License 2.0
571 stars 113 forks source link

IBeam cloud deployment becomes unresponsive after a while #214

Open DzananGanic opened 2 months ago

DzananGanic commented 2 months ago

Describe the bug I followed the instructions for cloud deployment and I hosted an IBeam instance on Digital Ocean and it works well. However, after some time, the gateway becomes unresponsive and I start getting 'connection refused'. When I log into my digital ocean droplet, check out logs and it suddenly starts working again.

Could it be that inactivity is making the IBeam instance sleep or something similar?

To Reproduce Steps to reproduce the behavior:

  1. Deployed to digital ocean
  2. Wait for ~24h
  3. Start getting connection refused
  4. Log in to Digital Ocean droplet, check the logs
  5. Starts working again

Expected behavior Gateway working without interruptions

Environment Docker image or standalone: voyz/ibeam:latest OS: Ubuntu 22.04 (DO Droplet)

Additional context When I checked docker logs I've got some errors that might be causing the issue. I did not touch anything after starting the gateway and this is a couple of last lines from the logs:


2024-09-09 14:29:51,974|I| Repeating status check attempts another 119 times
2024-09-09 14:29:57,607|I| AUTHENTICATED Status(running=True, session=True, connected=True, authenticated=True, competing=False, collision=False, session_id='becb77db54bea954724abb69163752a8', server_name=None, server_version=None, expires=593340)
2024-09-09 14:29:57,729|I| Gateway running and authenticated, session id: becb77db54bea954724abb69163752a8, server name: None
2024-09-09 14:30:50,608|I| Maintenance
2024-09-09 14:30:50,708|I| NOT CONNECTED Status(running=True, session=True, connected=False, authenticated=False, competing=False, collision=False, session_id='becb77db54bea954724abb69163752a8', server_name='JifN13096', server_version='Build 10.30.1l, Aug 27, 2024 6:01:26 PM', expires=540245)
2024-09-09 14:30:50,708|I| Authentication strategy: "B"
2024-09-09 14:30:50,709|I| Competing or disconnected Gateway session found, logging out and reauthenticating...
2024-09-09 14:30:50,808|I| Gateway logout successful
2024-09-09 14:30:51,336|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2024-09-09 14:30:51,337|E| Repeatedly reauthenticating failed 3 times. Killing the Gateway and restarting the authentication process.
2024-09-09 14:31:50,608|I| Maintenance
2024-09-09 14:31:50,609|I| Gateway not found, starting new one...
2024-09-09 14:31:50,609|I| Note that the Gateway log below may display "Open https://localhost:[PORT] to login" - ignore this command.
2024-09-09 14:31:50,609|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml']
running  
 runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/*
 config file  : root/conf.yaml
2024-09-09 14:31:50,641|I| Gateway started with pids: [17063]
2024-09-09 14:31:50,643|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused>
2024-09-09 14:31:50,644|I| Gateway connection established
2024-09-09 14:31:50,648|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused>
2024-09-09 14:31:50,649|I| Attempt number 2
2024-09-09 14:31:50,650|I| Gateway running but not serving yet. Consider increasing IBEAM_GATEWAY_STARTUP timeout. Error: <urlopen error [Errno 111] Connection refused>
2024-09-09 14:31:50,651|I| Max request retries reached after 2 attempts. Consider increasing the retries by setting IBEAM_REQUEST_RETRIES environment variable
2024-09-09 14:31:50,652|I| NO SESSION Status(running=True, session=False, connected=False, authenticated=False, competing=False, collision=False, session_id=None, server_name=None, server_version=None, expires=None)
2024-09-09 14:31:50,653|I| Authentication strategy: "B"
2024-09-09 14:31:50,653|I| No active sessions, logging in...
2024-09-09 14:31:50,654|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2024-09-09 14:31:52,769|E| Error encountered during authentication 
Exception:
  File "/usr/local/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/srv/ibeam/src/gateway_client.py", line 115, in _maintenance
    success, shutdown, status = self.start_and_authenticate(request_retries=self.request_retries)
  File "/srv/ibeam/src/gateway_client.py", line 62, in start_and_authenticate
    success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating
    return self._authentication_strategy_B(status, request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B
    return self._log_in(status)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in
    success, shutdown = self.login_handler.login()
  File "/srv/ibeam/src/handlers/login_handler.py", line 432, in login
    wait_and_identify_trigger = self.load_page(targets, driver, self.base_url, self.route_auth)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/ibeam/src/handlers/login_handler.py", line 395, in load_page
    driver.get(base_url + route_auth)
  File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 353, in get
    self.execute(Command.GET, {"url": url})
  File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 344, in execute
    self.error_handler.check_response(response)
  File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)

  <class 'selenium.common.exceptions.WebDriverException'> Message: unknown error: net::ERR_CONNECTION_REFUSED
  (Session info: chrome-headless-shell=120.0.6099.224)
Stacktrace:
#0 0x55c9563a5233 <unknown>
#1 0x55c95606f0fe <unknown>
#2 0x55c956067c66 <unknown>
#3 0x55c956059c05 <unknown>
#4 0x55c95605b660 <unknown>
#5 0x55c95605a12e <unknown>
#6 0x55c956058fd7 <unknown>
#7 0x55c956058e6c <unknown>
#8 0x55c956057943 <unknown>
#9 0x55c956057d94 <unknown>
#10 0x55c956072662 <unknown>
#11 0x55c956103c56 <unknown>
#12 0x55c9560e6422 <unknown>
#13 0x55c956103680 <unknown>
#14 0x55c9560e61a3 <unknown>
#15 0x55c9560ab13e <unknown>
#16 0x55c9560ac712 <unknown>
#17 0x55c95637a534 <unknown>
#18 0x55c95637d759 <unknown>
#19 0x55c95637d238 <unknown>
#20 0x55c95637dc05 <unknown>
#21 0x55c95636cabf <unknown>
#22 0x55c95637df8e <unknown>
#23 0x55c956356c86 <unknown>
#24 0x55c9563963e5 <unknown>
#25 0x55c9563965cb <unknown>
#26 0x55c9563a47bf <unknown>
#27 0x7f41d68c3ea7 start_thread

2024-09-09 14:31:52,771|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f2799943490> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="6281d435e5cdd34fa06c8d2f9506fbb1")>
2024-09-09 14:31:53,222|I| Logging in failed
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/srv/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
 -> mount demo on /demo
Java Version: 11.0.24
****************************************************
version: a27ed42161ad96c53e715ca5c5e3e3fa4cff5262 Mon, 24 Apr 2023 15:41:53 -0400
****************************************************
This is the Client Portal Gateway
for any issues, please contact api@ibkr.com
and include a copy of your logs
****************************************************
https://www.interactivebrokers.com/api/doc.html
****************************************************
Open https://localhost:5000 to login
App demo is available after you login under: https://localhost:5000/demo#/
``

**Suggest a Fix**
The only thing that comes to my mind is going idle because of inactivity, but I'm really not sure what could be causing this.
Voyz commented 2 months ago

Hey @DzananGanic thanks for trying out IBeam and for the detailed description of the issue 👍

after some time, the gateway becomes unresponsive and I start getting 'connection refused'. When I log into my digital ocean droplet, check out logs and it suddenly starts working again. Could it be that inactivity is making the IBeam instance sleep or something similar?

IBeam doesn't have any sleep mode. It should continue working without any difference.

It would be unreasonable to assume that DO puts the droplet in some kind of sleep mode, but to be fair - maybe it's worth addressing with their support?

I've looked through your logs, and tried searching for more info on the "ERR_CONNECTION_REFUSED" however to no avail. It seems to be some connectivity error between the droplet and the server, however I'm unable to find out a concrete reason behind it.