Voyz / ibeam

IBeam is an authentication and maintenance tool used for the Interactive Brokers Client Portal Web API Gateway.
Apache License 2.0
558 stars 112 forks source link

ValueError in urllib3 when authenticating #153

Closed bellerofonte closed 1 year ago

bellerofonte commented 1 year ago

Describe the bug Authentication fails with error:

File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/util/timeout.py", line 156, in _validate_timeout
    raise ValueError(
ValueError: Timeout value connect was <object object at 0x7f84696d49c0>, but it must be an int, float or None.
Full output ``` 2023-07-14 10:56:27,418|I| ############ Starting IBeam version 0.4.6 ############ 2023-07-14 10:56:27,420|I| Secrets source: env 2023-07-14 10:56:27,421|I| Health server started at port=5001 2023-07-14 10:56:27,421|I| Environment variable configuration: {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '.', 'GATEWAY_DIR': '/home/pgm/clientportal.gw', 'CHROME_DRIVER_PATH': '/opt/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'IBEAM_HEALTH_SERVER_PORT': 5001, 'GATEWAY_BASE_URL': 'http://localhost:5123', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'password', 'SUBMIT_EL': 'button.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'ibkey-promo-skip', 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_NOTIFICATION_EL': 'login-step-notification', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key'} 2023-07-14 10:56:27,450|I| Gateway not found, starting new one... 2023-07-14 10:56:27,450|I| Note that the Gateway log below may display "Open https://localhost:5000 to login" - ignore this command. 2023-07-14 10:56:27,450|I| Starting Gateway as Linux process with params: ['bash', 'bin/run.sh', 'root/conf.yaml'] running runtime path : root:dist/ibgroup.web.core.iblink.router.clientportal.gw.jar:build/lib/runtime/* config file : root/conf.yaml WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/home/pgm/clientportal.gw/build/lib/runtime/netty-common-4.1.15.Final.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release -> mount demo on /demo Java Version: 11.0.19 **************************************************** version: a27ed42161ad96c53e715ca5c5e3e3fa4cff5262 Mon, 24 Apr 2023 15:41:53 -0400 **************************************************** This is the Client Portal Gateway for any issues, please contact api@ibkr.com and include a copy of your logs **************************************************** https://www.interactivebrokers.com/api/doc.html **************************************************** Open http://localhost:5123 to login App demo is available after you login under: http://localhost:5123/demo#/ 2023-07-14 10:56:28,489|I| Gateway started with pid: 3397449 2023-07-14 10:56:29,180|I| Gateway connection established 2023-07-14 10:56:29,429|I| No active sessions, logging in... 2023-07-14 10:56:29,430|I| Loading auth webpage at http://localhost:5123/sso/Login?forwardTo=22&RL=1&ip2loc=on 2023-07-14 10:56:30,497|E| Error encountered during authentication Traceback (most recent call last): File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/ibeam/src/authenticate.py", line 253, in authenticate_gateway driver = start_driver(base_url, driver_path) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/ibeam/src/authenticate.py", line 467, in start_driver driver = new_chrome_driver(driver_path) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/ibeam/src/authenticate.py", line 66, in new_chrome_driver driver = webdriver.Chrome(driver_path, options=options) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/chrome/webdriver.py", line 76, in __init__ RemoteWebDriver.__init__( File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__ self.start_session(capabilities, browser_profile) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute response = self.command_executor.execute(driver_command, params) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/remote/remote_connection.py", line 374, in execute return self._request(command_info[0], url, body=data) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/selenium/webdriver/remote/remote_connection.py", line 397, in _request resp = self._conn.request(method, url, body=body, headers=headers) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/_request_methods.py", line 118, in request return self.request_encode_body( File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/_request_methods.py", line 217, in request_encode_body return self.urlopen(method, url, **extra_kw) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/poolmanager.py", line 432, in urlopen conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/poolmanager.py", line 303, in connection_from_host return self.connection_from_context(request_context) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/poolmanager.py", line 328, in connection_from_context return self.connection_from_pool_key(pool_key, request_context=request_context) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/poolmanager.py", line 351, in connection_from_pool_key pool = self._new_pool(scheme, host, port, request_context=request_context) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/poolmanager.py", line 265, in _new_pool return pool_cls(host, port, **request_context) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 196, in __init__ timeout = Timeout.from_float(timeout) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/util/timeout.py", line 190, in from_float return Timeout(read=timeout, connect=timeout) File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/util/timeout.py", line 119, in __init__ self._connect = self._validate_timeout(connect, "connect") File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/urllib3/util/timeout.py", line 156, in _validate_timeout raise ValueError( ValueError: Timeout value connect was , but it must be an int, float or None. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/pgm/.conda/envs/ibkr/lib/python3.10/site-packages/ibeam/src/authenticate.py", line 447, in authenticate_gateway raise RuntimeError('Error encountered during authentication') from e RuntimeError: Error encountered during authentication 2023-07-14 10:56:30,500|I| Cleaning up the resources. Display: | Driver: None 2023-07-14 10:56:30,546|I| Authentication process failed 2023-07-14 10:56:30,550|I| Starting maintenance with interval 60 seconds ```

To Reproduce Steps to reproduce the behavior:

  1. Download and unpack IB clientportal gateway
  2. Create new conda (miniforge) env, python=3.10
  3. install IBeam and chromedriver
  4. run ibeam_starter.py
  5. See error

Expected behavior Success authentication

Environment IBeam version: 0.4.6 Docker image or standalone: standalone Python version (standalone users only): 3.10.12 OS: Ubuntu 22.04.2 LTS

Additional context urllib3 version: 2.0.3 selenium version: 3.141.0

Suggest a Fix ibeam requires selenium version 3.* but it seems that issue is fixed in version 4.9.0 ⁉️ possibly outdated setup.py

Voyz commented 1 year ago

Hey @bellerofonte thanks for sharing your issue in detail πŸ‘

This seems to be related to selenium/chrome/chromedriver versions. Indeed, thanks for bringing the setup.py to my attention, it is outdated. The requirements.txt has the updated packages.

You can see more information about this issue under #140, particularly: https://github.com/Voyz/ibeam/issues/140#issuecomment-1559518308

Let me know if updating the versions helps solving the issue

bellerofonte commented 1 year ago

Hi @Voyz! Thank you for your effort! After some tries I gave up trying to run standalone version due to different issues related to the dependencies. However, I successfully launched IB GW using docker image 0.5.0-rc2:

docker run \
    --env IBEAM_ACCOUNT=****** \
    --env IBEAM_PASSWORD=****** \
    --env IBEAM_AUTHENTICATION_STRATEGY=B \
    -p 5000:5000 voyz/ibeam:0.5.0-rc2
Voyz commented 1 year ago

happy to hear you got it working otherwise @bellerofonte πŸ‘ if you'd like to, feel free to share your troubles with the standalone version.

bellerofonte commented 1 year ago

Hi @Voyz ! Sorry for a late response.

Here is the issue I am facing to when I run ibeam standalone:

2023-07-20 12:19:26,201|I| Gateway started with pid: 3604566
2023-07-20 12:19:26,967|I| Gateway connection established
2023-07-20 12:19:27,240|I| No active sessions, logging in...
2023-07-20 12:19:27,241|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2023-07-20 12:19:28,433|E| Error encountered during authentication
Traceback (most recent call last):
  File "/home/pgm/Projects/ibeam/ibeam/src/authenticate.py", line 253, in authenticate_gateway
    driver = start_driver(base_url, driver_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pgm/Projects/ibeam/ibeam/src/authenticate.py", line 479, in start_driver
    raise e
  File "/home/pgm/Projects/ibeam/ibeam/src/authenticate.py", line 467, in start_driver
    driver = new_chrome_driver(driver_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pgm/Projects/ibeam/ibeam/src/authenticate.py", line 66, in new_chrome_driver
    driver = webdriver.Chrome(driver_path, options=options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 84, in __init__
    super().__init__(
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 104, in __init__
    super().__init__(
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
    self.error_handler.check_response(response)
  File "/home/pgm/.conda/envs/ibkr/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary

Chrome driver is installed at /opt/chromedriver

I run ibeam via following command:

IBEAM_GATEWAY_DIR=/home/pgm/clientportal.gw \
IBEAM_CHROME_DRIVER_PATH=/opt/chromedriver \
IBEAM_OUTPUTS_DIR=. \
IBEAM_ACCOUNT=****** \
IBEAM_PASSWORD=****** \
IBEAM_AUTHENTICATION_STRATEGY=B \
python3 ibeam_starter.py

I have also tried IBEAM_CHROME_DRIVER_PATH=/opt/ but no luck either. I have tried to download lates release zip and to checkout the repository - same error in both cases. May be I have to install some package via apt?

Voyz commented 1 year ago

Thanks πŸ‘

  1. What's the path to your chrome?
  2. What's the chrome version?
  3. What's the chromedriver version?

See this: https://stackoverflow.com/questions/50138615/webdriverexception-unknown-error-cannot-find-chrome-binary-error-with-selenium

Do you have /usr/bin/google-chrome present?

bellerofonte commented 1 year ago

πŸ€¦β€β™‚οΈ

I've just installed chrome via latest stable .deb... and it works!

Thank you!

Voyz commented 1 year ago

Way to go! Congrats @bellerofonte πŸ‘ So the problem was that you didn't have chrome installed or that it was outdated?

bellerofonte commented 1 year ago

yep, I didn't have chrome installed

Voyz commented 1 year ago
Cool, glad that was the problem, and well done figuring it out! πŸ‘
Voyz commented 1 year ago

Hey @bellerofonte feel free to close this issue now πŸ‘