Gateway auto restart

[automiamo]

Hello, running ibeam on my AWS cloud (linux) pointing to my real account and it's fine for automated trading, but receiving 2FA on my smartphone every morning. I know Interactive Brokers gives the possibility to set auto-restart in global configuration on TWS and IB Gateway. I installed IB Gateway on my local Windows PC and i see global configuration and auto-restart option. Using ibeam on linux, instead, where and how can I access to IB gateway global configuration? I know and access only to conf.yaml config file. I read on broker documentation that auto-restart enabled requires to me only Sunday to do 2FA access, that would be better than now... Thanks

[automiamo]

I try to follow this guide to solve my 2FA issue that requires to me tapping my phone every day: https://github.com/Voyz/ibeam/wiki/Two-Factor-Authentication#google-messages-handler

Setting the 2 mentioned variables and they are readable with this simple python line code

import os TWO_FA_HANDLER = os.environ.get('IBEAM_TWO_FA_HANDLER', None) print(TWO_FA_HANDLER)

But they are not visible running ibeam, e.g TWO_FA_HANDLER is None

Logs

2023-07-09 09:56:12,722|I| ############ Starting IBeam version 0.4.6 ############ 2023-07-09 09:56:12,730|I| Custom conf.yaml found and will be used by the Gateway 2023-07-09 09:56:12,746|I| Secrets source: env 2023-07-09 09:56:12,758|I| Health server started at port=5001 2023-07-09 09:56:12,758|I| Environment variable configuration: {'INPUTS_DIR': '/srv/inputs/', 'OUTPUTS_DIR': '/srv/outputs', 'GATEWAY_DIR': '/srv/clientportal.gw', 'CHROME_DRIVER_PATH': '/usr/bin/chromedriver', 'GATEWAY_STARTUP': 20, 'GATEWAY_PROCESS_MATCH': 'ibgroup.web.core.clientportal.gw.GatewayStart', 'MAINTENANCE_INTERVAL': 60, 'SPAWN_NEW_PROCESSES': False, 'LOG_LEVEL': 'INFO', 'LOG_TO_FILE': True, 'LOG_FORMAT': '%(asctime)s|%(levelname)-.1s| %(message)s', 'REQUEST_RETRIES': 1, 'REQUEST_TIMEOUT': 15, 'RESTART_FAILED_SESSIONS': True, 'RESTART_WAIT': 15, 'REAUTHENTICATE_WAIT': 15, 'IBEAM_HEALTH_SERVER_PORT': 5001, 'GATEWAY_BASE_URL': 'https://localhost:5000', 'ROUTE_AUTH': '/sso/Login?forwardTo=22&RL=1&ip2loc=on', 'ROUTE_USER': '/v1/api/one/user', 'ROUTE_VALIDATE': '/v1/portal/sso/validate', 'ROUTE_REAUTHENTICATE': '/v1/portal/iserver/reauthenticate?force=true', 'ROUTE_AUTH_STATUS': '/v1/api/iserver/auth/status', 'ROUTE_TICKLE': '/v1/api/tickle', 'ROUTE_LOGOUT': '/v1/api/logout', 'USER_NAME_EL': None, 'PASSWORD_EL': 'password', 'SUBMIT_EL': 'button.btn.btn-lg.btn-primary', 'ERROR_EL': None, 'SUCCESS_EL_TEXT': 'Client login succeeds', 'OAUTH_TIMEOUT': 15, 'PAGE_LOAD_TIMEOUT': 15, 'ERROR_SCREENSHOTS': False, 'MAX_FAILED_AUTH': 5, 'MIN_PRESUBMIT_BUFFER': 5, 'MAX_PRESUBMIT_BUFFER': 30, 'MAX_IMMEDIATE_ATTEMPTS': 10, 'IBKEY_PROMO_EL_CLASS': 'ibkey-promo-skip', 'TWO_FA_EL_ID': 'twofactbase', 'TWO_FA_NOTIFICATION_EL': 'login-step-notification', 'TWO_FA_INPUT_EL_ID': 'chlginput', 'TWO_FA_HANDLER': None, 'STRICT_TWO_FA_CODE': True, 'TWO_FA_SELECT_EL_ID': 'sf_select', 'TWO_FA_SELECT_TARGET': 'IB Key'}

And in the logs I am not seeing the QR Code URL like this, mentioned in the guide:

Web messages is not authenticated. Open this URL to pair web messages with your android phone: http://api.qrserver.com/v1/create-qr-code/?color=000000&bgcolor=FFFFFF&qzone=1&margin=0&size=400x400&ecc=L&data=https%3A//support.google.com/messages/%3Fp%3Dweb_computer%23%3Fc%3DCj0AqX3tUQSryndVWbP6rfpNx26g0EvyR0fHTrw/MsDjcU0ORObLOBE22gJE1lxMBuHzJdhRG21voGwPwZaXEiD6Kd8NMqpjXmGHVOOrhGWc5rM7AosbrbF5jGJCyYizeBog%2B8zUHGyh%2BHOW%2BBsAf8CGrY3tC2xacO9b0O2x58LfVLA%3D

[Voyz]

hey @automiamo thanks for describing your issue in detail 👍

I know Interactive Brokers gives the possibility to set auto-restart in global configuration on TWS and IB Gateway. I installed IB Gateway on my local Windows PC and i see global configuration and auto-restart option.

Sorry, I'm not sure what auto-restart functionality you're talking about. Please bear in mind that IBeam works with the CP Web API Gateway, not TWS Gateway. 'IB Gateway' you're describing is the TWS Gateway, the functionality you're describing may not be present.

Using ibeam on linux, instead, where and how can I access to IB gateway global configuration?

If you mean the conf.yaml configuration file - which is the main way to configure the CP Web API Gateway - then you use the Inputs Directory: https://github.com/Voyz/ibeam/wiki/Inputs-And-Outputs

I read on broker documentation that auto-restart enabled requires to me only Sunday to do 2FA access, that would be better than now...

Can you link to that documentation? As far as I know, the docs state that the connection is reset once a day, not only on Sundays.

[automiamo]

Can you link to that documentation? As far as I know, the docs state that the connection is reset once a day, not only on Sundays.

Hello, here the link: _https://www.interactivebrokers.com/en/general/tws_notes_974.php#:~:text=Both%20TWS%20and%20the%20IB,once%2Fweek%2C%20each%20Sunday.

Set TWS/IB Gateway to Auto Restart Both TWS and the IB Gateway require daily restarts to refresh data. As a convenience, we now offer an auto-restart feature that will restart your application throughout the week and only require a manual restart and authentication once/week, each Sunday. TWS users will find this feature works most effectively with the TWS Offline version._

TWS and IB Gateway standalone jar file have this option, that's useful but don't know this could be set up in ibeam

[Voyz]

@automiamo right, this is the unfortunate naming that IBKR chose to make.

You're talking about the TWS Gateway: https://www.interactivebrokers.com/en/trading/ibgateway-stable.php

IBeam works with CP Web API Gateway: https://interactivebrokers.github.io/cpwebapi/

Some features from TWS Gateway are unavailable on the CP Web API Gateway. I've never heard of the auto restart, but feel free to look through the docs and see if it's there.

[automiamo]

Hello Voyz, i will write to IB support to dive into possibility to have auto restart on CP. thanks

[automiamo]

IBKR replied to me. The answer seems to be a good news but we have to wait.... Meanwhile I will apply for OAuth.

Thank you for reaching out to us.

As this time, only institutional clients are able to apply for OAuth access, which allows for uninterrupted CP API connectivity, and for OAuth users, the API connection will only be disconnected once a week, at 1:00 AM EST on Sunday For individual CP API users, the API connection will be re-connected each 2 hours.

And the OAuth is not supported by now, we are in the process of developing it. You can apply for the OAuth or the third party API first. Once it is released, you can obtain the service that can keep alive for 7 days. We apologize for any inconvenience this may cause and appreciate your understanding.

If you have any further questions or concerns, please do not hesitate to contact us.

[Voyz]

Great, thanks for reporting back @automiamo 👍 Does OAuth provide the auto restart you were hoping for?

[automiamo]

I hope doing automate trading with REST in the way as I developed on other brokers/exchanges:

1. webhook signal triggered from tradingview to my remote web server
2. Login to the broker with the token provided
3. POST request
4. Logout

[automiamo]

Adding one more bit. Tradingview may be able to use IBKR Oauth being istitutional company and never requires to me 2FA all days, even using TWS, session with tradingview is disconnected as usual, but then i can reconnect on IB with tradingview without 2FA (probably this is due to the not-expiring token managed by Tradingview). Only if manually logged out I have to manually re-login with 2FA. IBKR earnings are down, they need retail traders to increase eaninrgs and we need Oauth :-)

[lapcchan]

@Voyz with v0.5.0 reauthenticate do we still require 2fa daily?

[tredondo]

Sorry, I'm not sure what auto-restart functionality you're talking about. Please bear in mind that IBeam works with the CP Web API Gateway, not TWS Gateway. 'IB Gateway' you're describing is the TWS Gateway, the functionality you're describing may not be present.

Maybe it's worth avoiding the confusion in the README with a clear note about TWS and IB Gateway. I've sent a PR, https://github.com/Voyz/ibeam/pull/158. That's something we can do.

As for the confusion on IB's side, it's a mess. Not only are two different products named "Gateway", but the new REST API can't even pick one naming convention for its fields. You get camelCase, lower_snake, UPPER_SNAKE, sometimes both in the same response object, and duplicating a field (company_name/companyName) in one of the most prominent models (contract) while at it 🤦🏻 Let alone that that contract isn't actually about the contract details; that's called secDef, when in fact a security refers broadly to a class of financial asset, while a contract has specific details, like expiration or strike.

Anyway, as it has been the case for the past 15 years, the IB API continues to be a piece of garbage.

The last link explains why it's a bad idea to try to use the REST/Websockets API instead of the old IB Gateway (yes, its API is way clunkier, but at least it works).

To add my own experience, I've read all their docs on reconnecting (/tickle, /sso/validate, tic subscription - why so many beats me), implemented all those mechanisms, and a Websockets market data subscription still stops delivering data randomly, after 30 minutes to 9 hours (9h was the record; the market data subscription stopped in the middle of the trading day though).

The IB API has been put to shame by even fly-by-night crypto exchanges, let alone the big ones like BitMEX or Kraken, which have a far, far, far better modeled and stable API, despite far less time in the business.

So I'm curious if anyone uses IBeam and the IBCP for streaming data, when you clearly miss significant chunks of ticks even if you manage to reconnect automatically somehow. In other words, can you get streaming market data continuously over a week (logging back in on Sunday is fine)?

[Voyz]

@lapcchan yes, sadly, as far as I'm aware there is no way to avoid 2FA

[Voyz]

@tredondo

Maybe it's worth avoiding the confusion in the README with a clear note about TWS and IB Gateway. I've sent a PR, #158. That's something we can do.

Good idea, thanks!

So I'm curious if anyone uses IBeam and the IBCP for streaming data, when you clearly miss significant chunks of ticks even if you manage to reconnect automatically somehow. In other words, can you get streaming market data continuously over a week (logging back in on Sunday is fine)?

This seems to depend on a user (and we haven't identified why). Some people run it successfully for months, some fail every day.

---

Also, thanks for sharing all your other thoughts and observations regarding IBKR API. There isn't much I could add here unfortunately. If you wanna add more spice to the confusion, I recommend you read the source code for the Gateway webpage that we use to authenticate. The best we can do is to pressure IBKR to fix it, which, admittedly, isn't a great prospect in itself.

[tredondo]

In the meantime I switched my data collection to IB Gateway, with IBC Alpha. I only have to authenticate with 2FA once on Sundays, and market data arrives very reliably. IB Gateway doesn't terminate the session, or deauthenticate it, or suddenly stop sending data, or anything like that.

The downside is that the API is more complicated than the Websockets API, but the reliability is worth not having to deal with the daily and random frustrations of the IB Client Portal Gateway.

[Voyz]

I'm going to close this issue due to inactivity. Thanks for your contribution and please feel free to request a reopen if you'd like to continue the discussion 👍

[isaak4]

I have an update regarding OAuth with Client Portal API for individual accounts. A few days ago, I opened a ticket with IBKR to help me set up OAuth authentication. After initial rejection of my request because I was an individual user and not an institutional one, I persisted in asking what the requirements are to be considered an institutional user.

Today I got the following promising response: From the API point of view we categorize accounts as individual - registered for personal use, and corporate accounts that are opened for registered entities. At the moment, individual accounts can only use Client Portal API via local gateway. OAuth access can only be enabled for corporate accounts. This will change in the near future, but i can not provide any ETA at the moment. Kind regards, IBKR Client Services