IBeam not aware of lost connection?

[xiongnuren]

I use IBKR's web client portal API for trading. On average I loose the connection to the gateway 1-3 times everyday when live trading. My program checks the connection every minute using the tickle endpoint. So I know when the connect is lost and I had to restart the gateway and authenticate again before I can resume my trading.

So far when testing IBeam, I had several encounters of loosing connection to the gateway. However, on each of this occurrence it appeared to me that IBeam seems not aware of the lost connection. With the DEBUG option on, it continues to print out the 'Maintenance' and 'HTTP request to: https://localhost:5000/v1/portal/sso/validate' message every 60 seconds.

If I manually stop the IBeam and restart again, my script can communicate with IBKR again.

Let me know what test/command I can run to demonstrate the issue.

[Voyz]

Thanks for submitting this issue 👍

From what you wrote I'm understanding that it seems that your client isn't able to connect with the Gateway (ie. make requests) a few times a day, which you solve by restarting and reauthenticating the Gateway. It also seems that when that happens, IBeam continues making validate requests to the Gateway which seem succeed, therefore IBeam doesn't restart or reauthenticate the Gateway. If this is right, it leads me to hypothesise these there theories:

• IBeam fails to recognise that Gateway is not working correctly, or
• validate is not the correct route to verify the Gateway and tickle should be used instead, or
• The Gateway is working correctly (hence IBeam's requests succeed), but your client is unable to connect to it.

To narrow down the list of suspects, could I ask you to run a simple validate and tickle requests when the connectivity issue occurs? For instance:

Using Python:

empty_context = ssl.SSLContext()
r1 = urllib.request.urlopen("https://localhost:5000/v1/api/tickle", context=empty_context)
r2 = urllib.request.urlopen("https://localhost:5000/v1/portal/sso/validate", context=empty_context)
print(r1.read(), r2.read())

or

Using cURL:

curl -X GET "https://localhost:5000/v1/api/tickle" -k
curl -X GET "https://localhost:5000/v1/portal/sso/validate" -k

Then I'd assume the following:

• If both requests fail, then indeed IBeam is not recognising an issue with the Gateway.
• If validate succeeds and tickle fails, then using validate to verify the Gateway may not be the right choice.
• If both succeed, then I'd guess there may be an issue in you client code. I'm happy to help you debug it if that's the case.

Thanks!

[xiongnuren]

Thanks for the suggestions. Make sense to me. Today IB lost connection to me once, but IBeam also noticed the issue and was able to reauthenticate! I will keep running my test and if the discrepancy occurs again I will follow your suggestions and will report back what I found.

[xiongnuren]

OK, so I had one more discrepancy happened to me again. When my script lost connection and IBeam still seem connected, I test IBKR with three different endpoint, using my own python code. I don't have ssl installed, I'm using urllib3, so I'm not able to directly run your example code but my test code serves the purpose anyway.

• using the /tickle endpoint, the response is: {'collission': False, 'iserver': {'authStatus': {'authenticated': False, 'competing': False, 'connected': False}, 'tickle': True}, 'ssoExpires': 599999, }

• using the /sso/validate endpoint, the response is: {'AUTH_TIME': 1604894890842, 'CREDENTIAL_TYPE': 0, 'EXPIRES': 1604898154703, 'IS_FREE_TRIAL': False, 'IS_PENDING_APPLICANT': False, 'LOGIN_TYPE': 1, 'RESULT': True, 'SFT': '5.2a', 'SF_CONFIG': '5.2a', 'SF_ENABLED': True, 'features': {'bond': True, 'calendar': True, 'env': 'PROD', 'newMf': True, 'optionChains': True, 'realtime': True, 'wlms': True}, 'isGw': False, 'took': 7}

• using the /iserver/auth/status endpoint, the response is: {'MAC': ****, 'authenticated': False, 'competing': False, 'connected': False}

[Voyz]

That's great, thank you! From this I'd conclude the following:

• Gateway is still running, it hasn't stopped - otherwise the requests wouldn't complete.
• Gateway lost its authentication (ie. it logged out).
• Checking for whether validate request completes is not the valid way of verifying the authentication - which is what IBeam currently does.

Let me know if you have any other thoughts based on your results.

The authenticated flag from either the /iserver/auth/status or /tickle routes may be the solution here.

[Voyz]

@seismars I just created a PR #3 with fixes for this and #1 issue. I figured out that the authenticate: false situation occurs when IBeam authenticates once successfully but then you use the same login elsewhere (mobile app, website, TWS, etc.). This way the Gateway still contains an active session, yet its authentication is lost. Do you remember if you might have logged in somewhere else when you these authentication issues occurred?

Either way, now IBeam will attempt to reclaim the authentication during maintenance. I also ended up using the tickle endpoint for extending the session and checking the authentication status. I'm not sure if this is the right way, as IBKR's docs contradict themselves, yet according to their OpenAPI specification tickle should be suffice. I've submitted a support ticket to verify this.

Additionally, I introduced request timeout and retries settings, as well as updated the documentation regarding the 2FA as requested in #1.

I created and pushed a new docker image voyz/ibeam:0.2.0-rc1 containing these changes. Whenever you find a while, could I ask you to pull that image and test it to see if the issues you were experiencing are gone?

docker pull voyz/ibeam:0.2.0-rc1

Thanks for your help! 🙌

[xiongnuren]

Thanks for the update. I'm testing.

FYI, I did not use other IB sessions while testing, because I know when I login to another session my current session will be affected. I have been using IB's Client Portal API/gateway for several months now. Based on my own experience it is not stable. I lost connection/authentication about 1-3 times each day while trading.

[Voyz]

Thanks for the follow up! Good to know that these drops in authentication were not due to your activity - something certainly worth keeping in mind. Let me know if you'd spot some pattern indicating what may be causing these drops so I could potentially look for a workaround in IBeam - thanks in advance!

Btw. I got a reply from IBKR regarding tickle vs validate endpoints:

Both tickle and sso/validate endpoints will extend the session live time. The tickle endpoint send ping signal to server side to notify the session still alive. The sso/validate will send request to server for the authentication status check. Customer can choose either way to keep the session alive based on their convenience.

I'm going to stick with tickle for extending the session then and deprecate the support for validate.

Looking forward to seeing if the solution helped 🤞

[xiongnuren]

Today is Friday Nov 13, 2020. I run my trading with IBeam's latest version. Now IBeam is aware when IB's connection is lost. Also on issue #1: IBeam encountered 9 times of read timeout, all went through in its second attempt. If I was using the previous version I would have to authorize 9 more times. Big thumbs up!

[Voyz]

Fantastic! 😄 Thank you for verifying and coming back with the good news, I'm glad this helped. Hope you enjoy using IBeam!😊 And thanks for helping me nail down these two issues!