2FA with IBKey

[lunarflint]

Hi everyone, I would like to propose yet another (better?) 2FA approach. I am not quite sure on what people is seeing when asked for 2FA because my account has IBKey enabled from day 1. So allow me to walk you through the procedure.

1. After keying in username and password, a short text message is displayed below the textboxes: (vvv screenshot vvv) (^^^ screenshot ^^^)

2. At this moment your mobile device with IBKey (it is a part in mobile TWS now) should pop up and ask for a pin input (provided by you at the time you enable IBKey on your phone). I am not sure about how this is done (assuming the TWS mobile app intercepting notifications).

3. [OPTION 1] If you key in the correct pin in your mobile device, you should be logged in.

[OPTION 2] For any reasons that you cannot login with the method described above (sometime the pop up may not show or show after few minutes), you can alternatively choose to complete the 2FA with the challenge-response protocol. (vvv screenshot vvv) (^^^ screenshot ^^^)

4. (continue option 2) Now you can open up the IBKey app and navigate to a page like this: (vvv screenshot vvv) (^^^ screenshot ^^^) So you fill in the fields, click "generate response", get back to the web page and fill in the challenge-response. Login done!

The interesting part of [OPTION 2] is that IB Key does not need to communicate with IB servers at all and in fact it works even when the phone has no network (well, you need internet at the moment you set it up on your phone).

And therefore....... I encapsulated the whole thing into a HTTP server

I hooked up the android emulator(the one bundled with Android SDK) with Appium (an automation suite), and wrote an simple nodejs HTTP server doing the challenge-response process.

While I am still cleaning up the code I wrote before I can (shamelessly) share it publicly, I would like to see how can this be integrated to IBeam. Is anyone interested on this idea? Any directions for me?

Ideally I would like to package the whole thing into a docker image, exposing only one http endpoint. However I cannot do it easily on a window machine (you know, xvfb + android emulator + docker on window...). Let see what I can achieve in this weekend.

In conclusion, I think that the IBKey challenge-response way of 2FA is the most robust one in term of automation. It does not require any kind of messaging to any servers, 6-digit code in, 8-digit code out. Let me know how you think :)

[Voyz]

Hey @lunarflint welcome to IBeam! Thanks a mil for suggesting such a great way to handle 2FA, I'm really impressed with all that working out! It's very cool you decided to share your results with us 😊 Let me address various points in your proposal:

I am not quite sure on what people is seeing when asked for 2FA because my account has IBKey enabled from day

Just to clarify this - various users get to choose different options. I have no idea who sees what, but I'm sure there is a number of ways 2FA could be done.

[OPTION 2] For any reasons that you cannot login with the method described above (sometime the pop up may not show or show after few minutes), you can alternatively choose to complete the 2FA with the challenge-response protocol.

Can you clarify where do you make that choice? Is it on the mobile device or is it in the login page of the gateway?

I hooked up the android emulator(the one bundled with Android SDK) with Appium (an automation suite)

Super cool! What's the performance of this system? I remember Android emulators taking forever to launch (up to several minutes) on my dev machine, which on a low-end cloud instance could be even longer if we decided to use this system in production.

And therefore....... I encapsulated the whole thing into a HTTP server , and wrote an simple nodejs HTTP server doing the challenge-response process. I'd probably recommend we shift that HTTP server from Node.js to Python to make it easier to maintain for IBeam contributors, but unless I'm missing something obvious this should be a relatively trivial task. Happy to help out if you aren't familiar with serving HTTP with Python.

While I am still cleaning up the code I wrote before I can (shamelessly) share it publicly,

Haha I know the feeling 😄 Don't worry, put a draft up for us to review when you're ready, promise we won't be judgemental!

Ideally I would like to package the whole thing into a docker image, exposing only one http endpoint

Fantastic idea. Then we could docker-compose IBeam and this Android emulator server on one virtual instance, having them communicate in a closed network.

However I cannot do it easily on a window machine (you know, xvfb + android emulator + docker on window...).

Okay, could you clarify what's the problem here? I'm successfully developing with docker on my windows machine, using xvfb along with other things.

In conclusion, I think that the IBKey challenge-response way of 2FA is the most robust one in term of automation. It does not require any kind of messaging to any servers, 6-digit code in, 8-digit code out. Let me know how you think :) Is anyone interested on this idea? Any directions for me?

Great contribution! Let's figure out some of these details along the way, but I'm very happy you proposed this solution 👏

Thanks! 👋 Voy

[lunarflint]

Thanks for the warm welcome Voy,

While I am still busying with other stuff, let me quickly reply on your comments :)

Can you clarify where do you make that choice? Is it on the mobile device or is it in the login page of the gateway?

I really forgot the exact procedure of enabling IBKey. I see there is a "Add User" button on the IBKey page. I assume that It is something on top of the "normal" 2FA and users can opt-in for facilitating the 2FA process. I am not really sure whether you can deactivate it after enabling, so please be carefully if you want to try it. And.... here is the procedure for invoking IBKey, on the Mobile TWS app, you can see a "SERVICE" (yes, all CAP) button in the app menu (or top left hand corner right on the login page). Click on that, and click again on "Authenticate" or something similar.

Super cool! What's the performance of this system? I remember Android emulators taking forever to launch (up to several minutes) on my dev machine

Now, you can try out the x86_64 images. They are reasonably fast on a "normal" local dev machine. The ARM images are very slow.

which on a low-end cloud instance could be even longer if we decided to use this system in production.

Actually thanks for remaining me on this. I almost forget my develop machine is close to a super computer (compare to instances on cloud). Today, I just did some test on Linode and DigitalOcean, and here are the results:

For Linode, I cannot get hardware acceleration (Intel vmx or AMD svm) working on their instances. This means that I cannot use x86_64 android images and force to use the ARM images. The performance is VERY BAD and 100% cpu the whole time. I did not test this out on the top-tiers of instances. Well, it will not be financially viable even if that works. (And I can get a real device from AWS device farm with the same price anyway.)

For DigitalOcean, their instances support hardware acceleration. I have managed to boot a virtual device to its home screen on a $18(/month) intel instance.....and it crashed. As I am a new customer to them, I need to prepaid for unlocking those CPU intensive instances (priced at around $40 for the cheapest one), still talking with their customer support on this. If it works well on a $40 instance with a reasonable low CPU loading, maybe actually this is a viable choice.

Suddenly this becomes the most critical blocker. Wow...I did not expect this.

I am leaving some notes here and see if someone is interested. (I may have missed / mistyped one or two lines):

apt update
apt upgrade -y
apt install -y unzip default-jre libpulse0 libxcursor1 xvfb x11vnc

# update /etc/environment with the following
# PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/android/cmdline-tools/tools/bin:/opt/android/emulator"
# JAVA_HOME='/usr/lib/jvm/java-11-openjdk-amd64'
# ANDROID_SDK_ROOT=/opt/android
# ANDROID_SDK_HOME=/opt/android
# ANDROID_HOME=/opt/android
# ANDROID_AVD_HOME=/opt/android/avd

source /etc/environemnt

mkdir -p $ANDROID_SDK_ROOT/{cmdline-tools,platforms,platform-tools,avd}
# Believe it or not, the above line is very important. You will encounter errors if you continue without some of the dummy folders

(cd $ANDROID_SDK_ROOT/cmdline-tools && wget 'https://dl.google.com/android/repository/commandlinetools-linux-6858069_latest.zip' -O tools.zip && unzip -d tools tools.zip)

yes | sdkmanager --install 'system-images;android-30;google_apis;x86_64'
echo no | avdmanager --verbose create avd --force --name a30 --package 'system-images;android-30;google_apis;x86_64' --tag google_apis --abi x86_64

Xvfb :0 -ac -screen 0 1280x720x24 &
x11vnc -forever -display :0 -rfbport 12345 &
DISPLAY=:0 emulator @a30 -no-boot-anim -netdelay none -no-snapshot -skin 768x1280 -gpu off -qemu -m 1024

An alternative is to have a real phone plugged into a RaspberryPI and put this setup at home. The only issue here is the network connectivity. I mean, maybe less reliable comparing to a proper server room?

I'd probably recommend we shift that HTTP server from Node.js to Python to make it easier to maintain for IBeam contributors, but unless I'm missing something obvious this should be a relatively trivial task. Happy to help out if you aren't familiar with serving HTTP with Python.

Haha, actually I don't have any experience in making a http server with Python. I saw something like WSGI with nginx many years ago, but never dive into that. For a super hacky prove of concept program, I just went for nodejs without much thought. But anyway, my feeling is that it is best to make it standalone service for maximizing the reusability. You know, there are people still using TWS with IBC. Therefore I guess packaging the whole thing as a docker image is the way to go. Let me know how you feel :) From what I read in pull request #12, I guess the best steps to integrated my idea to IBeam are as follows:

1. Recognize (Selenium magic!) the clickable link (see the first screenshot of my original post) and get the challenge text (6 digit code).
2. Wire this information to (yet another?) custom 2fa handling function.
3. The custom 2fa handling function should returns a challenge response (the 8 digit code) or throws error when something went wrong.
4. Automatically fill in the textboxes with the challenge response and click submit
5. Ideally we can provide a default implementation to this custom handler. The implementation should be a simple http request. Did I miss anything? Please comment on this approach.

Okay, could you clarify what's the problem here? I'm successfully developing with docker on my windows machine, using xvfb along with other things.

Oh actually this is the same problem as the cloud instances. You cannot easily run VM inside VM. The machines need some special setup. For my case, running Docker on Windows is consuming one layer of virtualization already and I cannot get Android emulator (the x86_64 one) working.

On a side note, since the challenge-response can be done completely offline, theoretically one can revers- (cough cough), I mean mimic the behavior of that and complete ditch the whole emulator madness... But wow...

[Voyz]

hey @lunarflint thanks for all that info! It's all very valuable, and thanks for sharing that setup code. Addressing some of your points:

I just did some test on Linode and DigitalOcean,

Roughly how long it took for the whole login process to complete on these two setups?

Suddenly this becomes the most critical blocker.

Snap, yeah emulation is indeed quite an expensive task isn't it. Having near-supercomputers at home doesn't help either, I've been there! 😄

An alternative is to have a real phone plugged into a RaspberryPI and put this setup at home. The only issue here is the network connectivity. I mean, maybe less reliable comparing to a proper server room?

Yupp, I discussed with some people before - but like you say a fully cloud-based solution is going to be less error prone in the long run.

Haha, actually I don't have any experience in making a http server with Python. I saw something like WSGI with nginx many years ago, but never dive into that. For a super hacky prove of concept program, I just went for nodejs without much thought.

No worries, it's piss easy in both node.js and Python to put a simple server like that up. I totally understand why you went with the tech you're familiar with 👍

But anyway, my feeling is that it is best to make it standalone service for maximizing the reusability.

Definitely a good idea 👍

You know, there are people still using TWS with IBC

True! Only thing is that it would be harder to input the creds to TWS, as opposed to the Gateway Login page.

Therefore I guess packaging the whole thing as a docker image is the way to go. Let me know how you feel :)

Absolutely yes!

1. Recognize (Selenium magic!) the clickable link (see the first screenshot of my original post) and get the challenge text (6 digit code).

Yeah we can totally do that.

2. Wire this information to (yet another?) custom 2fa handling function.

Wouldn't the existing ExternalRequestTwoFaHandler suffice?

3., 4., 5.,

all make sense, we're already doing that in #12 👍

You cannot easily run VM inside VM.

Oh shit, yeah. I wouldn't even know how to bite this without researching into it. This could indeed be a roadblock. How do you see this happening?

(cough cough)

😄

While this would make the life of all users a ton easier, I wouldn't know how to do this, and it is unadvisable that you attempt. You definitely shouldn't try to do that. You definitely shouldn't. It would make everyone's life easier, but one should definitely not attempt to do so.

[lunarflint]

Ok, it has been a while. Let me put some updates. After some testing and setting up the IBKey from multiple emulators / phones back and forth, I seems triggered some alarms and IB froze my 2FA and I spent some time with their customer support and finally get my 2FA back working.

Long story short, I think that automating a real phone is the safest approach. Setting it up in an eumlator should also work. (But don't even try to mess with it. Disconnect it from internet.) And don't even try to mess with their app (like recompiling it with debugger-enabled) unless you want to get yourself into troubles.

I cleaned up a little bit the NodeJs code for the server but I am really busy these days on other stuff. You can check the code here.

Not sure when could I give the next update. But this is all I have for now.

[Voyz]

Hey @lunarflint thanks for an update 👍 Did they froze you for using an emulator? Is that somewhat against their TOC?

It sucks big time they put these restrictions on you - and nice of you to let us know of these issues so we know what to avoid.

Thanks for cleaning up and sharing your node code - that's helpful 👍

[Voyz]

Hey @lunarflint I'm going to close this issue due to lack of activity - let me know if you'd like to reopen it at some point 👍

[julianwachholz]

Would it be possible to use this with IBKey and just validate the prompt on my phone that I use normally? I tried setting the IBEAM_PAGE_LOAD_TIMEOUT to 3 minutes to give IB enough time to trigger the phone notification but it always fails authentication immediately. There is no need to enter a code in the gateway for me I think. I am getting the normal IBKey notifications triggered by IBeam.

[Voyz]

Hi @julianwachholz thanks for contributing and outlining your issue 👋

Could I ask you to screen record your authentication process using the Gateway without IBeam? I'm trying to understand how does this flow work for you, so I could think of how we could solve it. Please make sure to cover your credentials in the video.

[julianwachholz]

Thanks for the warm welcome @Voyz

1. I downloaded the Gateway ZIP from IBKR and started it in PowerShell.
2. I open https://localhost:5000 to this page:

3. This is the screen after entering credentials which prompts me to check my phone:

4. On my phone I get a notification:

5. I open the notification and confirm with my fingerprint:

6. The Gateway automatically redirects to "Client login succeeds":

From this point on I can access the Client API endpoints normally (e.g. with Postman).

[Voyz]

Great, thank you for posting all that process here @julianwachholz that's very useful to see 😊 Could I also ask you to post what output IBeam produces during this form of authentication? You can find info on how to acquire debug logs here: https://github.com/Voyz/ibeam/wiki/IBeam-Configuration#debugging

Just as a side note - IBeam is a tool that is to facilitate keeping the CP Web Gateway authenticated without the input from the user. To perform 2FA in the way you're suggesting - by completing the prompt manually on your phone - goes against this principle and is a setup that I think carries several risks. I'd suggest you look for other ways of 2FA authentication, however I'm happy to help you fix this method too 👍

[julianwachholz]

I know IBeam tries to have the Gateway open without user input. I am currently looking for a way to access my portfolio data in a read-only way but so far I haven't found anything?

My primary worry is that if I were to log in on another IBKR service (e.g. Mobile TWS) it would logout the Gateway session?

I think I would rather disable 2FA completely before I try and emulate a second factor on the same device any way. Especially when I only want readonly access.

[Voyz]

Cool, thanks for outlining your setup in more detail 👍

My primary worry is that if I were to log in on another IBKR service (e.g. Mobile TWS) it would logout the Gateway session?

Regrettably, that's true for all of IBKR, no matter whether you use TWS, CP Web API, Gateways, Website or others. When your bot is running you should ideally not use those same credentials. In either case - should you do that IBeam will force-reacquire the token and shut down whatever new session you've opened elsewhere. I remember there was a way to create various logins under same IBKR account - so maybe try talking to their support about it?

I think I would rather disable 2FA completely before I try and emulate a second factor on the same device any way. Especially when I only want readonly access.

I'm not sure whether readonly access would change this, but from what I heard we (very regrettably) cannot change or disable the 2FA easily. Let me know if that works out for you - I'd be very interested to hear if you manage to do it.

In either case, like I said I'm happy to try fixing your current setup, so feel free to post these logs whenever you've got some time 👍

[henkisdabro]

I know IBeam tries to have the Gateway open without user input. I am currently looking for a way to access my portfolio data in a read-only way but so far I haven't found anything?

My primary worry is that if I were to log in on another IBKR service (e.g. Mobile TWS) it would logout the Gateway session?

I think I would rather disable 2FA completely before I try and emulate a second factor on the same device any way. Especially when I only want readonly access.

Just pitching in to this conversation. I see the exact same login flow as @julianwachholz highlighted above in this thread and I share his wish to just build a simple read-only api endpoint to be able to pull my portfolio/positions into google sheets.

I logged in to IB to disable the 2-factor auth and got a stark warning that "you are on your own if you do so", so I backed off going through with it.

That said, not sure if @julianwachholz tested this yet, but we are allowed to create additional users inside IB inside the Client Portal, so you can then at least get a second set of user credentials that you can use for iBeam, without (hopefully) causing logoffs on your mobile or vice versa. Perhaps that is something to try, personally I'm not able to help out much with python programming and such I'm afraid, wish I could!