search

Voyz / ibeam

IBeam is an authentication and maintenance tool used for the Interactive Brokers Client Portal Web API Gateway.
Apache License 2.0
580 stars 115 forks source link

Overcome two factor authentication #8

Closed vishwajeetio closed 3 years ago

vishwajeetio commented 3 years ago

Describe the feature I am trying to access the API using a cash account but it's not working and giving the login failed message while it's working perfectly fine with a paper account. Can you tell me how can I fix that or if possible can we add that authentication feature?

Expected interaction Using a two-factor authentication device

Possible implications Only cash account authentications

Additional context Please suggest to me if there's any workaround to that

Voyz commented 3 years ago

Thanks for the suggested enhancement 😊 Am I understanding correctly you're trying to figure out how to overcome the two-factor authentication with IBeam?

vishwajeetio commented 3 years ago

yeah, I tried all the way but I am still unable to use my cash account on the container. can you suggest to me how can I use my main account with that? Also, I am willing to put effort into it to make the two-factor auth possible somehow using the IBKR app. Please help me know if there's something I missed out on.

Voyz commented 3 years ago

Right, I see. Unfortunately, IBeam does not handle 2FA automatically, therefore if this is what's stopping the authentication on your cash account then there is nothing that can currently be done by default. You'd need to complete the 2FA each time using your 2FA method or find another method to complete it automatically.

If however it isn't the 2FA then I'd suggest you check that you don't have any other places - such as TWS or web services - where you use the cash account credentials.

Could you share the IBeam output log in case there's something there that could indicate the problem? Please run it with IBEAM_LOG_LEVEL environment variable set to DEBUG.

jembl commented 3 years ago

I have solved this problem, contact me to help you out.

vishwajeetio commented 3 years ago

I have solved this problem, contact me to help you out.

that sounds great. Can you please explain to me how can I do that? thank you

Voyz commented 3 years ago

hey @jembl super cool you found a way around it! Could I ask you to share your solution here so others can use your findings in the future too? Thanks! 😊

jembl commented 3 years ago

yes, so here is how I did it:

  1. I got a www.twilio.com account and got a number, changed my IB phone number to twilio phone number.
  2. I add a webhook for SMS, when an SMS is received, it will call my webserver i.e. www.sample.com/smshook

  3. use this code to parse the SMS on the server side and upload it to a storage of my choice `public String receive(String message) { StringBuilder sb = new StringBuilder(); ResourceSet messages = Message.reader().limit(10).read();

    for(Message record : messages) {
    if (!smsStorageManager.getSmsWithSid(record.getSid()).isPresent()) {
        String messageSid = record.getSid();
        String messageBody = record.getBody();
        try {
            log.info("received message with SID " + record.getSid());
            log.info("received message with BODY " + record.getBody());
            String[] splits = messageBody.split(":");
            if (messageBody.startsWith("Your requested authentication code: ")) {
                String code = splits[1].trim();
                twoFactorCodeBlobStorageManager.insertIntoCodeBlob(code);
            }
            sb.append(messageSid + "\n");
        } catch (Exception e) {
            log.error("Failed to process the message " + messageBody, e);
        } finally {
            Message.deleter(messageSid).delete();
        }
    }
}

return sb.toString();

    }`

and the code to get SMS from storage: `def get_latest_authentication_code(): connection_string = "YAYAYAYAYYA" blob_client = BlobClient.from_connection_string( connection_string, container_name="sms", blob_name="code.text")

[START download_a_blob]

_LOGGER.info("downloading the code from blob")
download_stream = blob_client.download_blob()
code = download_stream.content_as_text()
_LOGGER.info("done downloading the code from blob ")
_LOGGER.info(str(code))
return code`

Hope this helps, ask me if you have any questions!

Voyz commented 3 years ago

That is fantastic @jembl! 👏

Thank you so much for following up with such a detailed explanation! I was hypothesising that such an approach would be doable but I haven't looked that direction yet. Super happy to see this method worked for you and seriously congrats on devising it on your own!

Hey would you be interested in contributing some of these changes to IBeam? So that conditionally IBeam could be set to download the code from a URL (or some other way) and paste it to the login page. This way IBeam's users would only need to set up the SMS client and a pipeline to store the code. I could help you get this done, although no worries if you're busy or not interested, this answer is already a massive help 😊

vishwajeetio commented 3 years ago

thank you so much @jembl I understand what you are trying to implement here and got the Twilio part working now working on adding this code to the container. I am still in doubt on exactly at what part do I need to add these lines. thank you

Voyz commented 3 years ago

Hey @odssh and @jembl 👋 I've got some good news! There's a new version of IBeam currently in testing that should facilitate implementing this pattern. In this new version you will have an opportunity to set up IBeam in a way that will attempt to pull the 2FA code from an external URL.

You can try it out by using the image with tag 0.3.0-rc1, eg:

docker pull voyz/ibeam:0.3.0-rc1

Then when running it, do the following:

Alternatively, if you require full customisation in how you acquire the code you can do the following:

Let me know if you'd have any feedback! 😊

vishwajeetio commented 3 years ago

Hi, I did test that and put it to run. It ran fine for a few hours so I put it on run for two days and I just checked on it and I got about more than a thousand messages. Something is really wrong. I am trying to identify it. Please let me know if you can suggest to me where it could break or any potential reason for that bug. thank you

Voyz commented 3 years ago

Hi @odssh thanks for testing it! Sorry to hear it started breaking after a while. Could you copy the output log over to here so I could give it a go at figuring out what's wrong? Thanks!

vishwajeetio commented 3 years ago

Hi @Voyz , Here's the log file for the whole session. It worked almost fine for the first day and then I got locked out of my account because of the excessive number of login trials as it kept making multiple login attempts each time it needed to log in. In the log file, the line below is printed by me: {"log":"Your requested authentication code: 433235\n","stream":"stdout","time":"2021-02-21T23:22:47.664576083Z"} Please let me know if there's any fix for these multiple login attempts as it causes the account to get locked out from the Interactive broker's side. thank you

Voyz commented 3 years ago

That's fantastic @odssh thanks for passing this over and for the explanation! I'm really sorry you hit the lock out from IB - it happened to me too when I was testing this new method. I think it may be a good idea to implement some safety mechanism that will stop IBeam from attempting after a number of times as to prevent the lock-out from happening.

As for the output log, I'd have a number of questions:

  1. Are you using IBeam as a Docker image, or standalone?
  2. How are you printing this line? Your requested authentication code: 433235? This must come from some custom code, yet IBeam doesn't support that other than through CustomTwoFaHandler.
  3. How is this message printed: Did not receive the authentication code? I cannot find it anywhere in IBeam. Have you modified it in some way? Could you let me see your fork if that's the case?
  4. What IBEAM_TWO_FA_HANDLER are you using? The log seems to suggest that you're using the EXTERNAL_REQUEST TwoFaHandler, however I'm surprised to see that when it prints in the log it does not contain any URL field, ie. url=None: 
    2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from:  ExternalRequestTwoFaHandler(method=GET, url=None, timeout=300, params=None, headers=None)

    If so, can you confirm that you're setting the URL correctly using IBEAM_EXTERNAL_REQUEST_URL?

  5. It seems the 2FA code acquisition you perform is not acting fast enough. In multiple places in the output log I can see that IBeam requests the code, then waits and eventually times out. Only once that happens, your custom code seems to print the 2FA code. Could you share how your 2FA acquisition code looks like just to verify it isn't the issue? (example of that behaviour):

    2021-02-21T23:22:28.233255303Z -- 2021-02-21 23:22:28,232|I| Credentials correct, but Gateway requires two-factor authentication.
2021-02-21T23:22:28.233567968Z -- 2021-02-21 23:22:28,233|I| Attempting to acquire 2FA code from: ExternalRequestTwoFaHandler(   method=GET, url=None, timeout=300, params=None, headers=None)
2021-02-21T23:22:32.279821579Z -- 2021-02-21 23:22:32,279|E| Did not receive the authentication code
2021-02-21T23:22:32.280179965Z -- 2021-02-21 23:22:32,279|I| No 2FA code returned.
2021-02-21T23:22:47.55094453Z -- 2021-02-21 23:22:47,549|E| Error encountered during authentication
2021-02-21T23:22:47.550985101Z -- Traceback (most recent call last):
2021-02-21T23:22:47.55099196Z --   File \"/srv/ibeam/src/authenticate.py\", line 164, in authenticate_gateway
2021-02-21T23:22:47.551019941Z --     trigger = WebDriverWait(driver, var.OAUTH_TIMEOUT).until(any_of(success_present,    error_displayed))
2021-02-21T23:22:47.551026047Z --   File \"/opt/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py\", line 80, in    until
2021-02-21T23:22:47.551031437Z --     raise TimeoutException(message, screen, stacktrace)
2021-02-21T23:22:47.551036576Z -- selenium.common.exceptions.TimeoutException: Message: 
2021-02-21T23:22:47.551041561Z -- 
2021-02-21T23:22:47.551046234Z -- 
2021-02-21T23:22:47.551050865Z -- The above exception was the direct cause of the following exception:
2021-02-21T23:22:47.551055927Z -- 
2021-02-21T23:22:47.551060554Z -- Traceback (most recent call last):
2021-02-21T23:22:47.551065397Z --   File \"/srv/ibeam/src/authenticate.py\", line 176, in authenticate_gateway
2021-02-21T23:22:47.551070461Z --     raise RuntimeError('Error encountered during authentication') from e
2021-02-21T23:22:47.551075534Z -- RuntimeError: Error encountered during authentication
2021-02-21T23:22:47.664339484Z -- 2021-02-21 23:22:47,663|I| Authentication process failed
2021-02-21T23:22:47.664509274Z -- Your requested authentication code: 433235

    Note how the authentication fails due to no code being returned at 23:22:32, while your message is printed at 23:22:47, over 10 seconds later. It seems that the code is received a little too late. It would be great if we could figure out why that hold happens.

If you could feed me in on as many details regarding these questions as possible, I should be able to understand what is causing the issue on your end. Appreciate your help in the matter!

vishwajeetio commented 3 years ago

thank you for your quick response @Voyz. Here's the python code that I have modified. In this file, I have changed only lines 12, 13, and 60-118. I know I should have gone with two_fa_handler.py but at the moment external_request_handler.py seemed easier. Here are the answers to your question:

  1. Docker
  2. line 116
  3. line 85
  4. EXTERNAL_REQUEST
  5. the code in the file attached checks for verification messages constantly and returns it as soon as it gets it. Right now I am not sure what causing the issue. Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied". thank you
Voyz commented 3 years ago

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials.

import json
import logging
import os
from pathlib import Path
from typing import Union

import requests

from ibeam.src.two_fa_handlers.two_fa_handler import TwoFaHandler

from datetime import datetime, timezone
from twilio.rest import Client

_LOGGER = logging.getLogger('ibeam.' + Path(__file__).stem)

_EXTERNAL_REQUEST_METHOD = os.environ.get('IBEAM_EXTERNAL_REQUEST_METHOD', 'GET')
"""Method to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_URL = os.environ.get('IBEAM_EXTERNAL_REQUEST_URL')
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_TIMEOUT = int(os.environ.get('IBEAM_EXTERNAL_REQUEST_TIMEOUT', 300))
"""URL to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_PARAMS = os.environ.get('IBEAM_EXTERNAL_REQUEST_PARAMS')
"""Params to use by the external request 2FA handler."""

_EXTERNAL_REQUEST_HEADERS = os.environ.get('IBEAM_EXTERNAL_REQUEST_HEADERS')
"""Headers to use by the external request 2FA handler."""

def parse_json(s):
    if s is None:
        return None
    try:
        return json.loads(s)
    except Exception as e:
        _LOGGER.exception(e)
        return None

class ExternalRequestTwoFaHandler(TwoFaHandler):

    def __init__(self, method: str = None,
                 url: str = None,
                 timeout: int = None,
                 params=None,
                 headers=None):

        self.method = method if method is not None else _EXTERNAL_REQUEST_METHOD
        self.url = url if url is not None else _EXTERNAL_REQUEST_URL
        self.timeout = timeout if timeout is not None else _EXTERNAL_REQUEST_TIMEOUT
        self.params = params if params is not None else parse_json(_EXTERNAL_REQUEST_PARAMS)
        self.headers = headers if headers is not None else parse_json(_EXTERNAL_REQUEST_HEADERS)

    def get_two_fa_code(self) -> Union[str, None]:
        try:
            # response = requests.request(method=self.method,
            #                             url=self.url,
            #                             timeout=self.timeout,
            #                             params=self.params,
            #                             headers=self.headers, )
            # response.raise_for_status()
            # return response.content
            return self.twilioAuthToken()

        except requests.exceptions.HTTPError as err:
            _LOGGER.error(err)
            return None

    def twilioAuthToken(self):
        vCode = None
        for i in range(20):
            vCode = self.getNewMessage()
            if vCode != None:
                break
            else:
                pass
        if vCode == None:
            _LOGGER.error("Did not receive the authentication code")
        print("got it", vCode)

        return vCode

    def getNewMessage(self):
        vcode = None
        account_sid = 'ACCOUNT_SID'
        auth_token = 'AUTH_TOKEN'
        dtf = '%Y-%m-%d %H:%M:%S'
        client = Client(account_sid, auth_token)

        messages = client.messages.list(
            limit=1,
            to='PHONE_NUMBER'
        )
        for record in messages:
            message = client.messages(record.sid).fetch()

            currentTime = datetime.now().timestamp() # current time in utc
            dts = str(message.date_created).split('+')[0]
            dtf = '%Y-%m-%d %H:%M:%S'
            dtSend = datetime.strptime(dts, dtf).replace(tzinfo = timezone.utc).timestamp() # message send time in utc

            if currentTime - dtSend <= 10:
                vcode2 = message.body
                try:
                    vcode1 = vcode2.split(':')[1]
                    vcode = int(vcode1.strip(' '))
                except:
                    vcode = None
            print(message.body)

        return vcode 

    def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"
Voyz commented 3 years ago

Now, thanks for your answers! That's all very helpful to know 😊

Also, one more thing about the docker(I am just getting started), why am I able to request localhost 5000 inside the container but not outside even though I added a -p tag with port while launching the container. It simply says "Access denied".

This is due to Gateway's config.yml not having allowed IPs set up. Have a look at Troubleshooting to learn about this issue and how to solve it - let me know if you'd have any troubles doing it and I'll be happy to help you.

Digging into the code I can recognise a few areas that may cause the issues, leading to the unexpected behaviour we're observing in your logs:

  1. Firstly, you run 20 attempts to acquire the code:

    for i in range(20):
   vCode = self.getNewMessage()

    That 20 seems like an arbitrary number. There also isn't any time delay between these attempts, so what seems to be happening is that all 20 happen at roughly the same time. The logs seem to confirm this, as 20 prints from line 116 appear within milliseconds of each other. I would suggest that you base that loop on a timeout, rather than on an arbitrary number, introducing a time delay between attempts, eg.:

    def twilioAuthToken(self):
    vCode = None
    start_time = time.time() 
    end_time = start_time + _TWILIO_TIMEOUT
    while time.time() < end_time:
        vCode = self.getNewMessage()
        if vCode != None:
            break
        time.sleep(1) # 1 second delay between requests

        _LOGGER.info(f"No code received, retrying for another {round(end_time - time.time())} seconds")

    if vCode == None:
        _LOGGER.error(f"Did not receive the authentication code after {round(time.time()-start_time)} seconds.")
    else:
        _LOGGER.info(f'Received code: {vCode}')

    return vCode

    I also increased verbosity of your logs to get more information. I suggest you set IBEAM_LOG_LEVEL env variable to 'DEBUG' in order to see even more information in your logs during this development stage.

    1. I cannot comment much on the way you're interacting with Twilio as I haven't used them - I'd double check that you're doing everything correctly there too. However, you seem to only look up the messages from the previous 10 seconds if I'm not mistaken:

      if currentTime - dtSend <= 10:

      I'd suggest to try to increase that buffer, as this may simply not be enough for a SMS message to deliver.

    2. Furthermore, I'd suggest to alter the way you parse the messages received:

      if currentTime - dtSend <= 10:
vcode2 = message.body
if 'Your requested authentication code' not in vcode2 or not vcode2[-6:].isdigit():
    _LOGGER.error(f'Unrecognised message body: {vcode2}')
else:

    try:
        vcode1 = vcode2.split(':')[1]
        vcode = int(vcode1.strip(' '))
    except Exception as e:
        _LOGGER.exception(e)
        vcode = None

      This way we can be more sure that the right messages are being parsed, and that we don't return invalid codes (eg. in your existing code a message that reads Error : Couldn't access code would return Couldn't access code as the 2FA code)

Lastly, I suggest you remove the following lines:

 def __str__(self):
        return f"ExternalRequestTwoFaHandler(method={self.method}, url={self.url}, timeout={self.timeout}, params={self.params}, headers={self.headers})"

As they are misleading, and replace them with:

 def __str__(self):
        return f"TwilioHandler()"

I hope these changes help you ensure the right code is returned. We could certainly introduce some changes on IBeam to make things more robust in case wrong code is returned too many times - I'll try to look into this soon. Let me know how it goes! 👍

vishwajeetio commented 3 years ago

Just a quick message - I have edited your last reply as the link you provided contains your authentication credentials. Below is your code with the credentials removed. I suggest you remove this gist as soon as possible as to not leak your credentials. ...

Hi @Voyz I am sorry for the confusion. Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

vishwajeetio commented 3 years ago

Thank you so much @Voyz I think it will definitely solve the issue as mow I am realizing that it might be the loop of random number 20. I will put the code on run and will let you know what I get. About the point

  1. 10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue. thank you so much for the time. It really helps a lot and will save a lot of time. I will let you know how it goes. thank you
Voyz commented 3 years ago

Actually, the credentials were not real and I just put them there as a placeholder. Also thank you for informing me about that.

Ah glad to hear that! Sorry I stepped over then! 😅

10 seconds was the time to check how old the message is so if the most recent message is older than 10 seconds then the script will keep waiting until a new message appears and that I did with a loop of 20 and I think that was causing the issue.

Right, this may be a tricky one to figure out. It would be helpful to mark messages as 'read' and 'unread' on Twilio side - have a look if this is possible. A rigid 10 seconds limit may be a cause of bugs in the future.

I will let you know how it goes.

Great, looking forward to it! 😊

Voyz commented 3 years ago

@odssh I've just released a new version -rc2, ie.: voyz/ibeam:0.3.0-rc2. It contains additional security measures, checking that the 2FA code provided is valid (ie. contains 6 digits). It can be disabled by setting IBEAM_STRICT_TWO_FA_CODE to False, and it is based on the assumption that all 2FA codes are 6-digit. Let me know if this is not the case for you. Additionally, the new version does a better job at handling cases where there is no 2FA code provided by the handler.

Looking forward to hearing how the improvements worked out for you 👍

vishwajeetio commented 3 years ago

Thank you for the detailed update @Voyz I think the verification codes are 6 digit all the time. I will definitely help me a lot. I am still working on previous fixes. I will update you on the status once i get it all running. Thank you 👍

vishwajeetio commented 3 years ago

Hi @Voyz I was running the container for the last two days and it broke yesterday. As you can see in this log file was authenticated for a long time but then all of a sudden it started authenticating every single minute and was able to authenticate but still kept doing it until it got locked out and then it still kept trying. I tried to search for the potential cause but unable to find it. Please let me know if there's any way to fix that. thank you

Voyz commented 3 years ago

hey @odssh I'm really sorry to hear about the second lock-out, that must be incredibly annoying and I'm going to try my best to help you avoid it in the future. Thanks for submitting the detailed log too. From that I can conclude the following:

These steps should move us forward towards understanding what is wrong. Once again, I'm really sorry you've run into these issues and I'm hoping we'll mange to get to the bottom of this for the benefit of all users of IBeam. Thanks for your patience in the meantime 👍

vishwajeetio commented 3 years ago

Hi @Voyz thank you for the detailed explanation. I will start testing it now and see where it takes me. I am still unable to get the port issue working with the docker. I tried the previous solution but it didn't work as it already had port 5000 allowed. I think I need to expose port 5000 from the docker file somewhere but all of my solutions failed. thank you again.

Voyz commented 3 years ago

Hi @odssh - sorry what port issue? In general yes, to be able to use IBeam as a Docker image you'd want to expose the correct port. You can read about this more here in the documentation: https://github.com/Voyz/ibeam/wiki/Installation-and-startup#using-docker-image-recommended

vishwajeetio commented 3 years ago

Hi @Voyz I am trying to build the docker image using the command below and the dockerfile from this repo: docker build -t ib_auth_api . And creating the container using the command below: docker run --env-file creds.list -d -p 5000:5000 ib_auth_api So I am able to access my account by open a bash shell inside the container but outside the container, it gives me the error saying access denied. I have also added all the additional files related to the gateway and all other needed. Can you please tell me how can I build my own image using your dockerfile. If possible can you share the command you used to build the image and the files that you have modified in the gateway directory. I have wasted more than a month on it and still unable to get it to work. Thank you

Voyz commented 3 years ago

hey @odssh - sorry to hear you're having so many troubles.

The command I use to build the image is this:

docker build -t ibeam .

The Docker commands you use seem fine to me.

I've recently realised that the original conf.yaml was indeed modified in IBeam - have a look at https://github.com/Voyz/ibeam/wiki/Gateway-Configuration to see the version IBeam uses. You may need to add 172.17.0.* to allowed IPs to communicate with IBeam from outside of Docker if using 'localhost' as the hostname. Have a look at #15 for more info on this. Also, check https://github.com/Voyz/ibeam/wiki/Troubleshooting#access-denied for more info on this error. Let me know if these help you with the Access Denied - otherwise try to describe exactly what you're doing and what commands you're using in as much detail as possible.

Also, could you outline why are you building the Docker image yourself instead of using voyz/ibeam:latest?

vishwajeetio commented 3 years ago

Thank you so much @Voyz It finally worked. I wish I had asked it a month ago but I thought It was more of a docker related issue. Again thank you for the quick reply. Hope you have a wonderful day

Voyz commented 3 years ago

Yay! @odssh Happy to hear! It was the 172.17.0.*? Yeah we only just figured it out recently, although you surely should feel free to report issues as soon as you encounter them, as possibly some of these are bugs and can be fixed relatively easily.

Can I nonetheless ask you to answer this question:

Also, could you outline why are you building the Docker image yourself instead of using voyz/ibeam:latest?

Just trying to understand whether there are some features you consider crucial that we could implement into IBeam by default, or some other issues that prevent you from using that image. Appreciate it 👍

vishwajeetio commented 3 years ago

I added both 0.0.0.0 and 172.17.0 and it seems to be working. I will test out both cases in detail later on. I am still learning the docker so I just love to do it and try to practice as much as I can. Also since I have added Twilio code so It feels a lot more familiar for me to edit. I will be more than happy to contribute to the project as soon as I will get good at it. Thank you so much for your time.

Voyz commented 3 years ago

Fantastic, thanks for your explanation 👍 It indeed is a great exercise to practice building someone else's code, and I'm both happy that you finally succeeded and sorry that you encountered so many troubles on your way. Other users also made their own forks of IBeam and shared it with us to the benefit of community, so if you'd be interested I'd highly encourage you to share your code with us and see what improvements you made that could benefit us all.

At this point I'd also highlight that in #12 we're implementing a standard way of handling 2FA, which I think may suit your Twilio use case. In fact, it would be an incredible test case if you managed to get Twilio to run 2FA with the version introduced in #12. It also contains many other improvements. If you'd be interested to test it out do let me know and I'll be happy to help you transition, although no worries if not - that version will be published soon nonetheless so you'll have an opportunity to try it out then.

All the best 👋 Voy

vishwajeetio commented 3 years ago

I will definitely share the changes I have made to the repo once I can get it all working. Since I have got the authenticator working so I am now moving on to my actual trader and just want to get it running somehow and then I will solely focus on IBeam authenticator to improve it as much as I can. Soon after I get my bot running, I will be working on adding the scheduler to the authenticator so that we only try to authenticate when we need to be authenticated. That feature would be great as an option as it can make the authenticator a lot more precise and efficient for the specific use cases. I highly appreciate the time and support you are giving and will try my best to make as much contribution as I can to this project. thank you

Voyz commented 3 years ago

hey @odssh thanks for outlining all of that! I'd be interested to see and hear what scheduler for the authenticator would do. Keep us posted on your progress 👍

And thanks for the kind words, I'm happy to be able to support you in your trading 😊

vishwajeetio commented 3 years ago

thank you for all the help and I will definitely make sure to update you on that scheduler thing. I have run into another issue. So while trying to add a proprietary certificate I got everything done but when I am running the container by mounting the inputs as a volume using the command below: 

docker run -v /inputs:/srv/inputs --env-file paper.list -d -p 5000:5000 ib_with_ssl

I am getting the error on making requests. so when I ran the command 

curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem

I am getting this error:

curl: (60) SSL certificate problem: self-signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

I followed the exact guide in the page tls Please let me know if there's anything I have done wrong. thank you

Voyz commented 3 years ago

Sorry to hear that @odssh!

Which OS are you calling cURL from?

What is the content of your /srv/inputs?

vishwajeetio commented 3 years ago

I am running the whole program on the ubuntu server and running the curl command from the same host on which the container is running. When I checked the container and went to /srv/inputs directory inside the container it had none of the files that I mounted. As you can see from the command I used I mounted the whole /inputs directory which contained three files: cacert.jks cacert.pem conf.yaml Please let me know if there's any other details you need thank you

Voyz commented 3 years ago

That's surprising. Can you try mounting that volume to another docker image and seeing if this time the directory contains the files?

vishwajeetio commented 3 years ago

I have tried to mount the inputs directory multiple times in all the ways possible but still the same error and works only when I give -k tag at the end. Should I try manually importing the files inside the container and then restarting the container? thank you

Voyz commented 3 years ago

So when you mount that directory to another container that uses a different image (not IBeam) you also don't see any files inside of the mounted directory? This would lead me to the following guesses:

The latter two are easier to verify, but if these aren't the cause of the issue then I'm sorry but I'm not entirely sure how to help you. You could try the setup on a different machine with a fresh installation of Docker and see if the issue exists there too?

vishwajeetio commented 3 years ago

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem curl: (60) SSL: unable to obtain common name from peer certificate More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

That is the error message I am getting now even after copying the ca certificate manually to the destination inside the container. It might be because of the certificate is invalid or something else.

Also, how can I use the default certificate that comes with a gateway with python instead of that custom one. I went with the custom as the default certificate wasn't working with the python thank you

Voyz commented 3 years ago

I don't know how IBeam would behave if you copy the certificates manually - this is an unexpected behaviour and isn't really supported. How about the points I brought up in my previous reply?

The default certificate needs to be provided through the Inputs Directory, but since you can't mount the volume correctly this will not be possible until that gets sorted

vishwajeetio commented 3 years ago

root@henon:~/gateway/inputs# curl -X GET "https://localhost:5000/v1/api/one/user" --cacert cacert.pem curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

Hi, @Voyz I got that error when using the beam's official docker image with the paper account. If possible can you please help me with where I am doing wrong? thank you

Voyz commented 3 years ago

Okay @odssh, I'm trying to help you but I'm feeling there's some miscommunication between us. I need you to address the points that I bring up. It's useful that you provide me with the error messages, but there's a number of questions or observations I replied you with that you haven't really addressed. Please let's do things in the right order and so we can figure out how to fix your setup, okay?

You need to start by discovering what is the reason for not being able to mount the Docker volume correctly. Have a look at this reply I made https://github.com/Voyz/ibeam/issues/8#issuecomment-813010891 and confirm for me which seems to be the case - or update me if this is no longer a problem and you can now mount volumes correctly. Otherwise please provide some screenshots of outputs of ls and pwd commands from your /inputs directory to verify its contents. Thanks 👍

vishwajeetio commented 3 years ago

Hi @Voyz thank you so much for your support. Finally got it to work with the custom SSL certificate. I was providing the wrong path for the input folder and I just kept trying with the same wrong path. Just another question I want to add, is how can I request the historical market data using that API? From the documentation, it seems like I need to make a get request to https://localhost:5000/v1/api/iserver/marketdata/history but when I do so I get the error message "Error 403 - Access Denied", Any idea on how can I fix that if you have done it before or if there's anything I am missing out? thank you

Voyz commented 3 years ago

Hey, happy to hear you got that sorted! Issues with Access Denied are described in the Troubleshooting:

https://github.com/Voyz/ibeam/wiki/Troubleshooting#access-denied

This should help you out 👍

vishwajeetio commented 3 years ago

Actually, it's working fine with the endpoint https://localhost:5000/v1/api/one/user but failing with the historical data endpoint. I think this issue is probably because of how I have written my request and have nothing to do with iBeam. Please let me know if there's any other endpoint you have worked with as I think I am providing the conid wrong. Any idea on how can I generate it? thank you

Voyz commented 3 years ago

You can check the conid's here:

https://contract.ibkr.info/v3.10/index.php?action=Stock%20Search&symbol=DDD&lang=en&ib_entity=&wlId=IB&showEntities=Y

Just type the symbol, click on 'Details' of the right stock and conid will be there.

Paste your full code for performing the request here, including the headers and parameters if you'd like me to see if I can spot something wrong with it.

michaeljherrmann commented 3 years ago

Hey @Voyz! I stumbled across your repo as I've been working on a similar project. I've come up with a solution to two factor authentication in case you're interested in another approach. 😄 https://github.com/michaeljherrmann/ib-gateway-service

Voyz commented 3 years ago

Hey @michaeljherrmann welcome to IBeam mate! 👋 Super cool of you to share your project, fantastic to see your approach!

The direct API requests you do are amazing, great job figuring this out! Just out of curiosity - do you know if your API interaction isn't against their TOC?

Going with Twilio is something that others have suggested in this thread, how do you find it working out for you?