IbcAlpha / IBC

Automation of Interactive Brokers TWS. You can download the latest release here: https://github.com/ibcalpha/ibc/releases/latest
GNU General Public License v3.0
1.07k stars 185 forks source link

Reset API order ID sequence on every start #214

Closed matiasrivera closed 10 months ago

matiasrivera commented 1 year ago

I have an automated trading software developed that can send orders to several different markets. I have a unique client id generator that shares those ids to every market it's connected to. IBAPI relies on the API order ID sequence from IB Gateway or TWS and you are not allowed to ever send a number lower than the last ID used. The ID has a maximum number that cannot exceed the java.Lang.Number size.

Error. Id:-1 Code: 320 Msg: Error reading request. Unable to parse data. java.lang.NumberFormatException: For input string: "2548181185"

Then I have to frequently reset this ID. The only way I've found to do it in IB Gateway is by clicking in the "Reset API order ID sequence" button in the API - Settings configuration screen. This is very tedious since I'm running several different IB Gateway connections and I have to manually reset them all.

image

Is it possible to automatically reset this ID every time the gateway is started?

Thank you,

Matías

rlktradewright commented 1 year ago

Short answer is yes, it would be possible to do this, but I'd be rather reluctant to allow it.

First of all, are you aware that the sequence of order ids that must never be less than the highest id used so far is specific to each client id? In other words, order ids for client id 1 are completely separate from those for client id 2. The only exception to this is for the master client: it receives ordersStatus and openOrder callbacks for every client id connected to the TWS, and if it places orders itself then it must use order ids higher than any from other client ids (but the other clients can just carry on following their own sequence): I've always found this to be a very odd implementation by IB, but that's the way it is.

Since the order id is a 32-bit integer, each client id can potentially place more than 2 billion orders before running out of order ids. That's about 140,000 orders per day over 40 years, which ought to be enough for most traders!

So I really don't understand why you would need to frequently reset the order id sequence.

Next, you say that 'I'm running several different IB Gateway connections and I have to manually reset them all'. But you can't reset the order id sequence fora specific client id: whien you click the Reset API order ID sequence button it resets the ids for all clients to 0. Perhaps you mean you're running several different Gateway instances?

Also, if you automatically reset the sequence at each Gateway start, have you considered what would happen if the previous Gateway session still had open orders? I don't actually know the answer to that question, but I suspect there could be trouble.

In conclusion, I think you need to reconsider whether you really have a valid use case for this idea.

I'm not saying that this should not be done (an alternative would be a RESETIDS command sent to the IBC command server, in a similar way to the STOP and RESTART commands). But I want to be sure that it's something that's actually going to be of real value, as opposed to just using a more economical approach to ids in your client apps.

matiasrivera commented 1 year ago

Hi Richard, First of all thank you very much for taking the time to answer my question. Secondly I apologize in advance for my english since it's not my mother tongue. I will try to explain in more detail my current situation. I have developed a sort of algorithmic trading platform that allows different financial institutions in my country (Argentina) to create robots that trade on different markets in different countries at the same. Most of the algorithms I have developed work as Market Making algorithms which means the change orders very frequently on hundred of securities during normal trading hours. No human user interacts or trades with these clients or accounts, only algorithmic trading is performed. My software uses a unique id for all markets (since it's the easiest way to manage that). I place around 100 orders per second which means around 450.000 orders per day. On the other markets I trade the ID has to be unique only during the current trading session. That means its possible to reset that ID every day. Since IBKR gateway doesn't allow to reset the API daily I have to do it manually once in a while, the problem is that my system runs autonomously in a remote server and of course, having to manually reset the id is something I would like to avoid if possible

I have made some changes this week and have taken the chance to manually reset the ids so now probably I have a couple of weeks until next reset. Anyhow, I was wondering if I can have the chance to totally foget about this and have it automatically done every day.

Thanks

Matías

On Thu, May 18, 2023 at 7:20 PM Richard L King @.***> wrote:

Short answer is yes, it would be possible to do this, but I'd be rather reluctant to allow it.

First of all, are you aware that the sequence of order ids that must never be less than the highest id used so far is specific to each client id? In other words, order ids for client id 1 are completely separate from those for client id 2. The only exception to this is for the master client: it receives ordersStatus and openOrder callbacks for every client id connected to the TWS, and if it places orders itself then it must use order ids higher than any from other client ids (but the other clients can just carry on following their own sequence): I've always found this to be a very odd implementation by IB, but that's the way it is.

Since the order id is a 32-bit integer, each client id can potentially place more than 2 billion orders before running out of order ids. That's about 140,000 orders per day over 40 years, which ought to be enough for most traders!

So I really don't understand why you would need to frequently reset the order id sequence.

Next, you say that 'I'm running several different IB Gateway connections and I have to manually reset them all'. But you can't reset the order id sequence fora specific client id: whien you click the Reset API order ID sequence button it resets the ids for all clients to 0. Perhaps you mean you're running several different Gateway instances?

Also, if you automatically reset the sequence at each Gateway start, have you considered what would happen if the previous Gateway session still had open orders? I don't actually know the answer to that question, but I suspect there could be trouble.

In conclusion, I think you need to reconsider whether you really have a valid use case for this idea.

I'm not saying that this should not be done (an alternative would be a RESETIDS command sent to the IBC command server, in a similar way to the STOP and RESTART commands). But I want to be sure that it's something that's actually going to be of real value, as opposed to just using a more economical approach to ids in your client apps.

— Reply to this email directly, view it on GitHub https://github.com/IbcAlpha/IBC/issues/214#issuecomment-1553733927, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSAMGXIKB3WV32NNZXLXG2OBNANCNFSM6AAAAAAYD552MM . You are receiving this because you authored the thread.Message ID: @.***>

rlktradewright commented 1 year ago

Your English is excellent, so no need to apologise!

You certainly seem to be in a rather unusual situation. I'll implement this and let you have a new version so you can try it out and see if it really helps.

rlktradewright commented 1 year ago

OK, I've implemented that request. Below I've linked zips for the new version 4.0.0-beta.1 (I don't know what platform you're using so I've linked them all).

IBCLinux-4.0.0-beta.1.zip IBCMacos-4.0.0-beta.1.zip IBCWin-4.0.0-beta.1.zip

The only things that have changed are:

IBC.jar version config.ini

Note that config.ini has two new settings, and you'll need to read the relevant notes for them. They are:

ResetOrderIdsAtStart ConfirmOrderIdReset

but note that ConfirmOrderIdReset is only relevant for TWS as Gateway never displays the confirmation dialog.

To make it automatically reset the order ids at every start, but prevent any other use of the reset button, you'll need to set these like this for TWS:

ResetOrderIdsAtStart=yes
ConfirmOrderIdReset=confirm/reject

For Gateway you can leave ConfirmOrderIdReset with no value. You can't prevent the user resetting the order ids manually for Gateway.

Let me know how you get on with this.

matiasrivera commented 1 year ago

Hi Richard, Thank you very much for taking the time to help me with this. I have downloaded the linux version you attached and tried it out in my local installation. So far the implementation has worked locally. Tomorrow I will be giving it a try in my production environment. I will let you know how it goes.

Matías

On Sun, May 21, 2023 at 9:21 PM Richard L King @.***> wrote:

OK, I've implemented that request. Below I've linked zips for the new version 4.0.0-beta.1 (I don't know what platform you're using so I've linked them all).

IBCLinux-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525823/IBCLinux-4.0.0-beta.1.zip IBCMacos-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525824/IBCMacos-4.0.0-beta.1.zip IBCWin-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525825/IBCWin-4.0.0-beta.1.zip

The only things that have changed are:

IBC.jar version config.ini

Note that config.ini has two new settings, and you'll need to read the relevant notes for them. They are:

ResetOrderIdsAtStart ConfirmOrderIdReset

but note that ConfirmOrderIdReset is only relevant for TWS as Gateway never displays the confirmation dialog.

To make it automatically reset the order ids at every start, but prevent any other use of the reset button, you'll need to set these like this for TWS:

ResetOrderIdsAtStart=yes ConfirmOrderIdReset=confirm/reject

For Gateway you can leave ConfirmOrderIdReset with no value. You can't prevent the user resetting the order ids manually for Gateway.

Let me know how you get on with this.

— Reply to this email directly, view it on GitHub https://github.com/IbcAlpha/IBC/issues/214#issuecomment-1556344650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSB73EYSBZ2B5IWBEBDXHKWQTANCNFSM6AAAAAAYD552MM . You are receiving this because you authored the thread.Message ID: @.***>

matiasrivera commented 1 year ago

Hi Richard, Today I implemented the new version in my production environment. This is perfect for my setup. Thanks again for your help.

Matías

On Mon, May 22, 2023 at 9:05 PM Matias Rivera @.***> wrote:

Hi Richard, Thank you very much for taking the time to help me with this. I have downloaded the linux version you attached and tried it out in my local installation. So far the implementation has worked locally. Tomorrow I will be giving it a try in my production environment. I will let you know how it goes.

Matías

On Sun, May 21, 2023 at 9:21 PM Richard L King @.***> wrote:

OK, I've implemented that request. Below I've linked zips for the new version 4.0.0-beta.1 (I don't know what platform you're using so I've linked them all).

IBCLinux-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525823/IBCLinux-4.0.0-beta.1.zip IBCMacos-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525824/IBCMacos-4.0.0-beta.1.zip IBCWin-4.0.0-beta.1.zip https://github.com/IbcAlpha/IBC/files/11525825/IBCWin-4.0.0-beta.1.zip

The only things that have changed are:

IBC.jar version config.ini

Note that config.ini has two new settings, and you'll need to read the relevant notes for them. They are:

ResetOrderIdsAtStart ConfirmOrderIdReset

but note that ConfirmOrderIdReset is only relevant for TWS as Gateway never displays the confirmation dialog.

To make it automatically reset the order ids at every start, but prevent any other use of the reset button, you'll need to set these like this for TWS:

ResetOrderIdsAtStart=yes ConfirmOrderIdReset=confirm/reject

For Gateway you can leave ConfirmOrderIdReset with no value. You can't prevent the user resetting the order ids manually for Gateway.

Let me know how you get on with this.

— Reply to this email directly, view it on GitHub https://github.com/IbcAlpha/IBC/issues/214#issuecomment-1556344650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSB73EYSBZ2B5IWBEBDXHKWQTANCNFSM6AAAAAAYD552MM . You are receiving this because you authored the thread.Message ID: @.***>

rlktradewright commented 1 year ago

Ok, good. I'll include it in the next release.

matiasrivera commented 9 months ago

Hi Richard, I'd like to know if this feature has been included in any release since you implemented it. You sent me version 4.0.0-beta.1 in the past and I don't see v4 already released.

Thanks,

Matias.

On Thu, Feb 1, 2024 at 9:24 AM Richard L King @.***> wrote:

Closed #214 https://github.com/IbcAlpha/IBC/issues/214 as completed.

— Reply to this email directly, view it on GitHub https://github.com/IbcAlpha/IBC/issues/214#event-11669372210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSH4ROW7VRK7CFD2TZLYROCOHAVCNFSM6AAAAAAYD552MOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGY3DSMZXGIZDCMA . You are receiving this because you authored the thread.Message ID: @.***>

rlktradewright commented 9 months ago

@matiasrivera yes, it was included in Release 3.17.0 on 27 June 2023. I decided not to move to the v4 version numbers yet.