IBC does not restart Gateway after it exits

[dkharrat]

I'm using IBC to run both a paper and live of IB Gateway using a docker package.

I noticed for some reason the paper Gateway shutdown and IBC did not restart it (at around 2:01 PM UTC). The Gateway for the live account is still running fine. I'm not sure why the paper Gateway shutdown. Restarting the container worked fine and both paper & live Gateways started fine. Here's the relevant logs (times are UTC):

2024-10-12 07:10:20:779 IBC: detected frame entitled: Starting application...; event=Lost focus
2024-10-12 07:10:20:783 IBC: detected frame entitled: Starting application...; event=Deactivated
2024-10-12 07:10:20:788 IBC: detected frame entitled: Starting application...; event=Closed
2024-10-12 16:01:11:766 IBC: detected frame entitled: Starting application...; event=Lost focus
2024-10-12 16:01:11:767 IBC: detected frame entitled: Starting application...; event=Deactivated
2024-10-12 16:01:11:767 IBC: detected frame entitled: Starting application...; event=Closed
2024-10-12 16:01:51:972 IBC: detected frame entitled: Connecting to server...; event=Activated
2024-10-12 16:01:51:973 IBC: detected frame entitled: Connecting to server...; event=Focused
2024-10-12 16:01:51:974 IBC: detected frame entitled: Connecting to server...; event=Opened
2024-10-12 16:01:52:408 IBC: detected dialog entitled: Shutdown progress; event=Opened
2024-10-12 16:01:52:423 IBC: detected frame entitled: Attempt 2: Authenticating...; event=Lost focus
2024-10-12 16:01:52:423 IBC: detected frame entitled: Attempt 2: Authenticating...; event=Deactivated
2024-10-12 16:01:52:424 IBC: detected dialog entitled: Shutdown progress; event=Activated
2024-10-12 16:01:52:424 IBC: detected dialog entitled: Shutdown progress; event=Focused
2024-10-12 16:01:52:425 IBC: detected dialog entitled: IB Gateway; event=Opened
2024-10-12 16:01:52:425 IBC: detected dialog entitled: Shutdown progress; event=Lost focus
2024-10-12 16:01:52:426 IBC: detected dialog entitled: Shutdown progress; event=Deactivated
2024-10-12 16:01:52:427 IBC: detected dialog entitled: IB Gateway; event=Activated
2024-10-12 16:01:52:431 IBC: detected dialog entitled: IB Gateway; event=Focused
2024-10-12 16:01:53,370 JTS-ShutdownTask-Shutdown_IA-35213 ERROR Attempted to append to non-started appender d
2024-10-12 16:01:53,371 JTS-ShutdownTask-Shutdown_IA-35213 ERROR Attempted to append to non-started appender d
2024-10-12 16:01:53,374 NIA-Input-Queue-2 ERROR Attempted to append to non-started appender d
2024-10-12 16:01:53,374 NIA-Input-Queue-2 ERROR Attempted to append to non-started appender d
2024-10-12 16:01:53,566 pool-2-thread-1 ERROR Attempted to append to non-started appender f
IBC returned exit status 0
autorestart file not found

Gateway finished

My settings are:

IBC Settings:
    AcceptBidAskLastSizeDisplayUpdateNotification=accept
    AcceptIncomingConnectionAction=
    AcceptNonBrokerageAccountWarning=yes
    AllowBlindTrading=
    AutoLogoffTime=
    AutoRestartTime=11:59 PM
    BindAddress=
    BypassBondWarning=
    BypassCalledBondWarning=
    BypassNegativeYieldToWorstConfirmation=
    BypassNoOverfillProtectionPrecaution=
    BypassOrderPrecautions=
    BypassPriceBasedVolatilityRiskWarning=
    BypassRedirectOrderWarning=
    BypassSameActionPairTradeWarning=
    BypassUSStocksMarketDataInSharesWarning=
    ClosedownAt=
    ColdRestartTime=
    CommandPrompt=
    CommandServerPort=0
    ConfirmCryptoCurrencyOrders=transmit
    ConfirmOrderIdReset=
    ControlFrom=
    DismissNSEComplianceNotice=yes
    DismissPasswordExpiryWarning=no
    ExistingSessionDetectedAction=
    ExitAfterSecondFactorAuthenticationTimeout=no
    FIX=no
    FIXLoginId=***
    FIXPassword=***
    IbDir=
    IbLoginId=***
    IbPassword=***
    LogStructureScope=known
    LogStructureWhen=never
    LoginDialogDisplayTimeout=60
    MinimizeMainWindow=no
    OverrideTwsApiPort=
    OverrideTwsMasterClientID=
    ReadOnlyApi=no
    ReadOnlyLogin=no
    ReloginAfterSecondFactorAuthenticationTimeout=yes
    ResetOrderIdsAtStart=
    SaveTwsSettingsAt=
    SecondFactorAuthenticationExitInterval=60
    SecondFactorAuthenticationTimeout=180
    SecondFactorDevice=
    SendMarketDataInLotsForUSstocks=
    StoreSettingsOnServer=
    SuppressInfoMessages=yes
    TradingMode=paper
    TrustedTwsApiClientIPs=
End IBC Settings

The interesting part is that the exist status is 0. I believe this indicates a normal shutdown? Based on the ibstart.sh script, it appears that IBC does not attempt to restart Gateway if there's no autorestart file or exist status does not match certain codes.

Any idea why this happened? Also, what's the reason IBC does not attempt to restart Gateway in all cases?

[rlktradewright]

You need to provide the whole log file, not just the bits you think might be relevant.

Please attach it to your reply. Don't paste it inline..

What reason do you have for thinking IBC should restart Gateway every time it shuts down? It's not intended to do that.

[dkharrat]

Sure, here's the log file: ibgateway.log

Any reason why Gateway would shutdown on its own? I couldn't find any indication in the logs to show the cause.

What reason do you have for thinking IBC should restart Gateway every time it shuts down? It's not intended to do that.

How do you recommend one handles cases where Gateway exists for whatever reason (say it crashed)? It would be ideal to restart Gateway if there were any issues for better reliability. In this scenario, I had to manually restart the container to start up Gateway again, but It would be better to have some automatic handling of such cases.

[rlktradewright]

Thanks for the logfile, and apologies for the delay in responding. I've actually soent far too many hours looking at this to make sense of the file and to try and work out what's going on.

First, let me say that I haven't kept up with progress on this Docker implementation, but it certainly looks like a pretty thorough job, so kudos to @gnzsnz. I'll have to look into it more to understand it fully (I'm no Docker expert). The fact that the logs from both Gateway instances are interleaved in the same file took a bit of getting used to! I might change IBC to add something to the log entries to make clear which instance is logging each message.

The short answer to your question why Gateway shut down at 4pm on Saturday afternoon is proably just that, well, it's Saturday afternoon, and IBKR don't commit to continuity of sessions on Saturdays (because they often have maintenance work then). You absolutely cannot expect Gateway or TWS to sail though the weekend, and indeed you are required to shut them down completely at some point during Sunday (see https://www.ibkrguides.com/riskprofile/usersguidebook/configuretws/auto_restart_info.htm).

If you read the IBC User Guide you'll find quite a bit of guidance on ways to restart Gateway/TWS should they fail, for bith Linux and Windows. In practice this is rather unnecessary these days, because they are both very stable, though it still has its uses. This guidance is relevant to normal installations rather than Docker images. As I said, I'm not a Docker expert but I would imagine that Docker plus Kubernetes would be able to provide everything you need.

Below is a summary of what the logfile shows. Don't feel you need to follow it all, but I needed to do this to make sense of things:

• the live Gateway instance started at 2024-10-11 03:30:09 (line 28)
• the paper Gateway instance started at 2024-10-11 at 03:30:24 (line 387)
• at 2024-10-11 06:15:15:046 it looks like one instance reconnected to the server - possibly this is the end of the nightly server reseet period (line 680)
• at 2024-10-11 06:19:18:715 the other instance also reconnnected (line 706)
• at 2024-10-11 23:54:30:825 the live instance began its auto-restart, displaying the Exit Session Setting dialog to warn the user of the impending restart (line 732)
• at 2024-10-11 23:54:30:922 the paper instance began its auto-restart (line 737)
• at 2024-10-11 23:59:07 both instances shut down (line 763), having first created the autorestart files, which are detected by the ibcstart.sh script and their locations logged
• both Gateway instances are restarted (lines 770 and 778) a couple of seconds later
• at 2024-10-11 23:59:08:721 the paper gating tries to re-start the login session (Line 967), followed by the live instance at 2024-10-11 23:59:08:965 (line 1041)
• at 2024-10-11 23:59:20:313 (line 1122) and 2024-10-11 23:59:21:047(line 1139) both instances report that the autorestart file which should contain the encrypted credentials needed to re-establish the login sessions actually do not contain valid credentials. The the login sessions cannot be automatically re-establ;ished, and so the IBC instances initiate full login requiring user id and password and (for the live instance) 2FA. This threw me and I'll have more to say about it below
• IBC initiated the full login for the paper Gateway at 2024-10-11 23:59:20:735 (line 1130), and for the live at 2024-10-11 23:59:21:818 (line 1163)
• the paper login completed at 2024-10-11 23:59:35:204 (line 1182)
• strangely the live login completes at 2024-10-11 23:59:35:666 (Line 1195) in spite of the supposed need for another 2FA which hasn't happened
• everything then proceeds nicely until the start of the daily server reset around 2024-10-12 05:28:29, and then continues on until about 2024-10-12 16:01:11 when one of the instances (not sure which) shuts diown. I suspect this was because it's now Saturday afternoon and IBKR often do maintenance tasks on Saturdays

I mentioned that IBC was reporting invalid restart login credentials for both live and paper Gateways, and I've spent hours today trying to work out what the cause of this is. And I was rather surprised that no-one has raised this before for the live Gateway since it would seem to imply a need to do a daily 2FA: but the fact that it did complete the re-login for your live instance makes me suspect that IBC is getting this wrong. I'll need to investigate this more thoroughly, which I'll do at the weekend (I can't upset my live sstem during the week).