[feature request] save decrypted logs

[anj00]

The logs for gateway are encrypted and (as far as I know) there is no way having them stored decrypted. For automated system one has to resort to either manually export (but has to be done before shutdown/cleaning up) or download encrypted logs to a local machine, login with the same user credentials/same gw version and try to export again. a lot of manual work. and prone to errors (somehow sometimes not full log gets decrypted)

Is it possible to create a command which would save decrypted logs (and API logs)?

At minimum File->Gateway Logs -> Export Today's Logs File->Api Logs -> Export Today's Logs

The idea is that one could run such commands before shutting down and thus have decrypted logs, store then next to other logs and as result simplify debugging of tricky issues. As GW logs could be stored next to trading logs and compared

[rlktradewright]

The short answer is that yes, it would probably be possible.

However I don't see any real value in it, and I suspect it would be difficult. I simply have neither the time nor the inclination to do this. Also it really isn't what IBC is for. The problem with log files is entirely of IBKR's making, and they should provide usable tools to enable easy access to them, rather than me trying to bend IBC into doing something that's way outside its real purpose.

IBKR could validly argue that they have provided such tools, via the manual method you refer to, which isn't really that onerous but of course is not much use in an automated, remotely hosted scenario.

However, I would suggest that if your software is not working sufficiently well that you need to look at the logs on a regular basis, then maybe you shouldn't be trying to run it automated and should just concentrate on fixing its problems in a more controlled manual context. That may just be a matter of running it locally rather than on a remote server, if that's what you're doing.

[anj00]

Richard, Of course this is not about debugging own software. For that of course own logging and debugging tools exist (and debugging remote system is not an issue). Lets get this out of the way.

Of course it would be ideal if IBKR could provide a more "reasonable" logging solution. But they haven't. Just like they haven't provided reasonable tools for automated auth :) . So it feels natural that if we have tools for automated authentication, then next step is collecting logs from the same software (one of the few things I cannot do via API or settings file: auth and collecting logs)

As to why getting logs would be great: after running remotely automated IB systems for close to 3 years now (enabled by IBC (bow) ) I noticed it is the same pattern I run into then it comes to bugs on IB side (there are several over the years which IB introduced and fixed, some hopefully with help of my reports/logs )

1. Some issue happens (typically a new release of a gateway or a new backend release).
2. I report the issue
3. IB asks for IB logs.
4. I need to monitor the machine and in case issues pops up again submit the logs (Ctrl+Alt+Q or exporting logs.) that pattern is not exactly devops friendly :) .

In case IB logs would be stored (exported by IBC) decrypted that debug loop would be so much less time pressing:

1. One can submit the logs after the fact (some issues do not show up every run, but show up once a week or so)
2. One can go and parse IB log history the same way one can do with own logs and look for patterns, output/monitor KPI's and see if they maintain correct numbers. Anything to do with delays for in/out data for example.

Example of #2 recently looking at the logs I noticed that every so often IB incoming thread stops processing my orders ("hungs" for 20-50 ms) it could be related to garbage collection, could be cpu starving, could be some more internal IB related stuff. If those KPIs (min/max/median/bottom 10% etc of delay) could be monitored one could start experimenting with aws/java virtual machine settings. Say give more cpu, does kpi improve? Give more memory, does kpi improve? different java settings: anything got better/worse? All such ideas/experiments depend on ability to process IB logs as plain text. (Of course IB is not about HFT. And I am not trying to. Those 30ms delays not going to kill the algorithm. However, there is no reason the have them and if there is something I can do to fix them, then it is great)

So to summarize: this functionality is about simplifying of

1. reporting IB issues to IB
2. optimization of infrastructure

AJ

[rlktradewright]

Well fine, I understand all that, but you don't seem to be hearing what I'm saying. I am not going to do this, because it would not be good use of my time, and the lack of this feature does not prevent anyone from using IBC to do what it's designed for.

So if you want this feature, you are perfectly entitled to develop it yourself. You could then offer it back as a pull request, though I won't guarantee that I'd accept it without further thought (here's a clue: I probably wouldn't).

If you don't feel you have the Java skills to do this development, find someone who does and is willing, or pay someone to do it. You are happy for me to commit hours or days of my precious time to do this, but do you want it enough to commit your own (or your company's) money to make it happen? And by the way, even if you offered to pay me I still wouldn't do it, it really really is not what I want to be spending my time on.

If I want to get today's logs from my TWS, it takes me about half a minute to set up a remote desktop connection to my server, do the necessary keystrokes on TWS, wait for the export, and then close the session. That is really not much pain, even if you want to do it every day.