search

brndnmtthws / thetagang

ThetaGang is an IBKR bot for collecting money
GNU Affero General Public License v3.0
1.9k stars 249 forks source link

Issue running thetagang with IBC in Poetry #329

Closed brndvg22 closed 7 months ago

brndvg22 commented 8 months ago

Good afternoon,

I'm trying to run Thetagang in Poetry to test some modifications later, but I can't get it to work with IBC. My computer runs Zorin OS and TWS 1025, I've also attempted in an Ubuntu virtual machine with TWS 1019, issues are a bit similar. IBC by itself works and Thetagang without IBC works as well so I'm pretty sure TWS and config files are set correctly

With Ubuntu/TWS1019 I got the following message: (Offline TWS/Gateway version 1019 is installed) Error: Offline TWS/Gateway version 1019 is not installed: can't find jars folder Make sure you install the offline version of TWS/Gateway IBC does not work with the auto-updating TWS/Gateway

With Zorin/TWS1025 I got the same message, then modified thetagang.py line 297 to point to TWS 1025. Then I got the following error msg WARNING: Illegal reflective access by feature.search.recorder.JtsMultiLookAndFeel (file:/home/zorinhp/Jts/1025/jars/twslaunch-1025.jar) to method javax.swing.UIManager.getLAFState() WARNING: Please consider reporting this to the maintainers of feature.search.recorder.JtsMultiLookAndFeel WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release

I also get different error messages depending on if I installed openjdk or default-jre. I'm guessing I'm not building the poetry environment right or my machine runs the wrong java version, but not sure how to address this. I haven't used autohooks as I'm not sure what this is meant to do. The poetry toml file looks as follows right now: [tool.poetry] name = "zorinhp-thetagang" version = "0.1.0" description = "zorinhp-thetagang" authors = ["Your Name [you@example.com](mailto:you@example.com)"] readme = "README.md" [tool.poetry.dependencies] python = "3.11.6" thetagang = "^1.8.1" [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"

What is missing to run the program properly in Poetry? Generally I'm a bit new in linux so hope I didn't miss some obvious things. Thanks a lot for any feedback

brndnmtthws commented 8 months ago

Trying to get the right combination of TWS and the JRE is tricky, but in this particular case it looks like you are missing some switches to make it work with more recent Java versions. Specifically, I think you're missing these: https://github.com/brndnmtthws/thetagang/blob/main/Dockerfile#L33-L38

You might also need to add --illegal-access=deny. It should work fine with the newer version of TWS, but it's really hard to get the combination of flags/config correct. See this related issue: https://github.com/IbcAlpha/IBC/issues/177

If any IBKR employees are paying attention: you should hire me for a contract gig to fix this gigantic mess.

brndvg22 commented 8 months ago

Thanks for the response. Just to be sure I tried installing again in a new Ubuntu VM with TWS 1025 and following all the steps as closely as possible from the thetagang dockerfile. Again I get these error messages when running Thetagang with IBC, both with pip install and poetry install, while Thetagang and IBC do work separately from each other

:thetagang -c /home/ubuntu/ibc/thetagang.toml Error: Could not find or load main class Error Caused by: java.lang.ClassNotFoundException: Error

:poetry run thetagang -c /home/zorin/ibc/thetagang.toml java.lang.reflect.InaccessibleObjectException: Unable to make private static javax.swing.UIManager$LAFState javax.swing.UIManager.getLAFState() accessible: module java.desktop does not "opens javax.swing" to unnamed module @45e626b5 at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:391) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:367) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:315) at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:203) at java.base/java.lang.reflect.Method.setAccessible(Method.java:197) at feature.search.recorder.JtsMultiLookAndFeel.a(JtsMultiLookAndFeel.java:107) at jclient.login.ak.i(ak.java:92) at ibgateway.GWClient.a(Unknown Source) at ibgateway.GWClient.main(Unknown Source) at ibcalpha.ibc.IbcTws.startGateway(Unknown Source) at ibcalpha.ibc.IbcTws.startTwsOrGateway(Unknown Source) at ibcalpha.ibc.IbcTws.load(Unknown Source) at ibcalpha.ibc.IbcGateway.main(Unknown Source)

So it definitely looks like those modules in the dockerfile are missing as you said, unfortunately I wasn't able to install these, also tried this with the same java version as the dockerfile

:JAVA_HOME=/opt/java/openjdk PATH=/opt/java/openjdk/bin:$PATH :javac --version && java --version javac 17.0.3 openjdk 17.0.3 2022-04-19 OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7) OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing)

I get the following error message (the module jar files are in the right path openjfx/lib)

:java --module-path /usr/share/openjfx/lib | tee -a /root/Jts/1025/tws.vmoptions && java --add-modules java.base,java.naming,java.management,javafx.base,javafx.controls,javafx.fxml,javafx.graphics,javafx.media,javafx.swing,javafx.web | tee -a /root/Jts/1025/tws.vmoptions Error occurred during initialization of boot layer java.lang.module.FindException: Module javafx.base not found

Googling this error message didn't really help, most are recommending using Maven or other programs to change Java VM settings but I have zero experience with Java or any of those programs. So if this is a Java issue, that will just take me a while to learn and figure out. Do you think there could be a way around this error and/or having to separately do these fixes to run Thetagang?

Best regards

brndnmtthws commented 8 months ago

:java --module-path /usr/share/openjfx/lib | tee -a /root/Jts/1025/tws.vmoptions && java --add-modules java.base,java.naming,java.management,javafx.base,javafx.controls,javafx.fxml,javafx.graphics,javafx.media,javafx.swing,javafx.web | tee -a /root/Jts/1025/tws.vmoptions Error occurred during initialization of boot layer java.lang.module.FindException: Module javafx.base not found

This doesn't look right. If everything works, your /root/Jts/1025/tws.vmoptions should look something like this (which is what thetagang uses):

#
# This file contains VM parameters for Trader Workstation 10.19.
# Each parameter should be defined in a separate line and the
# last line must be followed by a line feed. No leading or
# trailing whitespaces are allowed in the same line, where a
# parameter is defined.
#
# Lines starting with a '#' character are treated as comments
# and ignored. Additionally, if a line contains a
# '### keep on update' string, all parameters defined below will
# be preserved on update.
#

# maximum Java heap size
-Xmx768m

# GC settings
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=20
-XX:ConcGCThreads=5
-XX:InitiatingHeapOccupancyPercent=70

-Dinstaller.uuid=86288880-934a-4213-a6f4-6b20e2438370
-DvmOptionsPath=/root/Jts/1019/tws.vmoptions
-Dsun.awt.nopixfmt=true
-Dsun.java2d.noddraw=true
-Dswing.boldMetal=false
-Dsun.locale.formatasdefault=true

### keep on update
--module-path /usr/share/openjfx/lib
--add-modules java.base,java.naming,java.management,javafx.base,javafx.controls,javafx.fxml,javafx.graphics,javafx.media,javafx.swing,javafx.web
--add-opens java.desktop/javax.swing=ALL-UNNAMED
--add-opens java.desktop/java.awt=ALL-UNNAMED
--add-opens java.base/java.util=ALL-UNNAMED
--add-opens javafx.graphics/com.sun.javafx.application=ALL-UNNAMED
brndvg22 commented 8 months ago

Thanks for the response. I changed the vmoptions file and that resolved that specific error.

After this change I get the error message 'invalid twsinfo'. Also now TWS keeps restarting over and over after receiving a message 'this is not a brokerage account'. I noticed the 'invalid twsinfo' was discussed in IbcAlpha/IBC#151 and resolved by using openjdk17.0.4_8, so I tried that version, this does get rid of 'invalid twsinfo' but still has TWS looping and gives an error message 'java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found'. So as you said it is really difficult getting the right java version (at least, I hope I am setting up java in the right way since I'm new to this). The only version I found so far that does not give any error message in the TWS or terminal is openjdk 13.0.7, but Thetagang still will not start and TWS keeps looping in and out. Thetagang --without-ibc still works fine, though IBC alone will not run anymore after modifying tws.vmoptions as described.

Settings: In thetagang.toml I am pointing to javaPath = '/usr/lib/jvm/java-13-openjdk-amd64/bin' zorin2@zorin2:~$ java --version openjdk 13.0.7 2021-04-20 OpenJDK Runtime Environment (build 13.0.7+5-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 13.0.7+5-Ubuntu-0ubuntu120.04, mixed mode)

pyproject.toml file: [tool.poetry.dependencies] python = ">=3.10.13,<3.13" thetagang = "^1.8.1" zorin2@zorin2:~$ pyenv versions system

Thanks a lot for checking again. Is it sufficient to just point to a java/bin folder in the thetagang.toml file, without changing the system java version? Because I have been changing java versions via update-alternatives but I'm not sure if this is necessary at all. Any idea what other things I could try to get this running? It seems odd to me that Thetagang, TWS and IBC individually all work fine whatever java settings are, but it becomes complicated when running Thetagang with IBC together, maybe do you have any pointer as to where this problem could come from, where to look further? Otherwise for now I will spend some more time to learn and try out some more, I haven't tried to further modify the pyproject.toml and add some of the dependencies that are in the repository like autohooks etc, so will see what that does.

brndnmtthws commented 8 months ago

Setting the java path in thetagang.toml should be sufficient, but it's possible that some code somewhere is ignoring that setting. Maybe check that JAVA_HOME is unset, just in case something is trying to use that env var instead of the one specified in the config.

The code that launches TWS (via IBC) is here: https://github.com/erdewit/ib_insync/blob/master/ib_insync/ibcontroller.py

It should respect javaPath. It's just calling the ibcstart.sh script, which is here: https://github.com/IbcAlpha/IBC/blob/master/resources/scripts/ibcstart.sh

It also looks like you shouldn't include the trailing /bin in the path, which might be causing the fallback to JAVA_HOME, based on the code here: https://github.com/IbcAlpha/IBC/blob/master/resources/scripts/ibcstart.sh#L397-L454

brndvg22 commented 7 months ago

Thanks for the response. Unfortunately it doesn't resolve the issue that Thetagang is looping and won't start with IBC, but the original question in this issue is resolved. I'll continue trying other things to make it run with Poetry.