Reconnect Error

[PeXArtZ]

Hey, I just used the example of the reconnect strategy. There is the following code:

        // Make stuff run every time the query (re)connects
        config.setConnectionHandler(new ConnectionHandler() {

            @Override
            public void onConnect(TS3Api api) {
                stuffThatNeedsToRunEveryTimeTheQueryConnects(api);
            }

            @Override
            public void onDisconnect(TS3Query ts3Query) {
                // Nothing
            }
        });

I did that in my code, but IntelliJ is giving me this error:

Class 'Anonymous class derived from ConnectionHandler' must either be declared abstract or implement abstract method 'onConnect(TS3Query)' in 'ConnectionHandler

It says I cant use TS3Api in the onConnect method. Am I just dumb and I dont see the issue by myself? 🤔

Heres my code: https://hastebin.com/unejuduzik.java

[Tumbledore]

Hey Mate, The Error says it all: You need to override ‚onConnect(Ts3Query)‘ and Not ‚onConnect(Ts3 API)‘. You got. the wrong Argument there

[PeXArtZ]

Thats right, but if you look at the given example its the same there. Also I cant use 'TS3Query' there because this wouldnt work.

[rogermb]

Hi guys!

The reason for this confusion is a recent commit (3676af36ae2fb2b899d6d7c3f8001bc060db8b41) that changed the signature of the onConnect method. That commit hasn't made it into a release yet, so the examples you see on the repo don't match the JAR file you've downloaded.

To see the correct example, check out the repo @ version 1.2.0 (+ direct link to ReconnectExample.java).

Also, when writing onConnect methods, there is one important rule: Only use the TS3Query and TS3Api objects you get as arguments inside that onConnect method, and don't use these objects outside your onConnect method.

[PeXArtZ]

Thanks. That fixed it. I got some other classes in my bot too. I am using the api object there, but with the reconnect there is no public api object in my Load class anymore. Do you got a way to fix that?

[rogermb]

I'm not sure if I understand the problem correctly, but couldn't you add the TS3Api object as a parameter for these methods?

E.g. if you really need to run some method "foo" from your onConnect method, you could have

public void foo(TS3Api api) {
    api.whoAmI(); // or whatever
}

And just to make this clear: you can still use your regular ts3query.getApi() API object outside your onConnect handler. That API object will keep working even after the query reconnects. Just make sure you don't mix up the "regular" API objects and the API objects you're using in onConnect 😄

[PeXArtZ]

Thanks for your awesome help. It works now! :)

[rogermb]

Awesome, I'm glad to hear that! 😄

[PeXArtZ]

Unfortunately the bot went offline again at night. Between 1AM and 3AM it lost connection and reconnected everytime but at 3:39AM I got this error:

2019-03-06 03:39:07.611 [ERROR] ConnectionHandler disconnect task threw an exception
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:717)
    at com.github.theholywaffle.teamspeak3.QueryIO.<init>(QueryIO.java:90)
    at com.github.theholywaffle.teamspeak3.TS3Query.connect(TS3Query.java:128)
    at com.github.theholywaffle.teamspeak3.api.reconnect.ReconnectingConnectionHandler.onDisconnect(ReconnectingConnectionHandler.java:81)
    at com.github.theholywaffle.teamspeak3.TS3Query.handleDisconnect(TS3Query.java:255)
    at com.github.theholywaffle.teamspeak3.TS3Query.lambda$submitUserTask$0(TS3Query.java:232)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2019-03-06 03:44:07.524 [ERROR] Connection closed by the server.

I opened the task manager in putty and found out that there are ~10 tasks by the bot and the cpu usage was 100% permanently (average: ~5%-10%). Apparently the bot is creating a new task but is not closing the old one.

Heres my code: https://gist.github.com/PeXArtZ/6dc3ff339764639622635b39b5887b06

[Tumbledore]

Hey Mate, i think the Thread of console.start isnt closing properly. maybe use Java7 TryWithResource? or you could implement after your try/catch block a finally block where you can close the Thread.

maybe these links can help:

TryWithResource https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html

Try/Catch/Finally http://tutorials.jenkov.com/java-exception-handling/basic-try-catch-finally.html http://www.straub.as/java/exception/try-catch.html

[rogermb]

I checked whether repeated attempts to reconnect with v1.2.0 and my current v1.3.0 dev build of the API would cause alive threads to accumulate, but didn't find this to be the case. Even after hundreds of reconnect attempts, I still had just 1 socket reader, socket writer, and keep alive, as well as just as single pool-1-thread-1 user task thread.

@PeXArtZ It might be a good idea to let your application run for some time and then create a thread dump, just to see how many threads are running. I have a suspicion that one of your helper classes (AFKMover, TimerFunctions, Console, etc.) keeps creating new threads without you noticing 😃

[PeXArtZ]

I added } finally { Thread.currentThread().interrupt(); }

to my Console Try-Catch-Block and finally it works. There even no reconnect attempts and my cpu usage is fine. Thank you guys for your help. Hopefully its my last time commenting on this issue 😄

[PeXArtZ]

Indeed, I am here again, just for a quick question. How many threads are needed to use the reconnect? I started my bot on localhost (where I have 16 threads) and everything works fine. I have 2 threads on my vServer where I got all these errors. One of them is: java.lang.OutOfMemoryError: unable to create new native thread

So I think its just because I dont have enough threads, right?

[rogermb]

I think you're confusing CPU ""threads"" and Java Threads here. The former is just the number of "logical" cores your CPU has. If your CPU has Hyper-Threading, you'll have 2 threads per core.

However, these are almost entirely unrelated to Java Threads. So one Java Thread (in their current implementation) corresponds to one OS thread. Your operating system implements threads to allow processes to have multiple execution streams that run concurrently. More on wikipedia.

Each thread needs some memory for its stack, and that memory is limited. When you create more and more threads, at some point you'll run out of memory and get that OutOfMemoryError. But to reach that point, you'll have to create hundreds or even thousands of threads. There's no way that the few threads that this API creates cause this error on their own.

That's why I urged you to create a thread dump earlier: I'm almost certain that your code is creating more and more threads somewhere without you noticing, which causes you to eventually run out of memory. This is not because of the machine you're running your code on, but likely because of a bug in your code 😃

[PeXArtZ]

Alright I created a thread dump. This time there was only one reconnect. (Idk why sometimes there are like 15 in 10mins) Heres my thread dump about 30secs after I started the bot: https://hastebin.com/gikesaqiya.pl

thread dump about 18 hours later (after 1 reconnect): https://hastebin.com/afucuvecul.pl

[PeXArtZ]

Happened about 2 mins ago: Out of nothing the bot used 100% of my cpu. I wasn't able to create a thread dump at this point because all of my system resources were used.

[rogermb]

Sorry for taking so long to respond, got kinda wrapped up in IRL stuff.

thread dump about 18 hours later (after 1 reconnect):

That looks pretty normal. The total number of threads that have been spawned over these 18 hours is rather high (2500+), but most of them finished executing and only around 20 threads are still alive (counting VM-internal threads), so there's no way you should get an OutOfMemoryError from that.

Happened about 2 mins ago: Out of nothing the bot used 100% of my cpu. I wasn't able to create a thread dump at this point because all of my system resources were used.

Well that's rather strange. It's difficult to tell what's going on without logs or your complete code, though :/

[PeXArtZ]

Alright, heres my log: https://hastebin.com/rizafidoza.sql

Load.java: https://hastebin.com/inecowetew.cs

Console.java: https://hastebin.com/exevoqanow.java

AFKMover.java: https://hastebin.com/gojiricica.java

ChatBot.java: https://hastebin.com/leboselohu.js

Events.java: https://hastebin.com/xuxeketexi.java

OnlineCount.java: https://hastebin.com/wupahulexu.java

Support.java: https://hastebin.com/olofaradob.java

TimerFunctions.java: https://hastebin.com/hutulupati.java

WelcomeMessage: https://hastebin.com/bukemaqugo.java

Data.java: https://hastebin.com/asehewuxad.java

MessageStrings is just a simple Enum, so I think theres no need to post it here.

[rogermb]

So sorry for taking so long to get back to you, I just had one hell of a busy week.

I looked through the code, and everything looks to be okay. The log is a bit strange - you're really getting disconnected quite often. especially starting around "2019-03-10 22:06:06.381". And not just that - you're getting disconnected immediately after joining the server. I'm assuming that that's the point where the threads start to go wild?

I wonder, could it be that you're somehow getting flood banned from the server? If your query runs on a different IP than your TS3 server (i.e. if you're not connecting to "localhost"), you need to add the IP of your server query to the "query_ip_whitelist.txt" file in your TS3 server's installation directory to be able to use the FloodRate.UNLIMITED flood rate.

Also, from a threading perspective: you only seem to be creating a constant number of threads yourself, so that flood of threads has to come from this API's worker thread pool. Now, the question is, what are these threads doing?

For that, there are just two possibilities. 1: they're reconnecting, or 2: they're running your event listeners. In either case, turning on the communications logging by calling TS3Config#setEnableCommunicationsLogging(true) should help us figure out what's actually going on.

Again, I'm so sorry for taking so long to respond.

[PeXArtZ]

The bot is running on the same system as the server so there is no way its getting flood banned. I ran the bot for ~30 minutes. Heres the log: https://drive.google.com/file/d/1QpvxH0kvIGMxfSBrt8pHJRfub6ULtjyI/view?usp=sharing

[rogermb]

Hmm, this log file looks perfectly normal. I mean, you're sending commands quite often, but that shouldn't really cause any problems. Hmm :/