search

jenkinsci / jms-messaging-plugin

https://plugins.jenkins.io/jms-messaging
12 stars 38 forks source link

"CI event" build trigger keeps creating threads on Jenkins when given invalid client name #118

Closed wcheang closed 5 years ago

wcheang commented 5 years ago

Hi,

When given an invalid AMQ consumer name, e.g. Consumer.client-name.service.dev.VirtualTopic.eng.something.>, the jenkins job doesn't get triggered, but keeps creating threads non-stop. The thread number grows quickly and eventually eats up all the CPU on Jenkins. Disabling the job does not help eliminate the threads; we had to manually delete the threads on our jenkins monitoring, and actually the disabled job kept generating more threads.

Thanks, Wai

ggallen commented 5 years ago

Wai, I haven't seen this behavior. Can you point me to the Jenkins master and the job so I can take a look?

sochotnicky commented 5 years ago

I actaully haven't seen that behaviour and I think I was noticing what Wai reported here. What I actually saw was that when invalid consumer is defined (for example queue where the client has no permissions to read) - it will result in a Thread that seems to spin on a few points eating a lot of CPU. When that consumer queue is fixed - the old Thread is not cleaned up however it seems. The only way seems to be to go to monitoring and kill it.

I personally haven't seen creation of threads non-stop so perhaps Wai is talking about a separate issue

ggallen commented 5 years ago

@sochotnicky , that's the behavior I have seen as well. If the connection fails, it will keep trying to connect every minute. This is in fact desired behavior as if connectivity is lost for some reason we want it to reconnect when things are fixed.

We could handle the invalid consumer case and just stop the thread since that will never connect without a configuration change. But how would the user ever know about it?

@wcheang , I did check the logs on your Jenkins and from what I can tell it was behaving as "expected" in that it was trying to reconnect every minute. As I already said, I've never seen the behavior you have described with all the threads being created. Can we set up a time to try it on your system so I can see it happening?

sochotnicky commented 5 years ago

I want to be clear though - there is something wrong. The listener threads with wrong queues don't sit idly for 1 minute. They spin on CPU full-tilt. And they don't get cleaned up when config changes.

sochotnicky commented 5 years ago

@ggallen I can prep an example and we can test on our stage Jenkins instance (mostly any time). Will contact separately

wcheang commented 5 years ago

I mixed up the high CPU time on the monitoring page with high thread count. Sorry about the confusion. But like Stano said there is definitely something wrong here since the client is eating up all the CPU.

ggallen commented 5 years ago

OK, I have reproduced this and I see the problem. Now to figure out how to fix it....

I will keep you posted.

ggallen commented 5 years ago

@wcheang , this is fixed in #119 . Would you be willing to test it out?

wcheang commented 5 years ago

Awesome, thanks a lot @ggallen! If there's an update yeah I can pull it in our jenkins and test it out.