search

newrelic / newrelic-java-agent

The New Relic Java agent
Apache License 2.0
202 stars 143 forks source link

Detected 3 deadlocked threads #8.2.0 #1992

Closed aimhighrana closed 2 months ago

aimhighrana commented 3 months ago

Description

Hi, I am getting this below in logs and any jboss application is not responding.

2024-07-28T22:57:58,757+0000 [107 161] com.newrelic INFO: Using default event ingest URI: https://insights-collector.newrelic.com/v1/accounts/events
2024-07-28T22:57:58,758+0000 [107 161] com.newrelic INFO: Real user monitoring is enabled for application https://xxx.xx/business-central. Auto instrumentation is enabled.
2024-07-28T22:59:25,661+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:04:25,666+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:09:25,672+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:14:25,676+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:19:25,683+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:24:25,689+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:29:25,693+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:34:25,699+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:39:25,706+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:44:25,712+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:49:25,718+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:54:25,725+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-28T23:59:25,731+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:04:25,738+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:09:25,745+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:14:25,752+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:19:25,757+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:24:25,765+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:29:25,771+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:34:25,777+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:39:25,782+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:44:25,786+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:49:25,793+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:54:25,800+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T00:59:25,807+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:04:25,813+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:09:25,819+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:14:25,826+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:19:25,832+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:24:25,838+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:29:25,844+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:34:25,850+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:39:25,855+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:44:25,860+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:49:25,866+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:54:25,871+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T01:59:25,878+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:04:25,882+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:09:25,888+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:14:25,893+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:19:25,900+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:24:25,905+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:29:25,911+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:34:25,916+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:39:25,921+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:44:25,928+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:49:25,933+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:54:25,940+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T02:59:25,947+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:04:25,953+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:09:25,959+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:14:25,966+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:19:25,970+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:24:25,976+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:29:25,982+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:34:25,987+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:39:25,992+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:44:25,998+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:49:26,003+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:54:26,008+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T03:59:26,014+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:04:26,021+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:09:26,027+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:14:26,031+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:19:26,038+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:24:26,042+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:29:26,050+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:34:26,057+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:39:26,063+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:44:26,068+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:49:26,073+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:54:26,079+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T04:59:26,085+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:04:26,091+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:09:26,097+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:14:26,103+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:19:26,109+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:24:26,114+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:29:26,121+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:34:26,126+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:39:26,131+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:44:26,138+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:49:26,144+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:54:26,149+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T05:59:26,156+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:04:26,161+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:09:26,166+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:14:26,172+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:19:26,178+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:24:26,183+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:29:26,189+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:34:26,196+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:39:26,201+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:44:26,208+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:49:26,213+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:54:26,219+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T06:59:26,224+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:04:26,228+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:09:26,234+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:14:26,241+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:19:26,246+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:24:26,251+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:29:26,257+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:34:26,263+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads
2024-07-29T07:39:26,268+0000 [107 37] com.newrelic INFO: Detected 3 deadlocked threads

Steps to Reproduce

Java : registry.access.redhat.com/ubi8/openjdk-11:1.13-1.1653918221 JAVA_OPTS : -noverify -server -Xms24G -Xmx24G -XX:+UseStringDeduplication -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -javaagent:/usr/lib/jvm/newrelic.jar -XX:+UnlockExperimentalVMOptions -XX:+UseZGC

Impact in the application

The application is in running but not accepting any http request as says

image

workato-integration[bot] commented 3 months ago

https://new-relic.atlassian.net/browse/NR-296337

meiao commented 2 months ago

@aimhighrana can you provide more info? Which version of JBoss? Any application? Are there common frameworks/libraries to the applications? Which versions?

It looks like you are using agent 8.2.0, which is no longer supported. Can you upgrade it?

meiao commented 2 months ago

Also, it looks like you are using JBoss' Byteman, which seems to be another agent. The New Relic Java agent may have conflicts with it. Can you try running without Byteman?

aimhighrana commented 2 months ago

Hi @meiao Thanks for the reply , The application is so called , JBPM 7.73.0.Final (Business central). Which is one of the opensource from Jboss and Kie community. Release notes

Can you upgrade it? I would say no for a moment as this issue is coming on production instance and we can't upgrade it without compatibility testing.

aimhighrana commented 2 months ago

@aimhighrana can you provide more info? Which version of JBoss? Any application? Are there common frameworks/libraries to the applications? Which versions?

It looks like you are using agent 8.2.0, which is no longer supported. Can you upgrade it?

Not enabled as such , is this default for Jboss wildfly applications ?

meiao commented 2 months ago

I believe your wanted to quote the post about byteman in your last message. I think you are using it because of the system property you set -Djboss.modules.system.pkgs=org.jboss.byteman.

aimhighrana commented 2 months ago

Thanks @meiao , let me test without byteman .

deleonenriqueta commented 2 months ago

We've had no response for 2 weeks, so this issue will likely be closed as wontfix

aimhighrana commented 2 months ago

I upgraded newrelic java agent to 8.X and removed byteman

Seems its working .. haven't seen the same issue since 15 days ..

Closing this now Thanks