Core dump on Linux x64

[ilgrosso]

Running the hotswap profile from https://github.com/apache/syncope/blob/master/fit/core-reference/pom.xml#L1323 I get the core dump below.

Same Maven profile works fine with both Oracle JDK 8 and OpenJDK 8 after install DCEVM as side option.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fca3fcec8a1, pid=8351, tid=8394
#
# JRE version: OpenJDK Runtime Environment (11.0.1.6+7) (build 11.0.1.6+7-201901011600)
# Java VM: Dynamic Code Evolution 64-Bit Server VM (11.0.1.6+7-201901011600, mixed mode, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x5dc8a1]  ClassLoaderData::loader_name_and_id() const+0x1

hs_err_pid8351.log

[skybber]

Has -XX:-CMSClassUnloadingEnabled any effect?

[ilgrosso]

What do you mean? Shall I just remove that and try again?

[skybber]

Yes try it without -XX:+CMSClassUnloadingEnabled. I'm not sure why it is passed without -XX:+UseConcMarkSweepGC.

[ilgrosso]

Tried without -XX:+CMSClassUnloadingEnabled, same result. hs_err_pid16505.log

[ilgrosso]

Humm, even a simple Maven build with -DskipTests generates core dump

[skybber]

May be the problem is existence of hotswap-agent.jar in lib/ that triggers it. But even if you use the agent externally the problem will be back. I will look at this problem more deeply...

[skybber]

I've tried to rename hotswap-agent.jar to hotswap-agent.jar_backup in dcevm lib/hotswap folder and it's not crashing now.

[ilgrosso]

Thanks for update, I'll try that!

[markiewb]

Instead of renaming the agent disable the agent via -XX:+DisableHotswapAgent

It is documented at https://github.com/TravaOpenJDK/trava-jdk-11-dcevm/releases/tag/dcevm-11.0.1%2B7

On Sat, Feb 9, 2019, 11:31 AM Vladimir Dvorak <notifications@github.com wrote:

I've tried to rename hotswap-agent.jar to hotswap-agent.jar_backup in dcevm lib/hotswap folder and it's not crashing now.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/TravaOpenJDK/trava-jdk-11-dcevm/issues/7#issuecomment-462032758, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxWRxczRRoewK3xUqn6GlgHwk1UxsLVks5vLqNsgaJpZM4avfwZ .

[skybber]

Looks there is a bug in jdk11. It is called from Javassist.define method and it should not harm the jvm, there is no significant change in the dcevm code where it crashes.

[skybber]

Really, there was a bug in the jdk11/hotspot! I've fixed it

https://github.com/HotswapProjects/openjdk-jdk11u-dcevm/commit/345dfade41a011911800516177e58470d84441f2

and will make a new TravaOpenJDK release

[skybber]

Could you check https://github.com/TravaOpenJDK/trava-jdk-11-dcevm/releases/tag/dcevm-11.0.1%2B8 please?

[ilgrosso]

No more core dumps, I can build the whole app with maven without issues.

However, when attempting to start Tomcat (via cargo-maven2-plugin), it remains stuck right before the actual Tomcat boot.

It happens the same both with external hotswap agent (1.3.0):

-XX:+DisableHotswapAgent -javaagent:${settings.localRepository}/org/hotswapagent/hotswap-agent/${hotswap.version}/hotswap-agent-${hotswap.version}.jar=autoHotswap=true,disablePlugin=Spring
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Xmx1024m -Xms512m

and with internal hotswap agent:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Xmx1024m -Xms512m

I also tried to remove -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC, still same issue.

Am I doing some stupid mistake here?

[skybber]

I don't think so. I'll look at it. What exact command are you using, please?

[ilgrosso]

From root folder: mvn -T 1C -PskipTests,all

Then, from fit/core-reference: mvn -Photswap,all

[skybber]

I've modified pom.xml in following way:

lada@skybber ~/syncope/fit/core-reference $ git diff
diff --git a/fit/core-reference/pom.xml b/fit/core-reference/pom.xml
index 3063b63..bea20e7 100644
--- a/fit/core-reference/pom.xml
+++ b/fit/core-reference/pom.xml
@@ -1347,9 +1347,8 @@ under the License.
                 <properties>
                   <cargo.jvmargs>
                     -Dspring.profiles.active=embedded
-                    -XXaltjvm=dcevm -javaagent:${settings.localRepository}/org/hotswapagent/hotswap-agent/${hotswap.version}/hotswap-agent-${hotswap.version}.jar=autoHotswap=true,disablePlugin=Spring
                     -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-                    -XX:+CMSClassUnloadingEnabled -Xmx1024m -Xms512m</cargo.jvmargs>
+                    -Xmx1024m -Xms512m</cargo.jvmargs>
                 </properties>
               </configuration>
             </configuration>

then it does "something" - checked by kill -3 PID , jstack doesn't work in hotswap jvm. as I realized.

[ilgrosso]

Ok, this matches one or the multiple attempts I made (see also above): what do you mean by 'something'? I see that cargo does not start Tomcat but hangs indefinitely....

[skybber]

I don't know what it should do, but the process finished in my case. (May be it can be interesting project I'm looking for, for CXF plugin for HA development ...)

[ilgrosso]

Just tried again (sorry for the delay): I still see the same problem, e.g. mvn -Photswap,all (with your changes as above) hangs with last message

[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ syncope-fit-core-reference ---
[INFO] Compiling 82 source files to /home/ilgrosso/work/syncope/syncope2/fit/core-reference/target/test-classes

while it is supposed to start Tomcat via cargo, then to wait with message

[INFO] --- cargo-maven2-plugin:1.7.2:run (default-cli) @ syncope-fit-core-reference ---
[INFO] Press Ctrl-C to stop the container...

It is not clear to me what do you mean by

the process finished in my case

?

[skybber]

In my case the maven mvn -Photswap,all finished after some delay ~30s.

[markiewb]

@ilgrosso: Could you please provide an reproducible example at a public place. IMO "it does (not) work on my computer"-discussions never lead to satisfying results for both sides.

So how about this. Put the minimun failing example at GitLab and use a free Gitlab CO runner with a customized Docker image including your specific Maven-, JDK- and DCEVM-version to run your example. (Or use GitHub/TravisCI)

Once it runs within this infrastructure and it reproduces your issue, @skybber and other maintainers could use this setup against future regressions.

Best regards, Ben

On Wed, Feb 13, 2019, 8:49 AM Vladimir Dvorak <notifications@github.com wrote:

In my case the maven mvn -Photswap,all finished after some delay ~30s.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TravaOpenJDK/trava-jdk-11-dcevm/issues/7#issuecomment-463094505, or mute the thread https://github.com/notifications/unsubscribe-auth/ABxWR8544R8IqtSxXZPovsEfbTELL_Nrks5vM8ONgaJpZM4avfwZ .

[ilgrosso]

@markiewb thanks for input, I'll try to set up a build reproducing this issue.

[ilgrosso]

I set up https://travis-ci.org/ilgrosso/syncope-jdk11-dcevm/ which now works almost fine. It turned out that the actual "hanging" problem was usage of hotswap-agent 1.3.0.

The only issue left is that, with JVM options set as per https://github.com/ilgrosso/syncope-jdk11-dcevm/blob/master/core/pom.xml#L121-L124 it seems that cargo is not able to shutdown Tomcat properly.

[ilgrosso]

Works fine now (except from what reported above, but minor), closing.