search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 722 forks source link

JVMVRFY038 invokespecial on invalid target; class=die/verwandlung/GregorSamsa, #7684

Closed planetf1 closed 4 years ago

planetf1 commented 5 years ago

Java -version output

openjdk version "11.0.5" 2019-10-15 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10) Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.17.0, JRE 11 Mac OS X amd64-64-Bit Compressed References 20191016_371 (JIT enabled, AOT enabled) OpenJ9 - 77c1cf708 OMR - 20db4fbc JCL - 2a7af5674b based on jdk-11.0.5+10)

Summary of problem

When building ODPi egeria (https://github.com/odpi/egeria) master, the build fails on a maven step using the Apache RAT plugin 0.13 with

An API incompatibility was encountered while executing org.apache.rat:apache-rat-plugin:0.13:check: java.lang.VerifyError: JVMVRFY038 invokespecial on invalid target; class=die/verwandlung/GregorSamsa, method=template$dot$0(Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V, pc=698 [ERROR] Exception Details: [ERROR] Location: [ERROR] die/verwandlung/GregorSamsa.template$dot$0(Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V @698: JBinvokespecial [ERROR] Reason: [ERROR] The method invoked via invokespecial is invalid.

This was when attempting to reproduce #7539 (my environment doesn't yet have the fix), but I had also cleaned up my build environment by removing all of my ~/.m2 - as well as pulled current updates from master. I had previously only seen this failure with a nightly build

Diagnostic files

Full log from the exceptions is posted to https://gist.github.com/f640872cc6608793e8988f54c68363d7

DanHeidinga commented 5 years ago

@planetf1 Do you have the .class file that causes the verifyerror? We need to see the bytecodes to see why the check in the verifier is failing.

planetf1 commented 5 years ago

The maven dependency I'm using is from https://repo1.maven.org/maven2/org/apache/rat/apache-rat-plugin/0.13/ ie with the jar at https://repo1.maven.org/maven2/org/apache/rat/apache-rat-plugin/0.13/apache-rat-plugin-0.13.jar

but I'm unclear which class specifically you need ?

DanHeidinga commented 5 years ago

The VerifyError is flagging a problem in:

die/verwandlung/GregorSamsa.template$dot$0(Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V @698: JBinvokespecial

So I need the die/verwandlung/GregorSamsa class so I can look at the bytecodes for the template$dot$0(Lcom/sun/org/apache/xalan/internal/xsltc/DOM;Lcom/sun/org/apache/xml/internal/dtm/DTMAxisIterator;Lcom/sun/org/apache/xml/internal/serializer/SerializationHandler;I)V method

planetf1 commented 5 years ago

I found a few hits like https://stackoverflow.com/questions/49758534/java-lang-verifyerror-class-gregorsamsa-method-illegal-target-of-jump - or more helpfully perhaps http://www.jguru.com/faq/view.jsp?EID=1340163 ie this is in the XLST processor which I assume is bundled with the jvm?

planetf1 commented 5 years ago

This is an AdoptOpenJDK build

planetf1 commented 5 years ago

also some explanations at https://github.com/eclipse/openj9/issues/6642

DanHeidinga commented 5 years ago

I found a few hits like https://stackoverflow.com/questions/49758534/java-lang-verifyerror-class-gregorsamsa-method-illegal-target-of-jump

That link shows an issue with branch/jump targets and should be fixed in JDK10. According to the java -version above, you're using JDK11 so it may be related to the XLST processor but shouldn't be the same issue.

We need the bytecode to figure out what's going on

planetf1 commented 5 years ago

Can you recommend how to get the byte code? Any ideas? This class is being created dynamically by the XSLT processing as far as I can see (an area I'm unfamiliar with)

DanHeidinga commented 5 years ago

I don't know enough about the XSLT processing to know how to dump the generated files. @hzongaro Are you aware of a -D define or other option to have the XSLT transformer dump the generated classes to disk?

planetf1 commented 5 years ago

Another observation. The exact stage of the build the failure happens on varies. Sometimes it may get all the way through (supposition - since I originally only noticed this on a nightly). For example I ran now and it got 50% further. Still failing with the same report, managing RAT.

Our 'master' code is changing - a few updates constantly through the day, but it may also point to odd corruption perhaps?

planetf1 commented 5 years ago

Only seen on MacOS so far. trying same code/version on linux, but not seen yet.

hzongaro commented 5 years ago

Dan @DanHeidinga, I'm not aware of a -D option or other JVM option that can be specified to have the XSLT processor dump the generated classes. The only way I'm aware of doing it is programmatically.

There is a TransformerFactory.setAttribute method that can be used to specify settings on the XSLT processor. In particular, with XSLTC, there are attributes that can be used to indicate that generated classes should be written to a particular directory or a jar file - in particular generate-translet, destination-directory, jar-name.

The difficulty of course is that @planetf1 might not have access to the source that's processing the XSLT stylesheet in order to set those attributes on the TransformerFactory instance. If they do not, they might have to do something like override the default TransformerFactory implementation by specifying -Djavax.xml.transform.TransformerFactory=<TransformerFactoryImplemetationClass>. This implementation could delegate to com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl, but would also call setAttribute on the underlying implementation to request that the classes by written out, using some sort of sequence number on the destination directories and/or jar names.

planetf1 commented 5 years ago

Thanks for the update, but I think that will be rather tricky. Technically all of this will be open source but it involves the implementation of the rat plugin - not familiar, time...

The unfortunate issue is I was hoping to ensure we can run with J9, but I have hit a variety of issues - being worked through I now, but for now I think I'll have to recommend sticking with hotspot.

pshipton commented 5 years ago

Can we dump a core file when the VerifyError occurs and get the bytecode that way?

"-Xdump:system:events=systhrow,filter=java/lang/VerifyError,request=exclusive+prepwalk"

planetf1 commented 5 years ago

Typically I cannot get a failure this morning. I've set MAVEN_OPTS with the above, so if/when I get a failure hopefully I'll find a core (also added a ulimit -c unlimited in my build invocation).

planetf1 commented 5 years ago

After around 8 attempts I had another failure with

MAVEN_OPTS=-Xdump:system:events=systhrow,filter=java/lang/VerifyError,request=exclusive+prepwalk
10:16:06,717 [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (rat-check) on project it-infrastructure-spring: Could not load the translet class 'die.verwandlung.GregorSamsa'.: JVMCFRE068 class name is invalid; class=, offset=0 -> [Help 1]

Also ulimit -a shows

-c: core file size (blocks)         unlimited

However I can't see any core file in the current directory from where the sw was launched, nor the relevant maven directory nor indeed anywhere down the tree

DanHeidinga commented 5 years ago

Can you set the OPENJ9_JAVA_OPTIONS environment variable to "-Xdump:system:events=systhrow,filter=java/lang/VerifyError,request=exclusive+prepwalk"? I'm not sure how MAVEN_OPTS gets passed through

planetf1 commented 5 years ago

I will try that too, though I've used it for other JVM options and it appears to pass through fine (or through JAVA_TOOL_OPTIONS). But will set OPENJ9_JAVA_OPTIONS

planetf1 commented 5 years ago

Needed to do a build today and hit the issue again - those options are set but I still see no evidence of a core?

15:23:36,466 [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (rat-check) on project governance-daemon-connectors: Could not load the translet class 'die.verwandlung.GregorSamsa'.: JVMCFRE068 class name is invalid; class=, offset=0 -> [Help 1]
15:23:36,467 [ERROR]
15:23:36,467 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
15:23:36,467 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
15:23:36,467 [ERROR]
15:23:36,467 [ERROR] For more information about the errors and possible solutions, please read the following articles:
15:23:36,467 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
15:23:36,467 [ERROR]
15:23:36,467 [ERROR] After correcting the problems, you can resume the build with the command
15:23:36,468 [ERROR]   mvn <args> -rf :governance-daemon-connectors
➜  egeria git:(master) find . -name core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/bower/lib/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/sinon/lib/sinon/util/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/core-js/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/core-js/library/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/polymer-analyzer/lib/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/@babel/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/web-component-tester/node_modules/sinon/lib/sinon/util/core
./open-metadata-implementation/user-interfaces/access-services-user-interface/tools/node_modules/polymer-cli/node_modules/rx/ts/core
➜  egeria git:(master) env | grep J9
OPENJ9_JAVA_OPTIONS=-Xdump:system:events=systhrow,filter=java/lang/VerifyError,request=exclusive+prepwalk
➜  egeria git:(master) find . -name '*javacore*'
➜  egeria git:(master)
planetf1 commented 5 years ago 
➜  egeria git:(master) ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         unlimited
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2784
-n: file descriptors                256
planetf1 commented 5 years ago

New information - I just hit the same issue on linux (Ubuntu 19.10) - though I did not have the same environment set there to capture debug:

Again

15:30:35,183 [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (rat-check) on project platform-services: Could not load the translet class 'die.verwandlung.GregorSamsa'.: JVMCFRE068 class name is invalid; class=, offset=0 -> [Help 1]
➜  egeria git:(master) java -version
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.5+10)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.17.0, JRE 11 Linux amd64-64-Bit Compressed References 20191016_358 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 2a7af5674b based on jdk-11.0.5+10)
planetf1 commented 5 years ago

No sign of cores on linux either

pshipton commented 5 years ago

Try just -Xdump:system:events=systhrow,request=exclusive+prepwalk without the filter. I don't think any other exceptions would be thrown from the VM normally, and the class name is invalid may be a ClassFormatError instead of a VerifyError.

planetf1 commented 5 years ago

Tried that, and this time my maven build failed in 27s which is pretty impressive. Same error. No sign of a core. This was on ubuntu. Clearly the RAT plugin is the cause, yet I've never seen this with hotspot, nor with earlier J9 builds which I've been using intermittently since macOS came out from adoptopenjdk. The version of the rat plugin is current, and has been around for about a year.

DanHeidinga commented 5 years ago

JVMCFRE068 class name is invalid;

This is a ClassFormatError. Can you try with "-Xdump:system:events=systhrow,filter=java/lang/ClassFormatError,request=exclusive+prepwalk" in the OPENJ9_JAVA_OPTIONS?

We may need to hack a VM that dumps the classfile on a verifyerror/classformaterror....

pshipton commented 5 years ago

I expect that if -Xdump:system:events=systhrow,request=exclusive+prepwalk didn't work, then the version with the filter won't work either. The one without the filter should dump a core for any systhrow.

Another possibility is the core is going to an unexpected place. If the -Xdump command triggers, there are messages printed to stderr like the following.

JVMDUMP039I Processing dump event "throw", detail "java/lang/ClassNotFoundException" at 2019/11/06 16:46:23 - please wait.
JVMDUMP032I JVM requested System dump using 'peter/openj9/jdk8u212-b04/core.20191106.164623.20183.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.20200.

JVMDUMP010I System dump written to peter/openj9/jdk8u212-b04/core.20191106.164623.20183.0001.dmp
JVMDUMP013I Processed dump event "throw", detail "java/lang/ClassNotFoundException".
planetf1 commented 5 years ago

I don't see any messages like this, but of course I am running a maven build composed of 200+ modules, and within that the rat plugin is being invoked >200 times. The failure seems to happen randomly. I have no idea if anything about that environment is masking properties sent to the jvm. So there's a fair gap between the user perceived problem and the nitty gritty in the jvm. I can try to repeatedly call rat directly, but unsure I'll be able to repro.

I could try to go back to 11.0.4 and ensure the issue doesn't happen there, again. Would that help given any thoughts on changes going to 11.0.5 (or the packaging from adoptopenjdk)

pshipton commented 5 years ago

Did you try searching the file system for core.20191106.*?

planetf1 commented 5 years ago

No sign of a filename of the form core.20191106.* (especially on linux, from / )

pshipton commented 5 years ago

Try running java -Xcheck:dump and see what that shows. It validates the OS is setup properly for collecting dumps, although not sure how well it works on Mac, it may work better on Linux.

planetf1 commented 5 years ago

On MacOS, that had zero effect ie

java -Xcheck:dump -jar ~/src/egeria/open-metadata-implementation/user-interfaces/ui-chassis/ui-chassis-spring/target/ui-chassis-spring-1.2-SNAPSHOT.jar
 ODPi Egeria

^^^ that first line is our app starting up

On linux (where I did also see issues)

➜  egeria git:(master) ✗ java -Xcheck:dump -jar ~/src/egeria/open-metadata-implementation/user-interfaces/ui-chassis/ui-chassis-spring/target/ui-chassis-spring-1.2-SNAPSHOT.jar
JVMJ9VM135W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P" specifies that core dumps are to be piped to an external program. The JVM may be unable to locate core dumps and rename them.
 ODPi Egeria
pshipton commented 5 years ago

What version of Linux is it? It seems to be telling us the core may not be called core.20191106. since it couldn't be renamed, but might be core.

planetf1 commented 4 years ago 
➜  egeria git:(master) ✗ cat /etc/*release* | grep VERSION
VERSION="19.10 (Eoan Ermine)"
VERSION_ID="19.10"
VERSION_CODENAME=eoan
➜  egeria git:(master) ✗ uname -a
Linux dev1 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Though note that reference to core only happened when I 'broke' the jvm in another way (I experimented with class cache & had run out of disk space). - so it proves the jvm can dump, but in this case it was unrelated to the xslt issue.

planetf1 commented 4 years ago

My intent is to avoid this problem until 11.0.6 is released (skip rat), then I will retest and report back - since we don't seem to have been able to capture further debugging info.

kgibm commented 4 years ago

On Linux, the JVMJ9VM135W message means that the system has a core redirection configured into the /usr/share/apport/apport program, which may be deleting, renaming, and/or truncating the core from my experience. Try sudo systemctl stop apport.service and restart Java and see if JVMJ9VM135W is gone and then reproduce.

planetf1 commented 4 years ago

Note - I was able to use nightly builds from 28 Nov onwards on windows, mac, linux & no longer see this issue. I had been disabling this plugin (-Drat.skip=true) but am now back at default configuration. I intent to continue using the nightly some of the time now, and suggest closing, and I'll reopen if it resurfaces.

planetf1 commented 4 years ago

I had this again for the first time today under MacOS Catalina 19C56a with this build

openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+4-201911261910)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-58a679f1f, JRE 11 Mac OS X amd64-64-Bit Compressed References 20191126_392 (JIT enabled, AOT enabled)
OpenJ9   - 58a679f1f
OMR      - bad5f71b7
JCL      - e34da885d4 based on jdk-11.0.6+4)

No sign of any core.

pshipton commented 4 years ago

I'll re-open and assign to the next milestone.

kgibm commented 4 years ago

@planetf1 I recently wrote an article on how to configure or disable Linux core processing: https://publib.boulder.ibm.com/httpserv/cookbook/Troubleshooting-Troubleshooting_Java.html#Troubleshooting-Troubleshooting_Java-Core_Dumps-Ensure_core_piping_is_configured_properly_or_disabled_on_Linux

It is preferable to completely disable piping when running J9 JVMs because when the J9 JVM produces a core dump or handles a crash, it will post-process the core dump to rename it with additional information and also add some additional details into the core dump. The J9 JVM is not able to do this post-processing if a pipe program processes the core dump first. However, disabling piping will disable piping for all processes, so keep that in mind.

planetf1 commented 4 years ago

I primarily run on MacOS which is the environment where I am currently seeing this - for example I had this again just now

I tried 'java -Xcheck:dump' but that by itself reports a usage error, and combined with -jar xxx doesn't elicit any additional output on MacOS

I tried adding back the options to generate a dump

OPENJ9_JAVA_OPTIONS=-Xshareclasses:name=bld2,persistent,fatal -DargLine=-Xshareclasses:none -Xdump:system:events=systhrow,request=exclusive+prepwalk

but then just get

➜  egeria git:(danielaotelea-supported_types) bld
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/NoClassDefFoundError" at 2019/12/10 11:39:46 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jonesn/IdeaProjects/egeria/core.20191210.113946.29609.0001.dmp' in response to an event
JVMDUMP012E Error in System dump: The core file created by child process with pid = 29752 was not found. Expected to find core file with name "/cores/core.29752"
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/NoClassDefFoundError".
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/NoClassDefFoundError" at 2019/12/10 11:39:52 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jonesn/IdeaProjects/egeria/core.20191210.113952.29609.0002.dmp' in response to an event
JVMDUMP012E Error in System dump: The core file created by child process with pid = 29778 was not found. Expected to find core file with name "/cores/core.29778"
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/NoClassDefFoundError".
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/NullPointerException" at 2019/12/10 11:39:58 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jonesn/IdeaProjects/egeria/core.20191210.113958.29609.0003.dmp' in response to an event
JVMDUMP012E Error in System dump: The core file created by child process with pid = 29814 was not found. Expected to find core file with name "/cores/core.29814"
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/NullPointerException".
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/NullPointerException" at 2019/12/10 11:40:04 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jonesn/IdeaProjects/egeria/core.20191210.114004.29609.0004.dmp' in response to an event
^C

The core files are NOT generated, even with 'ulimit -c unlimited' - nothing at the paths given or under /cores

I've therefore unset the option again, and gone back to skipping the execution of apache rat - this is the only component so far I've seen the error in, but it occurs fairly often in my environment

DanHeidinga commented 4 years ago

Is /cores world writable? With the Catalina OS update the default changed to only be writeable by root.

planetf1 commented 4 years ago

No, I cannot write to /cores. I am using Catalina.

I've now made that dir writeable, and a simple maven build resulted in an immediate core (just running maven). At least that's good. > 2GB.

I then added the filter, and will now run a build. It appears to be progressing, so will look out for the originally reported problem and see if we get a core.........

planetf1 commented 4 years ago

Finally I had a core on MacOS

18:11:56,339 [INFO] --- apache-rat-plugin:0.13:check (rat-check) @ cohort-registry-store-connectors ---
18:11:56,342 [INFO] Added 3 additional default licenses.
18:11:56,342 [INFO] Enabled default license matchers.
18:11:56,342 [INFO] Added 2 custom approved licenses.
18:11:56,342 [INFO] Will parse SCM ignores for exclusions...
18:11:56,342 [INFO] Finished adding exclusions from SCM ignore files.
18:11:56,342 [INFO] 63 implicit excludes (use -debug for more details).
18:11:56,343 [INFO] 38 explicit excludes (use -debug for more details).
18:11:56,349 [INFO] 2 resources included (use -debug for more details)
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/ClassFormatError" at 2019/12/11 18:11:56 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jonesn/IdeaProjects/egeria/core.20191211.181156.33589.0001.dmp' in response to an event
JVMDUMP010I System dump written to /Users/jonesn/IdeaProjects/egeria/core.20191211.181156.33589.0001.dmp
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/ClassFormatError".

The file is 7.2GB which surprised me - please advise how you want it made available... or what should be run on it.

To clarify java version is a nightly build 

openjdk version "11.0.6" 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+4-201911261910)
Eclipse OpenJ9 VM AdoptOpenJDK (build master-58a679f1f, JRE 11 Mac OS X amd64-64-Bit Compressed References 20191126_392 (JIT enabled, AOT enabled)
OpenJ9   - 58a679f1f
OMR      - bad5f71b7
JCL      - e34da885d4 based on jdk-11.0.6+4)

I can of course update/revert to GA, but it's taken a while to get the dump you asked for, and here it is...

Have moved for safety:

-r--------  1 jonesn  wheel  7585607680 11 Dec 18:12 /Users/jonesn/etc/core.20191211.181156.33589.0001.dmp
planetf1 commented 4 years ago

Any of this help?

➜  ~ lldb -c /Users/jonesn/etc/core.20191211.181156.33589.0001.dmp
(lldb) target create --core "/Users/jonesn/etc/core.20191211.181156.33589.0001.dmp"
Core file '/Users/jonesn/etc/core.20191211.181156.33589.0001.dmp' (x86_64) was loaded.
(lldb) thread list
Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff6b2ad7fa libsystem_kernel.dylib`__pthread_kill + 10, stop reason = signal SIGSTOP
(lldb) thread backtrace
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x00007fff6b2ad7fa libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff6b36abc1 libsystem_pthread.dylib`pthread_kill + 432
    frame #2: 0x000000000bb15123 libj9prt29.dylib`omrdump_create + 275
    frame #3: 0x000000000bb7f2d9 libj9dmp29.dylib`doSystemDump + 297
    frame #4: 0x000000000bb83f55 libj9dmp29.dylib`protectedDumpFunction + 21
    frame #5: 0x000000000bb16505 libj9prt29.dylib`omrsig_protect + 949
    frame #6: 0x000000000bb83b56 libj9dmp29.dylib`runDumpAgent + 790
    frame #7: 0x000000000bb99c86 libj9dmp29.dylib`triggerDumpAgents + 1030
    frame #8: 0x000000000bb9a438 libj9dmp29.dylib`rasDumpHookExceptionSysthrow + 120
    frame #9: 0x000000000baea8ef libj9hookable29.dylib`J9HookDispatch(J9HookInterface**, unsigned long, void*) + 415
    frame #10: 0x000000000cca0b25 libj9vm29.dylib`internalSetCurrentExceptionWithCause + 1813
    frame #11: 0x000000000cc9f606 libj9vm29.dylib`setCurrentExceptionUTF + 230
    frame #12: 0x000000000cdd4e67 libj9vm29.dylib`internalDefineClass + 3783
    frame #13: 0x000000000eb0339f libjclse29.dylib`defineClassCommon + 1791
    frame #14: 0x000000000eaf5b3d libjclse29.dylib`Java_java_lang_ClassLoader_defineClassImpl + 173
    frame #15: 0x000000002e9fc6a9
    frame #16: 0x000000000cc8fc42 libj9vm29.dylib`sidecarInvokeReflectMethodImpl + 1474
    frame #17: 0x000000000cc904c1 libj9vm29.dylib`sidecarInvokeReflectMethod + 129
    frame #18: 0x000000000eb34fcd libjclse29.dylib`JVM_InvokeMethod_Impl + 109
    frame #19: 0x000000002e5a1760
    frame #20: 0x000000000cc8f5a9 libj9vm29.dylib`runCallInMethod + 745
    frame #21: 0x000000000cca69cd libj9vm29.dylib`gpProtectedRunCallInMethod(void*) + 125
    frame #22: 0x000000000bb16505 libj9prt29.dylib`omrsig_protect + 949
    frame #23: 0x000000000cd8623a libj9vm29.dylib`gpProtectAndRun + 202
    frame #24: 0x000000000cca6947 libj9vm29.dylib`gpCheckCallin + 71
    frame #25: 0x000000000cca6251 libj9vm29.dylib`callStaticVoidMethod + 145
    frame #26: 0x000000000ba447a0 libjli.dylib`JavaMain + 2732
    frame #27: 0x00007fff6b36ae65 libsystem_pthread.dylib`_pthread_start + 148
    frame #28: 0x00007fff6b36683b libsystem_pthread.dylib`thread_start + 15
(lldb)
pshipton commented 4 years ago

Pls compress the core file (it should compress nicely) and share it using your favorite file sharing service. Box should work in this case.

planetf1 commented 4 years ago

Ok. uploading to https://ibm.box.com/s/zjjc76l8o7yac05y5sqvqjywdee0qpx3 (all public open source so protection not needed including the code)

pshipton commented 4 years ago

I don't have access.

planetf1 commented 4 years ago

Ok I'm not that familar with Box. Let's try OneDrive -> https://1drv.ms/u/s!ApVqcIDT57-flq5Z_z3td2RT1aZGpw?e=g8Xf6Q

pshipton commented 4 years ago

For Box, you need to change the permissions and create a shared link. Anyway the OneDrive download worked.

I copied the core to triage/openj9-7684

planetf1 commented 4 years ago

Is there any news on this. I continue to get these exceptions - today was on windows.