debora-ito
commented
4 years ago @brainstorm You should not see version conflicts of jackson dependencies if you're only using the SDK. Can you provide the dependency tree with conflict resolution?
Closed brainstorm closed 4 years ago
debora-ito
commented
4 years ago @brainstorm You should not see version conflicts of jackson dependencies if you're only using the SDK. Can you provide the dependency tree with conflict resolution?
brainstorm
commented
4 years ago Thanks for the response @debora-ito, here is the ./gradlew dependencies output:
brainstorm
commented
4 years ago And here's the --scan version of that, for easier/closer inspection: https://scans.gradle.com/s/sbxh7xvpm6hze
debora-ito
commented
4 years ago This is for explicitly adding jackson-core-2.11.0 dependency? Can you run without adding jackson-core-2.11.0?
brainstorm
commented
4 years ago Yes, that was with the jackson-core-2.11.0 specified, here's the scan without it:
https://scans.gradle.com/s/thap7khieufku
debora-ito
commented
4 years ago Thank you for the scans, I don't see any version conflicts of jackson-core and I don't know what else could it be. Have you tried cleaning up the dependency cache and forcing gradle to download them again?
Update: I edited your comment with the dependency tree, made it collapsible, makes it easier to scroll down the thread.
brainstorm
commented
4 years ago Thanks @debora-ito! Yeah, I did that early on before submitting any issue, here's the output:
(base) romanlaptop:igv romanvg$ rm -rf build/*
(base) romanlaptop:igv romanvg$ rm -rf ~/.gradle/
(base) romanlaptop:igv romanvg$ ./gradlew createMacAppDist
Downloading https://services.gradle.org/distributions/gradle-5.5.1-all.zip
..................................................................................................................................
Welcome to Gradle 5.5.1!
Here are the highlights of this release:
- Kickstart Gradle plugin development with gradle init
- Distribute organization-wide Gradle properties in custom Gradle distributions
- Transform dependency artifacts on resolution
For more details see https://docs.gradle.org/5.5.1/release-notes.html
Starting a Gradle Daemon, 1 stopped Daemon could not be reused, use --status for details
> Task :compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
BUILD SUCCESSFUL in 7m 16s
5 actionable tasks: 5 executedThen I'm getting the following runtime issue as described in https://github.com/aws/aws-sdk-java/issues/1120#issuecomment-635132880, right after going Amazon->Log In within the IGV app:
(base) romanlaptop:igv romanvg$ ./build/IGV-MacApp-dist/IGV_user.app/Contents/MacOS/IGV
Using system JDK.
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment Corretto-11.0.7.10.1 (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.7.10.1 (build 11.0.7+10-LTS, mixed mode)
WARNING: package com.sun.java.swing.plaf.windows not in java.desktop
WARNING: package sun.awt.windows not in java.desktop
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment Corretto-11.0.7.10.1 (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.7.10.1 (build 11.0.7+10-LTS, mixed mode)
INFO [2020-06-05T09:58:35,588] [DirectoryManager.java:179] [main] IGV Directory: /Users/romanvg/igv
INFO [2020-06-05T09:58:40,375] [Main.java:159] [AWT-EventQueue-0] Startup IGV Version user not_set
INFO [2020-06-05T09:58:40,408] [Main.java:160] [AWT-EventQueue-0] Java 11.0.7 (build 11.0.7+10-LTS) 2020-04-14
INFO [2020-06-05T09:58:40,409] [Main.java:163] [AWT-EventQueue-0] Java Vendor: Amazon.com Inc. https://aws.amazon.com/corretto/
INFO [2020-06-05T09:58:40,411] [Main.java:165] [AWT-EventQueue-0] JVM: OpenJDK 64-Bit Server VM Corretto-11.0.7.10.1
INFO [2020-06-05T09:58:40,412] [DirectoryManager.java:84] [AWT-EventQueue-0] Fetching user directory...
INFO [2020-06-05T09:58:41,133] [Main.java:168] [AWT-EventQueue-0] Default User Directory: /Users/romanvg
INFO [2020-06-05T09:58:41,157] [Main.java:169] [AWT-EventQueue-0] OS: Mac OS X 10.15.4 x86_64
INFO [2020-06-05T09:58:41,191] [Main.java:220] [Thread-2] Unknown version: user
INFO [2020-06-05T09:58:43,633] [AmazonUtils.java:69] [AWT-EventQueue-0] AWS configuration found. AWS support enabled under 'Amazon' menu
INFO [2020-06-05T09:58:49,355] [GenomeManager.java:191] [pool-3-thread-1] Loading genome: /Users/romanvg/igv/genomes/hg38.genome
INFO [2020-06-05T09:59:15,166] [GenomeComboBox.java:90] [AWT-EventQueue-0] Enter genome combo box
INFO [2020-06-05T09:59:15,214] [GenomeManager.java:283] [pool-3-thread-1] Genome loaded. id= hg38
INFO [2020-06-05T09:59:15,229] [CommandListener.java:120] [Thread-4] Listening on port 60151
Paint All
Paint All
Paint All
Paint All
Paint All
Paint All
INFO [2020-06-05T09:59:40,726] [CommandListener.java:186] [Thread-4] GET /oauthCallback?code=185a3f83-c2ee-4bd0-a1a3-4da98b364219&state=095cf757-f9cb-4b60-bfcb-ac3a885b275b HTTP/1.1
ERROR [2020-06-05T09:59:43,555] [DefaultExceptionHandler.java:49] [Thread-4] Unhandled exception
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
at software.amazon.awssdk.protocols.json.internal.AwsStructuredPlainJsonFactory.<clinit>(AwsStructuredPlainJsonFactory.java:34) ~[aws-json-protocol-2.13.24.jar:?]
at software.amazon.awssdk.protocols.json.BaseAwsJsonProtocolFactory.getSdkFactory(BaseAwsJsonProtocolFactory.java:157) ~[aws-json-protocol-2.13.24.jar:?]
at software.amazon.awssdk.protocols.json.BaseAwsJsonProtocolFactory.<init>(BaseAwsJsonProtocolFactory.java:73) ~[aws-json-protocol-2.13.24.jar:?]
at software.amazon.awssdk.protocols.json.AwsJsonProtocolFactory.<init>(AwsJsonProtocolFactory.java:30) ~[aws-json-protocol-2.13.24.jar:?]
at software.amazon.awssdk.protocols.json.AwsJsonProtocolFactory$Builder.build(AwsJsonProtocolFactory.java:46) ~[aws-json-protocol-2.13.24.jar:?]
at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClient.<init>(DefaultCognitoIdentityClient.java:123) ~[cognitoidentity-2.13.24.jar:?]
at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClientBuilder.buildClient(DefaultCognitoIdentityClientBuilder.java:29) ~[cognitoidentity-2.13.24.jar:?]
at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClientBuilder.buildClient(DefaultCognitoIdentityClientBuilder.java:22) ~[cognitoidentity-2.13.24.jar:?]
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:124) ~[sdk-core-2.13.24.jar:?]
at org.broad.igv.util.AmazonUtils.GetCognitoAWSCredentials(AmazonUtils.java:126) ~[igv.jar:?]
at org.broad.igv.google.OAuthProvider.fetchTokens(OAuthProvider.java:229) ~[igv.jar:?]
at org.broad.igv.google.OAuthProvider.setAuthorizationCode(OAuthProvider.java:173) ~[igv.jar:?]
at org.broad.igv.google.OAuthProvider.setAuthorizationCode(OAuthProvider.java:166) ~[igv.jar:?]
at org.broad.igv.google.OAuthUtils.setAuthorizationCode(OAuthUtils.java:167) ~[igv.jar:?]
at org.broad.igv.batch.CommandListener.processClientSession(CommandListener.java:203) ~[igv.jar:?]
at org.broad.igv.batch.CommandListener.run(CommandListener.java:126) ~[igv.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonFactory
at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
... 17 moreThe offending line is this, the builder, which, from what I gather has nothing to do with the JSON itself, but it's Jackson class not loading properly... OTOH, ./gradlew run does not lead to this behaviour, so I believe it's just an app-level classpath problem on the Mac Bundle...
@davideby (original maintainer for that createMacAppDist gradle target), do you have any idea what this might be? Some .jar/classpath bundling issue when running createMacAppDist gradle target?
Do you do additional post-processing in your release engineering process to ship the IGV Mac App that are not reflected in the gradle task itself? This is blocking PR https://github.com/igvteam/igv/pull/811 at the moment, so I would like to ship this relatively fast to avoid users filing issues about the AWS functionality :-S
Cheers!
/cc @jrobinso @reisingerf
davideby
commented
4 years ago Well, yes and no. The Gradle file pulls together everything for that Mac.app build and zips it. We run it on our Jenkins server but all that really does is call the Gradle build with some properties set (paths to the embedded JREs, timestamp, version & build #, etc). Nothing very special happens there.
After that, I manually sign and notarize (as of the latest release) the bundle. It's conceivable that the notarization process would change some things, but it shouldn't make it go from not-running to running (though the reverse is definitely possible).
The jars and classpath/modulepath should be the same across all the bundles because the Gradle script is pulling them all from the same place (the IGV-dist directory). It's possible that there's a launcher script error, of course, but we're not hearing any reports of that in Production so it seems less likely (though obviously it could just be that no one hit this code path).
What I'd suggest is trying to launch at the CLI from that IGV-dist directory using either the igv.sh launcher or the igv.command launcher (the latter is similar but not identical to the mac.app launcher). My bet is against this being a Mac.app-specific bug but you might want to confirm that so you're not looking for something platform-specific if it's a bug in the general case.
Are you adding any new jars in this update? If those are referenced in the code then you'll need to add them to module-info.java (in the top-level "default" package). This, or something else module-related, is by far the most likely cause if you don't see any conflicts anywhere and you think the jars are in the right place.
You might also try a totally clean build with ./gradlew clean createMacAppDist. Pretty obvious, I know, but worth mentioning. We always do this in Jenkins, FWIW.
Lastly, one other issue I had when porting IGV from Java 8 up through Java 9,10,11 was that lots of these old-school libraries had bundled copies of other things within their jars (so-called "uber jars" or "fat jars") and that those often led to conflicts. So check if there is some other jar bundling a different version of this lib. You would think that wouldn't lead to a ClassNotFoundException but these things come out in strange ways sometimes.
Sorry for the length. I hope some of this helps.
brainstorm
commented
4 years ago Thanks for the pointers, @davideby, made me think a bit about it and found the issue on https://github.com/igvteam/igv/pull/811/commits/90edd1e8ec9eb123eed45f8e1daf7e12227315af#diff-b2e60e6dba0d888da1bf8248dbbc287dR43 ... although it shouldn't be needed to put that AWS transitive dependency on modules-info.java, it looks like it solved the NoClassDefFoundError issue.
@debora-ito Any idea why that fastxml require is required in module-info.java now? It definitely wasn't needed before and IGV does not use it anywhere, so it only comes from the AWS SDK :-S
/cc @reisingerf
debora-ito
commented
4 years ago @brainstorm if you're using a SDK service module that uses JSON, it will have a dependency on jackson-core.
brainstorm
commented
4 years ago Of course, @debora-ito, I agree! I only disagree on the bit that I have to define it explicitly on modules-info.java when it's an internal module dependency for AWS SDK.
The way I see it is that I should just be concerned about requiring/importing/using AWS SDK modules in module-info.java on my application.
Having to define any of the internal dependencies of any arbitrary AWS SDK module kind of defeats the purpose of a (self-contained) module, unless I'm missing some other crucial design or implementation details here? :-S
On top of that, failing at runtime instead of at compile time is pretty bad since it becomes tricky to explore all code paths in search for NoDefClassFound errors for things like "event processing" or "XML processing" with new AWS functionality I might introduce in the future? :/
debora-ito
commented
4 years ago @brainstorm I understand your frustration now. The SDK does not generate modules-info, we use the Automatic-Module-Name manifest entry, which does not prevent problems with transitive dependencies unfortunately. That's why you need to add the jackson-core dependency.
Using Automatic-Module-Name is just one step towards Java 9 modularization support, and we have one feature request that is kind of related to this: https://github.com/aws/aws-sdk-java-v2/issues/59.
I'll close this for now. Feel free to reopen if you have any further questions.
brainstorm
commented
4 years ago Thanks @debora-ito for the attention and making sure this is being tracked, much appreciated! :)
Please see the following AWS SDK Java v1 issue comment for complete bug report, context and example code: https://github.com/aws/aws-sdk-java/issues/1120#issuecomment-635132880 (opening this issue over here since I'm observing this issue while using AWS SDK for Java v2):
In a nutshell:
And when trying to bump the jackson xml version separatedly (instead of relying on the AWS SDK jar dependency):
None of this happened before with SDK version 2.8.5.
/cc @reisingerf