Dependencies are working only with jdk8

neworld commented 5 years ago

Using >8 JDK everything is working except dependency resolution:

[kscript] Resolving dependencies...
[kscript]     Resolving "org.openjsse:openjsse:1.1.0"...Exception in thread "main" java.lang.NoClassDefFoundError: org/ietf/jgss/GSSException
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.<clinit>(NettyAsyncHttpProvider.java:177)
    at org.sonatype.aether.connector.async.AsyncRepositoryConnector.getDefaultProvider(AsyncRepositoryConnector.java:246)
    at org.sonatype.aether.connector.async.AsyncRepositoryConnector.getProvider(AsyncRepositoryConnector.java:241)
    at org.sonatype.aether.connector.async.AsyncRepositoryConnector.<init>(AsyncRepositoryConnector.java:154)
    at org.sonatype.aether.connector.async.AsyncRepositoryConnectorFactory.newInstance(AsyncRepositoryConnectorFactory.java:106)
    at org.sonatype.aether.impl.internal.DefaultRemoteRepositoryManager.getRepositoryConnector(DefaultRemoteRepositoryManager.java:346)
    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:453)
    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
    at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:193)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:281)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:186)
    at org.sonatype.aether.impl.internal.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:191)
    at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:333)
    at com.jcabi.aether.Aether.fetch(Aether.java:228)
    at com.jcabi.aether.Aether.resolve_aroundBody2(Aether.java:180)
    at com.jcabi.aether.Aether$AjcClosure3.run(Aether.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:208)
    at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
    at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136)
    at com.jcabi.aether.Aether.resolve(Aether.java:177)
    at com.jcabi.aether.Aether.resolve_aroundBody0(Aether.java:163)
    at com.jcabi.aether.Aether$AjcClosure1.run(Aether.java:1)
    at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
    at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:208)
    at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
    at com.jcabi.aspects.aj.MethodLogger.wrapClass(MethodLogger.java:136)
    at com.jcabi.aether.Aether.resolve(Aether.java:156)
    at kscript.app.DependencyUtilKt.resolveDependenciesViaAether(DependencyUtil.kt:86)
    at kscript.app.DependencyUtilKt.resolveDependencies(DependencyUtil.kt:53)
    at kscript.app.KscriptKt.main(Kscript.kt:158)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
    at org.jetbrains.kotlin.runner.AbstractRunner.run(runners.kt:61)
    at org.jetbrains.kotlin.runner.Main.run(Main.kt:110)
    at org.jetbrains.kotlin.runner.Main.main(Main.kt:120)
Caused by: java.lang.ClassNotFoundException: org.ietf.jgss.GSSException
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 38 more

A first bug was reported #233

jdk8 is 5 years old and some people prefer to use newer versions of jdk.

holgerbrandl commented 5 years ago

Duplicate of #239

neworld commented 5 years ago

Recursive duplicated issue :). @holgerbrandl, could you specify duplicated issue, please? I can find nothing similar by myself and I would like to follow it.

holgerbrandl commented 5 years ago

Let's blame the early Monday morning. :-)

holgerbrandl commented 5 years ago

Let's test this with the recent dev builds from here - https://bintray.com/kotlin/kotlin-dev/kotlin because it seems related to https://youtrack.jetbrains.com/issue/KT-26624

cnusp commented 4 years ago

Any news on that problem? It still does not work (Java 11).

holgerbrandl commented 4 years ago

This week is super packed on my end, but I'll try to look into at the weekend. It goes without saying that PRs are welcome.

qmg-dhamilton commented 4 years ago

I wanted to retest it on 1.3.61 to see if KT-26624 fixes it, however for some reason KScript reports using Kotlin 1.2.41 (via KotlinVersion.CURRENT) even though that version is not (AFAICS) installed on my computer.

Is that version set at compile-time for KScript and immutable for anyone running it?

qmg-dhamilton commented 4 years ago

Sorry - pls ignore. Ironically one of the (already resolved) script dependencies was pulling in the old version of kotlin-stdlib. (So, unsurprisingly I guess KotlinVersion.CURRENT is defined in kotlin-stdlib).

At the moment, it doesn't appear that 1.3.61 resolves the issue.

Is there any way of confirming which version of kotlinc is being used by kscript?

holgerbrandl commented 4 years ago

Sure, which kotlinc. Under the hood, it's just creating a subprocess which is inferring the kotlinc to be used from your PATH. Afaik the only way to overrule this behaviour is declaring a KOTLIN_HOME variable in the shell environment where kscript is launched.

We were discussing if we should rather use the compiler API (see #102), but there were pros and cons but imho no major functional advantage, so the PR is still unresolved.

qmg-dhamilton commented 4 years ago

Thanks. Yes, I can confirm that Java 11 still causes the NoClassDefFoundError for org/ietf/jgss/GSSException on Kotlin 1.3.61

As an alternative approach, I attempted to include the java.security.jgss module via the approaches included in https://github.com/holgerbrandl/kscript/blob/master/examples/java_module_example.kts but none of them solved the issue.

geoand commented 4 years ago

Hi,

I am also seeing the exact same issue (java.lang.NoClassDefFoundError: org/ietf/jgss/GSSException) when using kscript with dependencies in a Github Actions pipeline that runs with Java 11 (the Java 8 version of the pipeline works just fine).

holgerbrandl commented 4 years ago

@qmg-dhamilton I think the problem is that somehow the GSSException class is needed by kscript itself when resolving dependencies via jcabi. Dependencies in a user script such as the java_module_example are unrelated to that issues.

I've filed a ticket in jcabi-aether to narrow down the root cause of the problem. See https://github.com/jcabi/jcabi-aether/issues/105

Feel welcome if you have ideas about how to fix this annoying issue.

holgerbrandl commented 4 years ago

I can also confirm that the issue is still present when using 1.3.61 with java 11 and is most likely a class loader problem (because it just works fine when running kscript directly with java instead of kotlin).

So I think the next step would be to submit a boiled down example as platform ticket to https://youtrack.jetbrains.com/oauth?state=%2Fissues%2FKT

holgerbrandl commented 4 years ago

I've tried 3 paths to fix this issue without sucess: `

Simply preload the missing class for java >8 with ClassLoader.getPlatformClassLoader().loadClass("org.ietf.jgss.GSSException").However this has no effect, because the NettyAsyncHttpProvider seems to resolve via reflection using it's own class loader.
Bump dependency versions --> compile("com.ning:async-http-client:1.9.40") however this fails because it is not longer api-compatible with aether-connector-asynchttpclient-1.13.1.jar.
Rebuild the affected dependency com.ning:async-http-client:1.6.5 to use the java9 approach with getPlatformClassLoader. However, I failed to identify the corresponding code bits because the stacktrace line numbers are somehow not correct.

While doing so, I've noticed the only actual lead again, which is that kscript.app.DependencyUtil#main runs perfectly fine on java11 when being run directly with java but gives the error from above when being run with ´kotlin(via test helperresolve_deps` from ´test_suite.sh´. So I suspect it could be fixed by matching classloading behaviour between kotlin and java. The corresponding ticket is https://youtrack.jetbrains.com/issue/KT-28475 (fyi @ligee)

Feel welcome if you have any ideas for another fix.

CLOVIS-AI commented 4 years ago

Looks like the problem is still there: Kotlin version 1.3.61-release-180 (JRE 13.0.2+8)

And well, based on how jcabi is asking you for money to fix it, I guess it's going to stay this way for some time...

holgerbrandl commented 4 years ago

The good news is that KT-28475 was fixed, so maybe the problem could go away with kotlin 1.4.

@ligee is there any public build available containing thisfix, which I could use for testing?

ligee commented 4 years ago

@holgerbrandl there are some dev 1.4 builds on the CI (https://teamcity.jetbrains.com/buildConfiguration/Kotlin_KotlinPublic_Aggregate?branch=&mode=branches), but I guess it would be more reliable to wait till 1.4-M1, it should happen relatively soon.

martinbonnin commented 4 years ago

I gave the latest CI build a shot but still got the same result.

What I did:

unzip the kotlin binary from build 3565: https://teamcity.jetbrains.com/repository/download/Kotlin_KotlinPublic_Compiler/2790655:id/kotlin-compiler-1.4.0-dev-3565.zip
build the kscript fat jar
invoke it with the 3565 kotlin binary: $unzip_dir/kotlinc/bin/kotlin -classpath build/libs/kscript.jar kscript.app.KscriptKt demo.kts

And it still gives the same error. Worth noting: while I used the 3565 binary to run the fatjar, the fatjar itself was build against 1.3.61 stable, not sure if this is a problem or not.

martinbonnin commented 4 years ago

With Kotlin 1.3.70 adding a lot of improvements around scripting, is it worth investigating this ticket more ?

I just checked and resolving dependencies work on *.main.kts files with Java 11 using kotlin 1.3.70 directly.

holgerbrandl commented 4 years ago

Indeed, it's great to see JBs continued interest in scripting support.

I'd be tempted to wait until a more complete EAP build becomes available. Not sure where to find them though.

binkley commented 4 years ago

@martinbonnin I'm using 1.3.70, and see the same behavior.

@holgerbrandl Lovely tool! I'm hoping JB recognizes this, and adopts kscript, or pulls in features such as caching.

martinbonnin commented 4 years ago

@binkley I think 1.3.70 actually does caching already. Name a file *.main.kts and call it twice, the second time should be much faster.

binkley commented 4 years ago

@martinbonnin Ah ... thank you for that. I named my script foo.kts (of course), and did not see the caching. kscript is still amazing with #!/usr/bin/env kotlin but did see caching for #!/usr/bin/env kscript.

holgerbrandl commented 4 years ago

This ticket is sitting here unresolved for way too long. :-(

I think I'm going to rollback dependency resolution to the old approach, where kscript will simply call maven/gradle internally (as a subprocess) to resolve dependencies. Not as pretty, but at least functional.

Should we do this jdk > 8 only?
Should we reenable this silently or with a flag (to indicate the workaround nature)?

ligee commented 4 years ago

I haven't looked into this issue deep enough, so don't know whether it could be relevant or not, but just for the information: Kotlin 1.4 will include a new API for the dependencies resolution in scripting and a new lib for resolving Maven artifacts, and the latter is not based on the (seemingly orphaned) jcabi-aether, but rather on the org.eclipse.aether and apache maven libs directly. https://github.com/JetBrains/kotlin/tree/master/libraries/scripting/dependencies-maven

LouisCAD commented 4 years ago

That's great news! Can we expect this in 1.4-M2, 1.4-M3 or later? Or will it be mentioned in the Kotlin blog when it lands (or launches I should say 😄)?

ligee commented 4 years ago

@LouisCAD it will be a part of 1.4-M2, I hope.

maxandersen commented 4 years ago

fyi, in jbang.dev I moved to using https://github.com/shrinkwrap/resolver - is a much nicer api to work with and enables interesting additional features.

holgerbrandl commented 4 years ago

@maxandersen I've looked into shrinkwrap which looks great apart from undocumented/missing/nonintuitive support for password-protected maven repos (which kscript supports at the moment via aether). In general, there are some references to authentication in the shrinkwrap repo. However, I could not find any documentation about how to configure a Maven.configureResolver().withRemoteRepo(...) to use login credentials. Any help would be appreciated here!

@apupier Also the intended support channel of shrinkwrap is not so clear to me, as there is no repo tracker being enabled nor are there any gitter/twitter links.

apupier commented 4 years ago

@apupier Also the intended support channel of shrinkwrap is not so clear to me, as there is no repo tracker being enabled nor are there any gitter/twitter links.

I'm not a maintainer of Shrinkwrap, I only reported an issue a while ago. Are you sure that you didn't want to ping another person?

Anyway, I think that all Shrinkwrap issues are reported either in this place https://github.com/shrinkwrap/shrinkwrap/issues or in this one https://issues.redhat.com/projects/SHRINKWRAP/issues

maxandersen commented 4 years ago

ah, I haven't tried to mimick the username/pwd setup kscript has - that was problematic for some of the auth approaches I've seen thus I for now just tell users to put their auth settings in ~/.m2/settings.xml which shrinkwrap honors.

holgerbrandl commented 4 years ago

@apupier sorry for the spam and thanks for the pointers.

@maxandersen We had a long discussion about the pros and cons of having credentials in the @MavenRepo declaration, which were finally added because it essentially was consistent with kscript intent to enable self-contained mini-applications written in Kotlin. Dropping credentials support now would be an incompatible API change, which should be avoided if possible.

Unfortunately, I did not manage yet to use aether for 1.8 (to preserve the current state) in conjunction with shrinkwrap as an experimental replacement for java 11. There are some conflicting classpath issues when adding both dependencies to the same project.

@ligee Will the resolver API in Kotlin 1.4 support repo authentication via API or also just via maven settings? Currently, I'd favour shrinkwrap because it would provide backwards-compatibility for older kotlin versions, but dropping existing features such as credentials support would imho outweigh this benefit.

maxandersen commented 4 years ago

I haven't tried but maybe you could on the fly generate the matching settings.xml and feed that to shrinkwrap resolver ?

akin to what is done in this test https://github.com/shrinkwrap/resolver/blob/master/maven/impl-maven/src/test/java/org/jboss/shrinkwrap/resolver/impl/maven/RepositoryAuthEnvTestCase.java#L47 ?

alekseytomin commented 4 years ago

@holgerbrandl I try to use 1.4-M2 and error is here.

$ kotlin -version Kotlin version 1.4-M2-release-92 (JRE 11.0.7+10-LTS) $ kscript -version Version : v2.9.3 $ ./some.kts [kscript] Resolving dependencies... [kscript] Resolving com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.0...Exception in thread "main" java.lang.NoClassDefFoundError: org/ietf/jgss/GSSException at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.(NettyAsyncHttpProvider.java:177) at org.sonatype.aether.connector.async.AsyncRepositoryConnector.getDefaultProvider(AsyncRepositoryConnector.java:246) ...

holgerbrandl commented 4 years ago

This is to be expected because it's unfortunately the java runtime which defines if kscript can resolve dependencies or not. With java 8 jcabi-aether (which is used under the hood) is working fine, but with java11 it is not. So your error simply indicates that you were running kscript with java >8.

Since jcabi-aether is no longer maintained, I'd like to use a replacement. There are 2 options

kotlin-1.4 ships with a dependency resolver, but this would limit backward compatibility of kscript
shrinkwrap looks great as well and works with older versions of kotlin, but does not support auth well (@maxandersen I'll certainly will try out your idea later this week which may be a good workaround to preserve auth support in @MavenRespository)

alekseytomin commented 4 years ago

Since jcabi-aether is no longer maintained, I'd like to use a replacement. There are 2 options

kotlin-1.4 ships with a dependency resolver, but this would limit backward compatibility of kscript

Sorry, I've thought It should be fixed by 1.4-M2 without any changes in ksctipt. I guess it's better to use the innovations from 1.4 and offer this as a version for jdk11

ligee commented 4 years ago

@holgerbrandl, the API supports direct authentication, but the use case is not well tested yet, so I'm not sure you can reliably use it as an argument for the new dependencies library.

lhmoraes commented 4 years ago

Hi @holgerbrandl do you have an idea when this issue could be fixed? Thanks

holgerbrandl commented 4 years ago

Currently, I'm tempted to wait for kotlin 1.4 to use the standard resolver shipping with the release. Sorry for the inconvenience until then.

jakubgwozdz commented 4 years ago

glad to hear this issue may soon be resolved now that 1.4 is out. I'm itching to run kscript as primary scripting language to perform db calls. And for that I need it to be able to fetch say mysql:mysql-connector-java:8.0.19, and to finally be able to run with jdk9+ modules like -Xadd-modules=java.sql

eschlenz commented 4 years ago

@holgerbrandl With 1.4 being officially released, any updates on this?

holgerbrandl commented 4 years ago

@ligee Are there examples and/or documentation about how to use the new resolver API?

ligee commented 4 years ago

@holgerbrandl, there is no documentation, unfortunately. Examples could be found here - https://github.com/Kotlin/kotlin-script-examples/blob/master/jvm/basic/jvm-maven-deps/MavenDeps.md and in the library tests - https://github.com/JetBrains/kotlin/blob/1.4.0/libraries/scripting/dependencies-maven/test/kotlin/script/experimental/test/MavenResolverTest.kt

CLOVIS-AI commented 4 years ago

Those official examples are a bit of a pain to use, we need to download a JAR with the annotations, etc. It'd be nice if this project could fix these issues and make this convenient :)

ligee commented 4 years ago

@CLOVIS-AI if you mean examples in the https://github.com/Kotlin/kotlin-script-examples repo - I'm not sure I understand what you mean. If you have some particular issues, please report them on the project page.

CLOVIS-AI commented 4 years ago

Sorry, I meant that the official Kotlin way of using scripts is a bit weird to implement and undocumented, and I really like kscript's system since it's much more straightforward, but it also doesn't work currently :/

holgerbrandl commented 3 years ago

I've changed the dependency resolver to use the provided api in kotlin 1.4 instead of aether in branch https://github.com/holgerbrandl/kscript/tree/fix239

There are 2 failing tests, which I'd love to address before rolling out a new release: 1) It does not seem to resolve pom dependencies, see test https://github.com/holgerbrandl/kscript/blob/master/test/resources/depends_on_with_type.kts

It does not seem to resolve dynamic dependencies. see test https://github.com/holgerbrandl/kscript/blob/master/test/resources/depends_on_dynamic.kts and https://docs.gradle.org/current/userguide/dynamic_versions.html

@ligee are these types of dependencies covered by the current implementation?

@ligee How is authentification supported? There don't seem to be any user/pw arguments in RepositoryCoordinates nor could I find an example for authentification yet.

ligee commented 3 years ago

@holgerbrandl, here is a test for a pom dependency that seems working to me - https://github.com/JetBrains/kotlin/blob/master/libraries/scripting/dependencies-maven/test/kotlin/script/experimental/test/MavenResolverTest.kt#L74, and test for the version range is around too. There is no specific test for the dynamic version, but the coordinates are parsed by aether lib, so I suppose it should work the same way as for cli mvn tool.

As of authentication - you can try to use a subclass of RepositoryCoordinates - MavenRepositoryCoordinates (https://github.com/JetBrains/kotlin/blob/master/libraries/scripting/dependencies-maven/src/kotlin/script/experimental/dependencies/maven/MavenDependenciesResolver.kt#L27) which have auth fields (only plain text creds are supported at the moment). There is no corresponding test in the kotlin repo, but I have been told that it works for the https://github.com/Kotlin/kotlin-jupyter project, please have a look.

holgerbrandl commented 3 years ago

Finally fixed in v3.0 :-)

Thank you all for your support and patience. Special thanks to @ligee for your continued great help and guidance.

kscripting / kscript

Dependencies are working only with jdk8 #239