jgit apache - Githubissues

appiepollo14 commented 8 months ago

Hi,

I'ld love to be able to use jgit apache with sshd in quarkus. I found the quarkiverse sshd extension. However, I can't use it in my opinion in combination with quarkus-jgit, as that plugin is based in jgit, not on jgit.apache.

What are your views on solving this question. How can I combine the two?

Is there any documentation on making an upstream dependency native compilable so I can use the apache version myself.
Should we create a new quarkiverse extension for jgit-apache?
Is there a plan on upgrading to the apache version as upstream for this extension quarkus.jgit?

Would love to talk.

gastaldi commented 8 months ago

Can you elaborate on what you need specifically from org.eclipse.jgit.ssh.apache that isn't provided by org.eclipse.jgit.ssh.jsch?

In our tests we found the jsch implementation to be less painful to integrate with and rather sufficient for the use cases we encountered so far.

appiepollo14 commented 8 months ago

The fact that it contains a sshdfactory to use sshd as the ssh client in jgit instead of jsch

gastaldi commented 8 months ago

There is https://github.com/quarkiverse/quarkus-sshd which aims to provide support for Apache Mina. Can you give it a try by depending on that extension instead of quarkus-jsch? Don't forget to also exclude org.eclipse.jgit:org.eclipse.jgit.ssh.jsch

appiepollo14 commented 8 months ago

I understand. But the factory is part of jgit itself. So there is no way to use sshd as the client with the quarkus jgit as the upstream doesn't contain that class. That's what in the classes here from the upstream. https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/master/org.eclipse.jgit.ssh.apache/src/org/eclipse/jgit/transport/sshd

If there is another way, please let me know.

gastaldi commented 8 months ago

Yes, in addition to excluding the above dependency, you need to add a dependency to org.eclipse.jgit:org.eclipse.jgit.ssh.apache

appiepollo14 commented 8 months ago

That is exactly my point. org.eclipse.jgit:org.eclipse.jgit.ssh.apache is a replacement for org.eclipse.jgit:org.eclipse.jgit.ssh. So when introducing the apache one, the quarkus jgit extension has no longer use in the project. When adding the apache dependency it will not natively compile anymore. Hence my initial question. Should this be a successor of the current jgit quarkus upstream dependency, should I make it natively compilable myself, than please are there any docs, of a completely new extension maybe?

gastaldi commented 8 months ago

As I said in https://github.com/quarkiverse/quarkus-jgit/issues/121#issuecomment-2025657418, my hope is that quarkus-sshd would fix the native compilation issues, since org.eclipse.jgit:org.eclipse.jgit.ssh.apache depends on the same library.

We could introduce separate extensions here to support the different SSH providers, so the jsch would depend on quarkus-jsch and the other quarkus-sshd if that makes sense.

gastaldi commented 8 months ago

@appiepollo14 I refactored the extension by separating the JSCH SSH provider into another extension. Same for SSHD SSH provider: https://github.com/quarkiverse/quarkus-jgit/pull/122

Please give it a try and let me know how it goes. This still needs some polishing before being merged.

Your project will look like this:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.quarkiverse.jgit</groupId>
            <artifactId>quarkus-jgit-bom</artifactId>
            <version>999-SNAPSHOT</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
     </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>io.quarkiverse.jgit</groupId>
        <artifactId>quarkus-jgit</artifactId>
    </dependency>
    <dependency>
        <groupId>io.quarkiverse.jgit</groupId>
        <artifactId>quarkus-jgit-sshd</artifactId>
    </dependency>
</dependencies>

appiepollo14 commented 8 months ago

Will do and let you know! Thanks for picking up so fast on this!!

appiepollo14 commented 8 months ago

I've tried to checkout the ssh_providers branch. When doing a clean install, there are multiple errors about versions missing in pom files. Hence why I cannot use this code for tests in another project.

gastaldi commented 8 months ago

@appiepollo14 try again please, I forgot to reference the relativePath from the new parent poms

appiepollo14 commented 8 months ago

@gastaldi it now does a clean install! Great! In my project I've setup the dependencies as instructed by you above. The functions I need, do work. Great!

However, when compiling my project natively, it fails. There are some issues with deps??

`14:58:17,123 INFO [org.apa.ssh.com.uti.sec.bou.BouncyCastleSecurityProviderRegistrar] getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider 14:58:17,136 INFO [org.apa.ssh.com.uti.sec.edd.EdDSASecurityProviderRegistrar] getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider 14:58:17,183 INFO [org.apa.ssh.com.io.DefaultIoServiceFactoryFactory] No detected/configured IoServiceFactoryFactory; using Nio2ServiceFactoryFactory

....

Error: Discovered unresolved type during parsing: sun.security.x509.X509Key. This error is reported at image build time because class net.i2p.crypto.eddsa.EdDSAEngine is registered for linking at image build time by command line and command line. Error encountered while parsing net.i2p.crypto.eddsa.EdDSAEngine.engineInitVerify(EdDSAEngine.java:147) Parsing context: at java.security.Signature.initVerify(Signature.java:505) at root method.(Unknown Source) `

Adding: `

net.i2p.crypto

        <artifactId>eddsa</artifactId>
        <version>0.3.0</version>
    </dependency>`

to the project doesn't make a difference. Any pointers?

gastaldi commented 8 months ago

@appiepollo14 I've pushed a1cf821 (#122) which may fix this issue, can you pull and try again?

gastaldi commented 8 months ago

I've also created https://github.com/quarkiverse/quarkus-sshd/issues/59 asking to resolve that in the quarkus-sshd extension

appiepollo14 commented 8 months ago

Great! New errors occur: believe also should be fixed in quarkus-sshd. Errors:

` Error: Unsupported features in 2 methods Detailed message: Error: Discovered unresolved method during parsing: org.apache.sshd.common.util.security.eddsa.Ed25519PEMResourceKeyParser.decodeEd25519KeyPair(byte[]). This error is reported at image build time because class org.apache.sshd.common.config.keys.loader.pem.PKCS8PEMResourceKeyPairParser is registered for linking at image build time by command line and command line. Error encountered while parsing org.apache.sshd.common.config.keys.loader.pem.PKCS8PEMResourceKeyPairParser.extractKeyPairs(PKCS8PEMResourceKeyPairParser.java:129) Parsing context: at org.apache.sshd.common.config.keys.loader.pem.PKCS8PEMResourceKeyPairParser.extractKeyPairs(PKCS8PEMResourceKeyPairParser.java:90) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.extractKeyPairs(AbstractKeyPairResourceParser.java:198) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.extractKeyPairs(AbstractKeyPairResourceParser.java:167) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117) at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130) at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:523) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider.loadKey(CachingKeyPairProvider.java:156) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider.loadKey(CachingKeyPairProvider.java:131) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider$CancellingKeyPairIterator.hasNext(CachingKeyPairProvider.java:221) at jdk.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:513) at com.fasterxml.jackson.databind.util.ClassUtil.closeOnFailAndThrowAsIOE(ClassUtil.java:527) at com.fasterxml.jackson.databind.ObjectWriter._writeCloseable(ObjectWriter.java:1295) at com.fasterxml.jackson.databind.ObjectWriter._writeValueAndClose(ObjectWriter.java:1269) at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsString(ObjectWriter.java:1140) at com.fasterxml.jackson.databind.node.InternalNodeMapper.nodeToString(InternalNodeMapper.java:34) at com.fasterxml.jackson.databind.node.BaseJsonNode.toString(BaseJsonNode.java:242) at root method.(Unknown Source)

Error: Discovered unresolved type during parsing: net.i2p.crypto.eddsa.EdDSAPublicKey. This error is reported at image build time because class org.apache.sshd.common.util.buffer.keys.SkED25519BufferPublicKeyParser is registered for linking at image build time by command line and command line. Error encountered while parsing org.apache.sshd.common.util.buffer.keys.SkED25519BufferPublicKeyParser.getRawPublicKey(SkED25519BufferPublicKeyParser.java:45) Parsing context: at org.apache.sshd.common.util.buffer.keys.SkED25519BufferPublicKeyParser.getRawPublicKey(SkED25519BufferPublicKeyParser.java:34) at org.apache.sshd.common.util.buffer.keys.BufferPublicKeyParser$2.getRawPublicKey(BufferPublicKeyParser.java:102) at org.apache.sshd.common.util.buffer.Buffer.getRawPublicKey(Buffer.java:568) at org.apache.sshd.common.util.buffer.Buffer.getPublicKey(Buffer.java:543) at org.apache.sshd.common.util.buffer.Buffer.getPublicKey(Buffer.java:528) at org.apache.sshd.common.util.buffer.keys.OpenSSHCertPublicKeyParser.getRawPublicKey(OpenSSHCertPublicKeyParser.java:82) at org.apache.sshd.common.config.keys.impl.OpenSSHCertificateDecoder.decodePublicKey(OpenSSHCertificateDecoder.java:75) at org.apache.sshd.common.config.keys.impl.OpenSSHCertificateDecoder.decodePublicKey(OpenSSHCertificateDecoder.java:46) at org.apache.sshd.common.config.keys.loader.openssh.OpenSSHKeyPairResourceParser.readPublicKey(OpenSSHKeyPairResourceParser.java:221) at org.apache.sshd.common.config.keys.loader.openssh.OpenSSHKeyPairResourceParser.extractKeyPairs(OpenSSHKeyPairResourceParser.java:133) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.extractKeyPairs(AbstractKeyPairResourceParser.java:198) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.extractKeyPairs(AbstractKeyPairResourceParser.java:167) at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117) at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139) at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130) at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:523) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider.loadKey(CachingKeyPairProvider.java:156) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider.loadKey(CachingKeyPairProvider.java:131) at org.eclipse.jgit.internal.transport.sshd.CachingKeyPairProvider$CancellingKeyPairIterator.hasNext(CachingKeyPairProvider.java:221) at jdk.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:513) at com.fasterxml.jackson.databind.util.ClassUtil.closeOnFailAndThrowAsIOE(ClassUtil.java:527) at com.fasterxml.jackson.databind.ObjectWriter._writeCloseable(ObjectWriter.java:1295) at com.fasterxml.jackson.databind.ObjectWriter._writeValueAndClose(ObjectWriter.java:1269) at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsString(ObjectWriter.java:1140) at com.fasterxml.jackson.databind.node.InternalNodeMapper.nodeToString(InternalNodeMapper.java:34) at com.fasterxml.jackson.databind.node.BaseJsonNode.toString(BaseJsonNode.java:242) at root method.(Unknown Source) `

gastaldi commented 8 months ago

Can you please add this stacktrace to the issue I created earlier along with a reproducer?

appiepollo14 commented 8 months ago

Will do

appiepollo14 commented 8 months ago

@gastaldi make a small reproducer.

However, it won't even natively compile now due to: Caused by: java.lang.SecurityException: class "org.eclipse.jgit.transport.sshd.SshdSessionFactory"'s signer information does not match signer information of other classes in the same package Looks to be similar to: https://github.com/spring-cloud/spring-cloud-contract/issues/1822

How to continue?

gastaldi commented 8 months ago

@zakkak could this be a GraalVM issue? Here is the complete stacktrace:

com.oracle.svm.core.util.VMError$HostedError: InternalFeature defined by com.oracle.svm.hosted.reflect.ReflectionFeature unexpectedly failed with a(n) java.lang.SecurityException
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:86)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.handleFeatureError(FeatureHandler.java:287)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:92)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:964)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:590)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
Caused by: java.lang.SecurityException: class "org.eclipse.jgit.transport.sshd.SshdSessionFactory"'s signer information does not match signer information of other classes in the same package
    at java.base/java.lang.ClassLoader.checkCerts(ClassLoader.java:1173)
    at java.base/java.lang.ClassLoader.preDefineClass(ClassLoader.java:917)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1025)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoader.defineClass(NativeImageClassLoader.java:500)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoader.findClassViaClassPath(NativeImageClassLoader.java:452)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageClassLoader.loadClass(NativeImageClassLoader.java:640)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:534)
    at java.base/java.lang.Class.forName(Class.java:513)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageClassLoader.forName(ImageClassLoader.java:307)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageClassLoader.forName(ImageClassLoader.java:303)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageClassLoader.findClass(ImageClassLoader.java:296)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.RegistryAdapter.resolveType(RegistryAdapter.java:77)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ReflectionRegistryAdapter.resolveType(ReflectionRegistryAdapter.java:48)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.configure.ReflectionConfigurationParser.parseClass(ReflectionConfigurationParser.java:94)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.configure.ReflectionConfigurationParser.parseClassArray(ReflectionConfigurationParser.java:74)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.configure.ReflectionConfigurationParser.parseAndRegister(ReflectionConfigurationParser.java:69)
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.configure.ConfigurationParser.parseAndRegister(ConfigurationParser.java:73)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:130)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$2(ConfigurationParserUtils.java:116)
    at java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:214)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:109)
    at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:332)
    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
    at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310)
    at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:515)
    at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:473)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:118)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:75)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionFeature.duringSetup(ReflectionFeature.java:265)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$setupNativeImage$16(NativeImageGenerator.java:964)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:90)
    ... 7 more

appiepollo14 commented 7 months ago

@gastaldi Or possible because jgit might be referenced twice in the dependency tree based on the separated providers?

zakkak commented 7 months ago

@zakkak could this be a GraalVM issue?

I don't think so and I haven't seen this again.

@gastaldi Or possible because jgit might be referenced twice in the dependency tree based on the separated providers?

That sounds more plausible.

appiepollo14 commented 7 months ago

@gastaldi how can we proceed with this topic? Can we steer this branch to include the jgit only once per ssh provider or should we separate this extension into two extensions (jsch/sshd)?

gastaldi commented 7 months ago

It seems that Quarkus is modifying that artifact (I can see a modified-org.eclipse.jgit.org.eclipse.jgit.ssh.apache-6.9.0.202403050737-r.jar in the quarkus-reproducer-1.0.0-SNAPSHOT-native-image-source-jar\lib directory.

I'm investigating what's causing this

gastaldi commented 7 months ago

@appiepollo14 I created https://github.com/quarkusio/quarkus/pull/40001 to fix the signed JAR issue

appiepollo14 commented 7 months ago

@gastaldi can we test the result https://github.com/quarkusio/quarkus/pull/40001 for the quarkus-jgit issue before a new Quarkus version is released? If yes, please tell me how.

gastaldi commented 7 months ago

Sure, you just need to build Quarkus (using mvn -Dquickly) and use io.quarkus:quarkus-bom:999-SNAPSHOT in your sample project instead

appiepollo14 commented 7 months ago

Did it, thanks. Native compilation fails again due to https://github.com/quarkiverse/quarkus-sshd/issues/59, will focus on that one again.

quarkiverse / quarkus-jgit

jgit apache #121