jruby-gradle / jruby-gradle-plugin

A Gradle plugin for working with JRuby projects for fun and profit. Mostly profit.
http://jruby-gradle.org
Other
93 stars 38 forks source link

Custom gem repository not working with jruby-gradle-plugin 2.0.1 (plain command line gem does work with the same custom repo in .gemrc) #429

Open tkrah opened 3 years ago

tkrah commented 3 years ago

Hi,

I tried to get a custom gem repository working with jRuby but it does not work like described.

I did configure my local .gemrc with that URL which it should use (custom mirror - artifactory) and it works just fine with "gem install ..." etc. - so the upstream mirror is ok.

I choose to modify the example from the project:

https://github.com/jruby-gradle/jruby-gradle-plugin/tree/master/examples/run-script-with-jruby-args

and modified the repository line from

ruby.gems()

to

ruby.gems("https://tkrah:MY_CREDENTIALS@artifacts.localhost.local/artifactory/api/gems/gems-repos/")

Running the gradle task runGradleTest now just results in this output instead of a successful build:

exception thrown for request to /rubygems/colorize/
com.github.jrubygradle.api.core.ApiException: Count not retrieve list of versions for colorize
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:97)
        at com.github.jrubygradle.internal.core.AbstractIvyXmlProxyServer.getDirectoryListing(AbstractIvyXmlProxyServer.groovy:181)
        at com.github.jrubygradle.internal.core.IvyXmlRatpackProxyServer.lambda$run$2(IvyXmlRatpackProxyServer.java:98)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.MethodHandler.handle(MethodHandler.java:44)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.insert(DefaultContext.java:179)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:61)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.lambda$start$4(DefaultContext.java:104)
        at ratpack.exec.internal.DefaultExecution.lambda$new$0(DefaultExecution.java:82)
        at ratpack.exec.internal.DefaultExecution$InitialExecStream.exec(DefaultExecution.java:403)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:224)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:217)
        at ratpack.exec.internal.DefaultExecution.drain(DefaultExecution.java:187)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at ratpack.exec.internal.DefaultExecController$ExecControllerBindingThreadFactory.lambda$newThread$0(DefaultExecController.java:137)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: groovyx.net.http.HttpException: Not Found
        at groovyx.net.http.NativeHandlers.failure(NativeHandlers.java:69)
        at groovyx.net.http.HttpBuilder$ResponseHandlerFunction.apply(HttpBuilder.java:2305)
        at groovyx.net.http.OkHttpBuilder.execute(OkHttpBuilder.java:290)
        at groovyx.net.http.OkHttpBuilder.doGet(OkHttpBuilder.java:184)
        at groovyx.net.http.HttpObjectConfigImpl.nullInterceptor(HttpObjectConfigImpl.java:47)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:346)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:1297)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.getData(DefaultRubyGemRestApi.groovy:159)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:95)
        ... 27 more
exception thrown for request to /rubygems/colorize/
com.github.jrubygradle.api.core.ApiException: Count not retrieve list of versions for colorize
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:97)
        at com.github.jrubygradle.internal.core.AbstractIvyXmlProxyServer.getDirectoryListing(AbstractIvyXmlProxyServer.groovy:181)
        at com.github.jrubygradle.internal.core.IvyXmlRatpackProxyServer.lambda$run$2(IvyXmlRatpackProxyServer.java:98)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.MethodHandler.handle(MethodHandler.java:44)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.insert(DefaultContext.java:179)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:61)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.lambda$start$4(DefaultContext.java:104)
        at ratpack.exec.internal.DefaultExecution.lambda$new$0(DefaultExecution.java:82)
        at ratpack.exec.internal.DefaultExecution$InitialExecStream.exec(DefaultExecution.java:403)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:224)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:217)
        at ratpack.exec.internal.DefaultExecution.drain(DefaultExecution.java:187)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at ratpack.exec.internal.DefaultExecController$ExecControllerBindingThreadFactory.lambda$newThread$0(DefaultExecController.java:137)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: groovyx.net.http.HttpException: Not Found
        at groovyx.net.http.NativeHandlers.failure(NativeHandlers.java:69)
        at groovyx.net.http.HttpBuilder$ResponseHandlerFunction.apply(HttpBuilder.java:2305)
        at groovyx.net.http.OkHttpBuilder.execute(OkHttpBuilder.java:290)
        at groovyx.net.http.OkHttpBuilder.doGet(OkHttpBuilder.java:184)
        at groovyx.net.http.HttpObjectConfigImpl.nullInterceptor(HttpObjectConfigImpl.java:47)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:346)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:1297)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.getData(DefaultRubyGemRestApi.groovy:159)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:95)
        ... 27 more
exception thrown for request to /rubygems/colorize/
com.github.jrubygradle.api.core.ApiException: Count not retrieve list of versions for colorize
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:97)
        at com.github.jrubygradle.internal.core.AbstractIvyXmlProxyServer.getDirectoryListing(AbstractIvyXmlProxyServer.groovy:181)
        at com.github.jrubygradle.internal.core.IvyXmlRatpackProxyServer.lambda$run$2(IvyXmlRatpackProxyServer.java:98)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.MethodHandler.handle(MethodHandler.java:44)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.insert(DefaultContext.java:179)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:61)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.path.internal.PathHandler.handle(PathHandler.java:63)
        at ratpack.handling.internal.DefaultContext.next(DefaultContext.java:157)
        at ratpack.handling.internal.DefaultContext.lambda$start$4(DefaultContext.java:104)
        at ratpack.exec.internal.DefaultExecution.lambda$new$0(DefaultExecution.java:82)
        at ratpack.exec.internal.DefaultExecution$InitialExecStream.exec(DefaultExecution.java:403)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:224)
        at ratpack.exec.internal.DefaultExecution.exec(DefaultExecution.java:217)
        at ratpack.exec.internal.DefaultExecution.drain(DefaultExecution.java:187)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:335)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
        at ratpack.exec.internal.DefaultExecController$ExecControllerBindingThreadFactory.lambda$newThread$0(DefaultExecController.java:137)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: groovyx.net.http.HttpException: Not Found
        at groovyx.net.http.NativeHandlers.failure(NativeHandlers.java:69)
        at groovyx.net.http.HttpBuilder$ResponseHandlerFunction.apply(HttpBuilder.java:2305)
        at groovyx.net.http.OkHttpBuilder.execute(OkHttpBuilder.java:290)
        at groovyx.net.http.OkHttpBuilder.doGet(OkHttpBuilder.java:184)
        at groovyx.net.http.HttpObjectConfigImpl.nullInterceptor(HttpObjectConfigImpl.java:47)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:346)
        at groovyx.net.http.HttpBuilder.get(HttpBuilder.java:1297)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.getData(DefaultRubyGemRestApi.groovy:159)
        at com.github.jrubygradle.internal.core.DefaultRubyGemRestApi.allVersions(DefaultRubyGemRestApi.groovy:95)
        ... 27 more
> Task :jrubyPrepare FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jrubyPrepare'.
> Could not resolve all files for configuration ':gems'.
   > Could not resolve rubygems:colorize:0.7.7+.
     Required by:
         project :
      > Failed to list versions for rubygems:colorize.
         > Could not list versions using Ivy pattern 'http://localhost:38737/[organisation]/[module]/[revision]/ivy.xml'.
            > Could not get resource 'http://localhost:38737/rubygems/colorize/'.
               > Could not GET 'http://localhost:38737/rubygems/colorize/'. Received status code 500 from server: Internal Server Error

So how do you use a custom repository with jRuby gradle plugin here? Just using the same URL which does work for command line running gem is not going to work here.

tkrah commented 3 years ago

Some debugging revealed some interesting things - the actual request made is e.g.:

https://tkrah:MY_CREDENTIALS@artifacts.localhost.local/api/v1/versions/colorize.json

This won't work - the repository does not run under that URL - it's missing the whole part about the gem-repos url part:

/artifactory/api/gems/gems-repos/

Additionally the OkHttpBuilder has a auth part called

applyAuth(requestBuilder, chainedConfig);

which has this:

    private static void applyAuth(final Request.Builder requestBuilder, final ChainedHttpConfig chainedConfig) {
        final HttpConfig.Auth auth = chainedConfig.getChainedRequest().actualAuth();
        if (auth != null) {
            switch (auth.getAuthType()) {
                case BASIC:
                    requestBuilder.addHeader("Authorization", Credentials.basic(auth.getUser(), auth.getPassword()));
                    break;
                case DIGEST:
                    // supported in constructor with an interceptor
            }
        }
    }

but authentication was not configured for that request (although it's included in the configured URL) - not sure if it will be respected or not, can't tell for sure because the whole URL does not match the configured one.

Maybe someone has an idea what's wrong.

silverhammermba commented 1 year ago

I have this problem as well. I'm using this plugin to run a Ruby script (with gems) as part of my gradle build. Everything works fine if I use the default ruby.gems() repository, but I can't find a way to tell it to use my artifactory instance instead.

Using bundler, I changed my Gemfile from source "https://rubygems.org" to source "https://myartifactory.jfrog.io/artifactory/api/gems/gems/" and stored my credentials in an environment variable

export BUNDLE_MYARTIFACTORY__JFROG__IO="user:pwd"

and that works great for bundler. I tried doing the obvious thing similar to @tkrah and changing the repo to ruby.gems("https://user:pwd@myartifactory.jfrog.io/artifactory/api/gems/gems") but that fails to resolve with a similar error

404 client error for request to /rubygems/twine/1.1.1/ivy.xml

> Task :myproj:jrubyPrepare FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':myproj:jrubyPrepare'.
> Could not resolve all dependencies for configuration ':myproj:gems'.
   > Could not find rubygems:twine:1.1.1.
     Searched in the following locations:
       - http://localhost:61344/rubygems/twine/1.1.1/ivy.xml
       - https://myartifactory.jfrog.io/artifactory/libs-release/rubygems/twine/1.1.1/twine-1.1.1.pom
     Required by:
         project :myproj
silverhammermba commented 1 year ago

I think the fact that it searched http://localhost:61344/rubygems/twine/1.1.1/ivy.xml is promising, because that's the internal ivy proxy that the jruby plugin sets up. The annoying part is I can't find any way to see how that proxy is trying to resolve my custom URI. So it's hard to debug.

tkrah commented 1 year ago

@silverhammermba look at https://github.com/jruby-gradle/jruby-gradle-plugin/issues/429#issuecomment-937835472 - I've debugged through the code - the whole url resolution is wrong if you specify a custom repository and auth is unconfigured.

raelik commented 2 months ago

As far as I can tell, this fails because the authentication is never configured here:

https://github.com/jruby-gradle/jruby-gradle-plugin/blob/024713cc95d84fb823cef02cc48a5b881719048b/core-plugin/src/main/groovy/com/github/jrubygradle/internal/core/DefaultRubyGemRestApi.groovy#L167-L175

I believe fixing this would require either parsing the credentials out of the URI, or providing them from an external file or environment variable (I would probably allow for all 3 options) in the two DefaultRubyGemRestApi constructors, and creating a new function signature like this:

    private static HttpBuilder getHttpBuilder(URI uri, String username, String password) {
        configure {
            request.uri = uri
            request.auth.basic username, password
            client.clientCustomizer { OkHttpClient.Builder builder ->
                builder.followRedirects(true)
                builder.followSslRedirects(true)
            }
        }
    }

The constructors would call this signature is authentication was provided. There's probably a cleaner way to fix this, but that's the first way I could think of.