Closed Raniz-at-Sony closed 9 years ago
This is the same as https://jira.grails.org/browse/GRAILS-11967
It is a documentation issue, you need to configure your proxy appropriately
Remark For some reason I can not comment the issue https://jira.grails.org/browse/GRAILS-11967 on the Grails Jira page (probably a permission thing). That is why I will post the comment here. If someone could add this to the Grails Jira page, I would be very grateful.
Environment OS: Ubuntu 14.04.2 LTS Dev: Groovy enVironment Manager 2.4.1, Groovy 2.4.3, Grails [3.0.0, 3.0.1, 3.0.2] Proxy: Squid (version unknown), requires authentication
Settings In the file '/home/user/.bashrc':
export http_proxy=http://xxx:xxx@proxy.xxx.ch:8080
export https_proxy=http://xxx:xxx@proxy.xxx.ch:8080
export GRAILS_OPTS="-Dhttp.proxyHost=proxy.xxx.ch -Dhttps.proxyHost=proxy.xxx.ch -Dhttp.proxyPort=8080 -Dhttps.proxyPort=8080 -Dhttp.proxyUser=xxx -Dhttp.proxyPassword=xxx -Dhttps.proxyUser=xxx -Dhttps.proxyPassword=xxx"
export JAVA_OPTS="-Dhttp.proxyHost=proxy.xxx.ch -Dhttps.proxyHost=proxy.xxx.ch -Dhttp.proxyPort=8080 -Dhttps.proxyPort=8080 -Dhttp.proxyUser=xxx -Dhttp.proxyPassword=xxx -Dhttps.proxyUser=xxx -Dhttps.proxyPassword=xxx"
Problem Grails 3.0.0 and above does not work with corporate proxies that requires authentication.
Description With Grails 2.5.0 the corporate proxy can be set using the ProxySettings.groovy file. With Grails 3.0.0 and newer versions, this is no longer possible (i.e. the file is ignored). The use of the environment variables http_proxy and https_proxy does not help either (the settings are correct since cURL and gvm are using them and they work fine). Setting either of the environment variables GRAILS_OPTS and JAVA_OPTS will prompt Grails to pick up the correct proxy configuration but it somehow misses the authentication part (username and password) as indicated by the error message '407 Proxy Authentication Required'.
Output Using grails without setting either of the environment variables GRAILS_OPTS and JAVA_OPTS gives:
user@virt:~/tmp/test$ grails create-app data --verbose --stacktrace
| Error Problem updating profiles from origin git repository
org.eclipse.jgit.api.errors.TransportException: https://github.com/grails/grails-profile-repository: cannot open git-upload-pack
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:139)
at org.grails.cli.profile.git.GitProfileRepository.fetchAndRebaseIfExpired(GitProfileRepository.groovy:140)
at org.grails.cli.profile.git.GitProfileRepository.createOrUpdateRepository(GitProfileRepository.groovy:109)
at org.grails.cli.profile.git.GitProfileRepository.getProfile(GitProfileRepository.groovy:56)
at org.grails.cli.profile.commands.CreateAppCommand.handle(CreateAppCommand.groovy:70)
at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:173)
at org.grails.cli.GrailsCli.main(GrailsCli.groovy:99)
Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/grails/grails-profile-repository: cannot open git-upload-pack
at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:524)
at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:309)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1115)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)
... 6 more
Caused by: java.net.ConnectException: Connection timed out github.com
at org.eclipse.jgit.util.HttpSupport.response(HttpSupport.java:175)
at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:475)
... 11 more
| Error Problem updating profiles from origin git repository
| Error Could not checkout tag for Grails release [3.0.2]: Ref v3.0.2 can not be resolved
org.eclipse.jgit.api.errors.RefNotFoundException: Ref v3.0.2 can not be resolved
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:244)
at org.grails.cli.profile.git.GitProfileRepository.checkoutTagForRelease(GitProfileRepository.groovy:128)
at org.grails.cli.profile.git.GitProfileRepository.createOrUpdateRepository(GitProfileRepository.groovy:116)
at org.grails.cli.profile.git.GitProfileRepository.getProfile(GitProfileRepository.groovy:56)
at org.grails.cli.profile.commands.CreateAppCommand.handle(CreateAppCommand.groovy:70)
at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:173)
at org.grails.cli.GrailsCli.main(GrailsCli.groovy:99)
| Error Could not checkout tag for Grails release [3.0.2]: Ref v3.0.2 can not be resolved
| Error Cannot find profile web
Using grails after setting either of the environment variables GRAILS_OPTS and JAVA_OPTS gives:
user@virt:~/tmp/test$ grails create-app data --verbose --stacktrace
| Error Problem updating profiles from origin git repository
org.eclipse.jgit.api.errors.TransportException: https://github.com/grails/grails-profile-repository: 407 Proxy Authentication Required
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:139)
at org.grails.cli.profile.git.GitProfileRepository.fetchAndRebaseIfExpired(GitProfileRepository.groovy:140)
at org.grails.cli.profile.git.GitProfileRepository.createOrUpdateRepository(GitProfileRepository.groovy:109)
at org.grails.cli.profile.git.GitProfileRepository.getProfile(GitProfileRepository.groovy:56)
at org.grails.cli.profile.commands.CreateAppCommand.handle(CreateAppCommand.groovy:70)
at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:173)
at org.grails.cli.GrailsCli.main(GrailsCli.groovy:99)
Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/grails/grails-profile-repository: 407 Proxy Authentication Required
at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:516)
at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:309)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:136)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1115)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)
... 6 more
| Error Problem updating profiles from origin git repository
| Error Could not checkout tag for Grails release [3.0.2]: Ref v3.0.2 can not be resolved
org.eclipse.jgit.api.errors.RefNotFoundException: Ref v3.0.2 can not be resolved
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:244)
at org.grails.cli.profile.git.GitProfileRepository.checkoutTagForRelease(GitProfileRepository.groovy:128)
at org.grails.cli.profile.git.GitProfileRepository.createOrUpdateRepository(GitProfileRepository.groovy:116)
at org.grails.cli.profile.git.GitProfileRepository.getProfile(GitProfileRepository.groovy:56)
at org.grails.cli.profile.commands.CreateAppCommand.handle(CreateAppCommand.groovy:70)
at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:173)
at org.grails.cli.GrailsCli.main(GrailsCli.groovy:99)
| Error Could not checkout tag for Grails release [3.0.2]: Ref v3.0.2 can not be resolved
| Error Cannot find profile web
Note I have tried all possible variations of proxy settings (with or without https, different combinations of environment variables) but nothing seems to work.
Comment I think that Grails is an extremely important project and you guys are doing a fantastic job! Please help us solve this problem so that we can use the eagerly awaited version 3.0.0 in our office. Thank you very much.
@michaelbauer1 The JIRA instance is to be decommissioned, we have migrated to Github issues.
Maybe you have a Socks5 proxy? See this post: http://joshuaredstone.blogspot.com.es/2009/01/using-jgit-with-socks5-proxy-on-ubuntu.html
You may have to configure socksProxyHost
, socksProxyPort
.. underneath the hood Grails uses JGit which uses Java's standard ProxySelector mechanism. There is an article here about configuring it http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html
I don't know what we, the Grails team, can do about this because this is standard Java proxy configuration stuff.
@graemerocher it seems I am unable to fix it at the moment (I spent the last two days working on the problem). It is definitely not socks since cURL uses the same http settings (and works just fine). I have a similar issue with git itself. On my Ubuntu 12.04 host everything is working fine. With Ubuntu 14.04 I get 'Received HTTP code 407 from proxy after CONNECT' error messages (same settings but for some reason the newer version of git doesn't want to use 'Basic Authentication'). Now the same happens with Grails. Version 2.5.0 and ProxySettings.groovy worked like a charm but version 3.0.0 and JGit changed all that.
When I sniff the wire with wireshark I can see the following. Ubuntu 14.04 (problem):
* Couldn't find host github.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
* Trying 161.62.241.81...
* Connected to proxy.xxx.ch (111.111.111.111) port 8080 (#0)
> GET http://github.com/less/less.js.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.9.1
Host: github.com
Accept: */*
Accept-Encoding: gzip
Proxy-Connection: Keep-Alive
Pragma: no-cache
Ubuntu 12.04 (ok):
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to proxy proxy.insel.ch port 8080 (#0)
* Trying 161.62.241.91... * Connected to proxy.xxx.ch (111.111.111.111) port 8080 (#0)
* Establish HTTP proxy tunnel to github.com:443
* Proxy auth using Basic with user 'username'
> CONNECT github.com:443 HTTP/1.1
Host: github.com:443
Proxy-Authorization: Basic asdffeuashdfausdfa
User-Agent: git/1.7.9.5
Proxy-Connection: Keep-Alive
Pragma: no-cache
Somehow JGit/Git does seem to ignore the authentication settings. Btw, the wireshark output is the same for JGit and Git (missing 'Proxy auth using Basic with user'). It is safe to say that this is one hell of a regression.
Anyway, thanks you for your help.
Will look into it further... thanks for the feedback
Thanks you. Please tell me if I can be of assistance (Testing etc.).
It does look like setting -Dhttp.proxyHost and -Dhttp.proxyPort in JAVA_OPTS / GRAILS_OPTS is being picked up, but not Dhttp.proxyUser and Dhttp.proxyPassword.
C:\Grails>grails -stacktrace create-app test | Error Problem updating profiles from origin git repository (NOTE: Stack trace has been filtered. Use --verbose to see entire trace.) org.eclipse.jgit.api.errors.TransportException: https://github.com/grails/grails-profile-repository: 407 Proxy Authentication Required
Setting default Authenticator in stand alone test opens "https://github.com/grails/grails-profile-repository" with proxy fine:
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType() == RequestorType.PROXY) {
String user = System.getProperty("http.proxyUser");
String password = System.getProperty("http.proxyPassword");
if (getRequestingHost().toLowerCase().equals(PROXY)) {
if (Integer.parseInt(port) == getRequestingPort()) {
return new PasswordAuthentication(user, password.toCharArray());
}
}
}
return null;
}
});
+1 for this, I'm behind an auth proxy too.
+1
+1 same issue here
+1 and about to drop Grails for something else...I always have these kind of issues and I am tired of it.
@jfl1977 Will look at this as a matter of priority for 3.0.8 which will be out very soon. I will try and make the proxy configuration and flexible as possible and also make an offline mode for when no internet is available at all.
Thank you!
Sent from my iPhone
On Sep 17, 2015, at 2:43 PM, graemerocher notifications@github.com wrote:
@jfl1977 Will look at this as a matter of priority for 3.0.8 which will be out very soon. I will try and make the proxy configuration and flexible as possible and also make an offline mode for when no internet is available at all.
— Reply to this email directly or view it on GitHub.
Thank you for the answer. I have been on the fence about grails lately so trying to upgrade and having problems is pretty dissapointing! If I create the app elsewhere am I going to run in problems later on at the office behind the proxy? Le 2015-09-17 2:43 PM, "graemerocher" notifications@github.com a écrit :
@jfl1977 https://github.com/jfl1977 Will look at this as a matter of priority for 3.0.8 which will be out very soon. I will try and make the proxy configuration and flexible as possible and also make an offline mode for when no internet is available at all.
— Reply to this email directly or view it on GitHub https://github.com/grails/grails-core/issues/594#issuecomment-141183666.
@jfl1977 This issue only occurs if you are behind a proxy and doesn't occur for everyone who is behind a proxy. Some people have been able to configure their proxy settings and get it working, whilst others have not been able to. We will expose settings which anyone can set to correctly configure a proxy, we will also make the profile repository available in GRAILS_HOME as a fallback. I'm not sure what else can be done other than what I have just mentioned.
Sounds good. Le 2015-09-17 8:29 PM, "graemerocher" notifications@github.com a écrit :
@jfl1977 https://github.com/jfl1977 This issue only occurs if you are behind a proxy and doesn't occur for everyone who is behind a proxy. Some people have been able to configure their proxy settings and get it working, whilst others have not been able to. We will expose settings which anyone can set to correctly configure a proxy, we will also make the profile repository available in GRAILS_HOME as a fallback. I'm not sure what else can be done other than what I have just mentioned.
— Reply to this email directly or view it on GitHub https://github.com/grails/grails-core/issues/594#issuecomment-141295505.
So I believe the following commit will provide a solution to this problem, however not being behind a proxy myself it would be nice if someone could test and verify prior to release of 3.0.8:
https://github.com/grails/grails-core/commit/03083fa851b51e24bf002e8e9ab4f4a1179ce5cb
The solution is two fold, first by default we provide an authenticator that looks at the system properties for the http.proxyUser
and http.proxyPassword
system properties. This should resolve the issue that some have mentioned about the authenticator not being set.
Second as a user you have complete control over the proxy selector and authenticator used since we added the ability to specify these in ~/.grails/settings.groovy
. Example:
grails.proxy.authenticator = new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
if(getRequestorType() == RequestorType.PROXY) {
return new PasswordAuthentication(
System.getProperty(BuildSettings.PROXY_HTTP_USER, ""),
System.getProperty(BuildSettings.PROXY_HTTP_PASSWORD, "").toCharArray());
}
return null
}
}
The grails.proxy.selector
setting can be used in settings.groovy
as well to specify a custom ProxySelector
, however for most people simply configuring the system properties will get things working I believe.
We are also going to work on a mode for Grails 3.1 that doesn't require connecting to the remote Github repo, but that is too big a change for 3.0.x
Sorry for not being able to test earlier. Works as expected with setting proxy uid / pwd. Thank you!
Sent from my iPhone
On Sep 23, 2015, at 6:05 AM, graemerocher notifications@github.com wrote:
So I believe the following commit will provide a solution to this problem, however not being behind a proxy myself it would be nice if someone could test and verify prior to release of 3.0.8:
03083fa
The solution is two fold, first by default we provide an authenticator that looks at the system properties for the http.proxyUser and http.proxyPassword system properties. This should resolve the issue that some have mentioned about the authenticator not being set.
Second as a user you have complete control over the proxy selector and authenticator used since we added the ability to specify these in ~/.grails/settings.groovy. Example:
grails.proxy.authenticator = new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() { if(getRequestorType() == RequestorType.PROXY) { return new PasswordAuthentication( System.getProperty(BuildSettings.PROXY_HTTP_USER, ""), System.getProperty(BuildSettings.PROXY_HTTP_PASSWORD, "").toCharArray()); } return null }
} The grails.proxy.selector setting can be used in settings.groovy as well to specify a custom ProxySelector, however for most people simply configuring the system properties will get things working I believe.
We are also going to work on a mode for Grails 3.1 that doesn't require connecting to the remote Github repo, but that is too big a change for 3.0.x
— Reply to this email directly or view it on GitHub.
@adrach13 Good to hear!
@graemerocher It might be worth running tests against a local Charles (web proxy) instance to check whether stuff is working. Alternatively, run tests using Betamax on an offline machine. That probably won't help with proxies that require usernames and passwords though.
I am behind a corporate proxy and still have the problem! I am using, via sdkman, the 3.1.0.RC1 which is supposed to have this commit (03083fa).
Whatever I do, setting the JAVA_OPTS="-Dhttp.proxy=... with http.proxyUser
, the GRAILS_OPTS
in the same way and/or the
grails.proxy.authenticator = new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
System.out.println("Some trace here to see if it is called !!!");
if(getRequestorType() == RequestorType.PROXY) {
return new PasswordAuthentication(
System.getProperty(BuildSettings.PROXY_HTTP_USER, ""),
System.getProperty(BuildSettings.PROXY_HTTP_PASSWORD, "").toCharArray());
}
return null
}
}
in ~/.grails/settings.groovy
I always get the 407 proxy authentication required!
Looking at the code above for the authenticator, I expect to at least see the trace message appear but I never see it and therefore believe that the authenticator is not used.
I would like to understand what the command: grails create-app test
does behind the scene...
~/.gradle/gradle.properties
with all the systemProp.http.proxyXXX
settings could have an impact !git config --global url.https://github.com/.insteadOf git://github.com/
HTTP_PROXY="username:password@proxyUrl:portNumber"
-Dhttp.proxy=...
Since we use AetherGrapeEngine
in 3.1 to resolve the profiles this is probably caused by https://github.com/spring-projects/spring-boot/blob/47d079d937765de486f28e505acb9e5c8c1a217d/spring-boot-cli/src/main/java/org/springframework/boot/cli/compiler/grape/AetherGrapeEngine.java#L99 in 3.1
Yes ... that would indeed cause the 407 whatever you configure...
What I am unclear about is then:
The authenticator used to be used when we used jgit, that is no longer the case in 3.1
AetherGrapeEngine
is used here https://github.com/grails/grails-core/blob/master/grails-shell/src/main/groovy/org/grails/cli/profile/repository/MavenProfileRepository.groovy
Ok.
I've setup a proxy that does not require authentication.
Now I get a DNS resolution error.
In a corporate network, (at least ours) the DNS resolution is done by the proxy itself but is not allowed to be done directly by the tools.
i.e.: an nslookup github.com
does not work and never will !
However, curl https://repo.grails.org/grails/webapp/#/home
with the right environment variables configured will actually work because the proxy itself resolves the domain !
For some reason, the aether transporter and the BasicRepositoryConnector
running the task is trying to resolve itself the domain repo.grails.org
and obviously fails.
Error |
Failed to load latest version of profile [angular]. Trying Grails release version
org.springframework.boot.cli.compiler.grape.DependencyResolutionFailedException: org.eclipse.aether.resolution.DependencyResolutionException: Failed to collect dependencies at org.grails.profiles:angular:jar:3.1.0.RC1
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.resolve(AetherGrapeEngine.java:307)
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.grab(AetherGrapeEngine.java:112)
at org.springframework.boot.cli.compiler.grape.AetherGrapeEngine.grab(AetherGrapeEngine.java:104)
at org.grails.cli.profile.repository.MavenProfileRepository.getAllProfiles(MavenProfileRepository.groovy:128)
[...]
Caused by: java.net.UnknownHostException: repo.grails.org
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
at java.net.InetAddress.getAllByName(InetAddress.java:1171)
at java.net.InetAddress.getAllByName(InetAddress.java:1105)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:278)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:162)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:137)
at org.eclipse.aether.transport.http.HttpTransporter.execute(HttpTransporter.java:287)
at org.eclipse.aether.transport.http.HttpTransporter.implGet(HttpTransporter.java:243)
at org.eclipse.aether.spi.connector.transport.AbstractTransporter.get(AbstractTransporter.java:59)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:447)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:350)
I believe that the AetherGrapeEngine.resolve
is not a DNS resolution but the dependency resolution...
I'm not sure how this could/should be solved and if I should enter a bug report for it...
Hi, I had the same problem. After installing maven and setting the proxy authentication info as described here the dependencies are now found...
...then there will be still problem with the gradle wrapper as it tries to download the distribution zip (gradle-2.9-bin.zip
is referred in my case). By downloading the appropriate one manually and placing it to the gradle/dists
folder e.g. and changing the configuration file at gradle-wrapper.properties
in order to point to it as follows:
distributionUrl=../dists/gradle-2.9-bin.zip
it will work.
Added documentation on how to configure a proxy here https://github.com/grails/grails-doc/commit/5f1bb09b6e02d3a07fce82a74f4dff7ec6a0e618
The app seems to be created, but there are exceptions thrown.