Spring boot cli ignores http proxy settings

[gerjantd]

Spring boot cli fails when trying to run the simple groovy web app example behind a corporate proxy. I'm having the same issue as the author of this post: http://forum.spring.io/forum/spring-projects/boot/726134-spring-boot-cli-proxy-server-configuration. Tried from Bash after 'export JAVA_OPTS="-Dhttp.proxyHost=... -Dhttp.proxyPort=..."' but the started Java process ignores these settings.

[dsyer]

What does you app need those settings for, and how do you know they were ignored (AFAIK this is handled in the JVM itself, and nothing to do with Spring)? JAVA_OPTS should be included by the CLI (of you use the spring shell script that we provide). Can you see it in your running app (e.g. in visualvm or in an actuator /env endpoint or similar)? Can you provide some more detail, or steps to reproduce?

[gerjantd]

I've attached 3 screenshots to illustrate the issue. The 'purple' screenshot shows a system which has been granted proxyless access to the internet. The 'yellow' screenshots show a regular system, required to go via our corporate http proxy. On both systems I'm running spring run webapp.groovy in the terminal on the left, while monitoring outgoing connections to port 80 in the terminal on the right (sudo watch -n 2 "netstat -tp | grep 'http '") Before running the spring boot CLI, I delete my local .m2 repository to make sure the required dependencies have to be downloaded. I also display the java version and the start time, and, in the case of the proxied system, I explicitly set JAVA_OPTS with the proxy values. (This shouldn't be necessary since the JVM ought to pick them up from the environment in any case.) The non-proxied system runs the command within 30 seconds, with many http connections showing as a result of the dependencies being downloaded. The proxied system hangs for ~ 4 minutes, with just the one connection attempt showing, then fails with a stack trace indicating a connection time-out. Note the connection attempt is to port 80/http rather than to 8080/http-alt, as you would expect if the JVM heeded the proxy settings. My guess is the JVM picks up the proxy settings allright, but the connection has to be made using these proxy settings explicitly (if they exist).

[dsyer]

I'm not really sure what the netstat sessions mean but I don't see the same. I set up a tinyproxy locally and watched its logs. I did this:

$ export JAVA_OPTS="-Dhttp.proxyHost=localhost -Dhttp.proxyPort=8888"
$ spring run spring-boot-cli.samples/web.groovy

The proxy clearly gets all the requests to download dependencies, e.g.

CONNECT   Nov 27 17:05:13 [10946]: Connect (file descriptor 6): localhost [127.0.0.1]
CONNECT   Nov 27 17:05:13 [10946]: Request (file descriptor 6): GET http://repo.spring.io/snapshot/org/springframework/boot/spring-boot-starter-tomcat/0.5.0.BUILD-SNAPSHOT/maven-metadata.xml HTTP/1.1
INFO      Nov 27 17:05:13 [10946]: No upstream proxy for repo.spring.io
CONNECT   Nov 27 17:05:13 [10946]: Established connection to host "repo.spring.io" using file descriptor 7.
INFO      Nov 27 17:05:13 [10946]: Closed connection between local client (fd:6) and remote client (fd:7)

[dsyer]

Actually I remember 1.7.0_25 having some pretty major problems with dependency resolution back when we still used ivy. If I were you I'd upgrade that and try again.

Update: I tried _25 and it worked fro me. Not sure what is different for you at this point.

[gerjantd]

I did a manual install of 1.7.0_45 and set up tinyproxy as you did. I see no connection logged by tinyproxy, but netstat logs lots of connections going straight to port 80 on (presumably) the external maven repository sites

tcp6       0      0 peertje.rivm.nl:60288   ec2-54-236-89-235.:http ESTABLISHED 26395/java
tcp6       0      0 peertje.rivm.nl:60289   ec2-54-236-89-235.:http ESTABLISHED 26395/java
tcp6       0      0 peertje.rivm.nl:60286   ec2-54-236-89-235.:http ESTABLISHED 26395/java
tcp6       0      0 peertje.rivm.nl:60285   ec2-54-236-89-235.:http ESTABLISHED 26395/java
tcp6       0      0 peertje.rivm.nl:60287   ec2-54-236-89-235.:http ESTABLISHED 26395/java

This happens even when I issue export http_proxy=http://localhost:8888 (= tinyproxy) immediately before running the CLI. Could it be possible that some transitive dependency, e.g. org.eclipse.aether.* ignores the proxy settings? Incidentally, I can't find see any org.eclipse stuff in my .m2/repository/. Where does the CLI find this dependency anyway?

[dsyer]

OK, I tried this again with M6 and see the same behaviour as you (i.e. dependencies resolved directly and not through the proxy). I must have been using an older version before. We need to look at it (maybe @wilkinsona can comment?).

I guess export http_proxy=http://localhost:8888 wouldn't have any effect on a Java process normally so it's not a surprise that had no effect. The JAVA_OPTS should have worked though.

Aether and friends ship with the spring CLI script (look in the lib directory adjacent to the script) so it's also not a surprise not to see it in ~/.m2/repository

[gerjantd]

Could the cause be an insufficiently populated DefaultRepositorySystemSession in the constructor of org.springframework.boot.cli.compiler.grape.AetherGrapeEngine? I'm having some trouble finding my way in the code, but the stack trace seems to point to this class, and http://wiki.eclipse.org/Aether/Creating_a_Repository_System_Session appears to suggest the newly created session could be populated with settings derived from the system's or user's Maven settings.xml.

public AetherGrapeEngine(GroovyClassLoader classLoader,
        List<RemoteRepository> remoteRepositories) {
    this.classLoader = classLoader;
    this.repositorySystem = createServiceLocator().getService(RepositorySystem.class);
    DefaultRepositorySystemSession session = MavenRepositorySystemUtils.newSession();
    LocalRepository localRepository = new LocalRepository(getM2RepoDirectory());
    LocalRepositoryManager localRepositoryManager = this.repositorySystem
            .newLocalRepositoryManager(session, localRepository);
    session.setLocalRepositoryManager(localRepositoryManager);
    this.session = session;
    this.repositories = new ArrayList<RemoteRepository>(remoteRepositories);
    this.progressReporter = getProgressReporter(session);
}

[dsyer]

Commit 47d079d should fix the immediate problem. I'm not sure it's really finished, but you can try it out and see if you can suggest improvements.

[gerjantd]

Works for me, thanks for the quick fix. I've found this (ancient) thread - http://jira.codehaus.org/browse/MNG-728 - with some background of why Maven has its own proxy config (in settings.xml) instead of relying on any JVM settings. I can't really judge whether or not this is (still) valid, but to parse Maven's settings.xml would seem more consistent when using Maven repos. Of course it's more of a pain to add this, just as you said in your commit comment. I'll have another look at the suggestions in http://wiki.eclipse.org/Aether/Creating_a_Repository_System_Session. If I'll manage to create something along the lines of http://git.eclipse.org/c/aether/aether-ant.git/tree/src/main/java/org/eclipse/aether/ant/AntRepoSys.java, I'll submit it for review.

[dsyer]

I'm actually more interested in the authentication piece (since it seems quite likely that people will want global configuration for that as well). I didn't see any docs in the JDK about conventions for that. Any ideas?

[gerjantd]

Well, if you're going to stick to JVM properties passed from the environment for now, you might as well look for and use http.proxyUser and http.proxyPassword. There's an alternative Proxy constructor that takes an extra Authentication argument. I confess I'm a bit confused by the abstractions in aether, but I think you could substitute the first return statement in your AetherGrapeEngine.defaultProxy method with something like this:

            String proxyUser = System.getProperty("http.proxyUser");
            String proxyPassword = System.getProperty("http.proxyPassword");
            if ((proxyUser != null) && (proxyPassword != null)) {
                AuthenticationBuilder auth = new AuthenticationBuilder();
                auth.addUsername(proxyUser).addPassword(proxyPassword);
                return new Proxy("http", proxyHost, new Integer(
                        System.getProperty("http.proxyPort", "80")),
                        auth.build());
            } else {
                return new Proxy("http", proxyHost, new Integer(
                        System.getProperty("http.proxyPort", "80")));
            }

[gerjantd]

The http://git.eclipse.org/c/aether/aether-ant.git/tree/src/main/java/org/eclipse/aether/ant/AntRepoSys.java takes a different approach. It sets a ProxySelector on the DefaultRepositorySystemSession. Interestingly, there's a org.eclipse.aether.util.repository.JreProxySelector implementation that uses the OS's proxy configuration. I'm not sure how to glue all this together.

/**
 * A proxy selector that uses the {@link java.net.ProxySelector#getDefault() JRE's global proxy selector}. In
 * combination with the system property {@code java.net.useSystemProxies}, this proxy selector can be employed to pick
 * up the proxy configuration from the operating system, see <a
 * href="http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html">Java Networking and Proxies</a> for
 * details. The {@link java.net.Authenticator JRE's global authenticator} is used to look up credentials for a proxy
 * when needed.
 */

[dsyer]

I tried that trick with the ProxySelector and it didn't work, perhaps because we inject the RemoteRepository instances manually into the RepositorySystem. It's worth having a look again though, especially if it deals with the authentication in some predictable semi-familiar way.

[bmancini42]

This seems to be still a problem, at least for me. (i.e. tried the Getting Started example behind proxy with spring run, "no route to host" errors). Running a @Grab in a Groovy script with the groovy command connects successfully.

[philwebb]

@bmancini42 Could you raise a new issue for the proxy problem that you are seeing? I don't want to re-open this one as the original fix went in some time ago.