Add Proxy support to DefaultFtpSessionFactory [INT-2505]

[spring-operator]

Sloan Seaman opened INT-2505 and commented

DefaultFtpSessionFactory does not have Proxy support (though DefaultSftpSessionFactory does) and the only way to add proxy support is via System.properties (examples: http://marc.info/?l=jakarta-commons-user&m=107877944806547&w=2) however this then means that ALL FTP connections in the JVM will utilize the proxy which may not be desired (as in my case).

Other that writing custom code (and placing it in a org.springframework.integration.ftp.session package because FtpSession is protected) there is no way to have a specific FTP use a proxy while others do not.

Example custom code (not mine):

package org.springframework.integration.ftp.session;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.SocketException;

import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.core.NestedIOException;
import org.springframework.integration.MessagingException;
import org.springframework.integration.file.remote.session.Session;
import org.springframework.util.Assert;

/**
 * Taken from
 * http://forum.springsource.org/showthread.php?120012-Restarting-a-previously-stopped-FTP-inbound-channel-adapter
 *
 * @author gunnar@springsource forum
 *
 */
public class ProxyFtpSessionFactory extends AbstractFtpSessionFactory<FTPClient> {

    private static final Logger LOG = LoggerFactory.getLogger(ProxyFtpSessionFactory.class);

    private String proxyHost;
    private int proxyPort;
    private String proxyUsername;
    private String proxyPassword;

    @Override
    public Session<FTPFile> getSession() {
        try {
            return new FtpSession(this.createClient());
        }
        catch (Exception e) {
            throw new IllegalStateException("failed to create FTPClient", e);
        }
    }

    private FTPClient createClient() throws SocketException, IOException {
        final FTPClient client = this.createClientInstance();
        Assert.notNull(client, "client must not be null");
        client.configure(this.config);
        Assert.hasText(this.username, "username is required");

        this.postProcessClientBeforeConnect(client);

        // Connect
        boolean usingProxy = false;
        if (proxyHost != null && proxyHost.length() > 0) {
            // Connect using proxy with authentication
            Assert.isTrue(proxyPort > 0, "proxyPort number should be > 0");
            usingProxy = true;
            // connect
            try {
                client.connect(((InetSocketAddress) new Proxy(Proxy.Type.HTTP,
                        new InetSocketAddress(proxyHost, proxyPort)).address()).getAddress());
            } catch (IOException ioe) {
                throw new NestedIOException("Connecting to proxy [" +
                        proxyHost + ":" + proxyPort + "] failed. Please check the connection.", ioe);
            }
            // check reply
            if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
                throw new MessagingException("Connecting to proxy [" +
                        proxyHost + ":" + proxyPort + "] failed. Please check the connection.");
            }
            LOG.debug("Connected to proxy [{}:{}]", proxyHost, proxyPort);
        } else {
            // Connect using no proxy
            try {
                client.connect(this.host, this.port);
            } catch (IOException ioe) {
                throw new NestedIOException("Connecting to server [" +
                        host + ":" + port + "] failed. Please check the connection.", ioe);
            }
            // check reply
            if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
                throw new MessagingException("Connecting to server [" +
                        host + ":" + port + "] failed. Please check the connection.");
            }
            LOG.debug("Connected to server [{}:{}]", host, port);
        }

        if (usingProxy && proxyUsername != null) {
            // login using proxy
            try {
                if (!client.login(username + "@" + host + " " + proxyUsername, password, proxyPassword)) {
                    throw new IllegalStateException("Login to server [" + username + "@" + host + ":" + port + "] using proxy [" +
                            proxyUsername + "@" + proxyHost + ":" + proxyPort + "] failed. The response from the server is: " +
                            client.getReplyString());
                }
            } catch (IOException ioe) {
                throw new NestedIOException("Login to server [" + username + "@" + host + ":" + port + "] using proxy [" +
                        proxyUsername + "@" + proxyHost + ":" + proxyPort + "] failed.", ioe);
            }
        } else {
            // direct login
            try {
                if (!client.login(username, password)) {
                    throw new IllegalStateException("Login to server [" + username + "@" + host + ":" + port +
                            "] failed. The response from the server is: " +
                            client.getReplyString());
                }
            } catch (IOException ioe) {
                throw new NestedIOException("Login to server [" + username + "@" + host + ":" + port +
                        "] failed.", ioe);
            }
        }
        LOG.debug("Connected to server [{}:{}]: {}", new Object[]{host, port, client.getReplyString()});

        this.postProcessClientAfterConnect(client);

        this.updateClientMode(client);
        client.setFileType(fileType);
        client.setBufferSize(bufferSize);
        return client;
    }

    private void updateClientMode(FTPClient client) {
        switch (this.clientMode) {
            case FTPClient.ACTIVE_LOCAL_DATA_CONNECTION_MODE:
                client.enterLocalActiveMode();
                break;
            case FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE:
                client.enterLocalPassiveMode();
                break;
            default:
                break;
        }
    }

    @Override
    protected FTPClient createClientInstance() {
        return new FTPClient();
    }

    @Required
    public void setProxyHost(String proxyHost) {
        this.proxyHost = proxyHost;
    }

    @Required
    public void setProxyPort(int proxyPort) {
        this.proxyPort = proxyPort;
    }

    public void setProxyUsername(String proxyUsername) {
        this.proxyUsername = proxyUsername;
    }

    public void setProxyPassword(String proxyPassword) {
        this.proxyPassword = proxyPassword;
    }
}

---

Affects: 2.1 GA

Reference URL: http://forum.springsource.org/showthread.php?125107-DefaultFtpSessionFactory-and-Proxy

3 votes, 5 watchers

[spring-operator]

Sloan Seaman commented

Update to code that I am using (previous version was untested and taken from a forum, this version I have tested against a Blue Coat proxy)

package org.springframework.integration.ftp.session;

import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.net.SocketException;

import org.apache.commons.net.ftp.FTPClient;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.core.NestedIOException;
import org.springframework.integration.MessagingException;
import org.springframework.integration.file.remote.session.Session;
import org.springframework.util.Assert;

/**
 * Taken from
 * http://forum.springsource.org/showthread.php?120012-Restarting-a-previously-stopped-FTP-inbound-channel-adapter
 *
 * @author gunnar@springsource forum
 *
 */
public class ProxyFtpSessionFactory extends AbstractFtpSessionFactory<FTPClient> {

    private static final Logger LOG = LoggerFactory.getLogger(ProxyFtpSessionFactory.class);

    private String proxyHost;
    private int proxyPort;
    private String proxyUsername;
    private String proxyPassword;

    @Override
    public Session<FTPFile> getSession() {
        try {
            return new FtpSession(this.createClient());
        }
        catch (Exception e) {
            throw new IllegalStateException("failed to create FTPClient", e);
        }
    }

    private FTPClient createClient() throws SocketException, IOException {
        final FTPClient client = this.createClientInstance();

        // if we don't have this we get  Host attempting data connection xx.xx.xx.xx is not same as server xx.xx.xx.xx
        client.setRemoteVerificationEnabled(false);

        Assert.notNull(client, "client must not be null");
        client.configure(this.config);
        Assert.hasText(this.username, "username is required");

        this.postProcessClientBeforeConnect(client);

        // Connect
        boolean usingProxy = false;
        if (proxyHost != null && proxyHost.length() > 0) {
            // Connect using proxy with authentication
            Assert.isTrue(proxyPort > 0, "proxyPort number should be > 0");
            usingProxy = true;
            // connect
            try {
                client.connect(((InetSocketAddress) new Proxy(Proxy.Type.HTTP,
                        new InetSocketAddress(proxyHost, proxyPort)).address()).getAddress());
            } catch (IOException ioe) {
                throw new NestedIOException("Connecting to proxy [" +
                        proxyHost + ":" + proxyPort + "] failed. Please check the connection.", ioe);
            }
            // check reply
            if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
                throw new MessagingException("Connecting to proxy [" +
                        proxyHost + ":" + proxyPort + "] failed. Please check the connection.");
            }
            LOG.debug("Connected to proxy [{}:{}]", proxyHost, proxyPort);
        } else {
            // Connect using no proxy
            try {
                client.connect(this.host, this.port);
            } catch (IOException ioe) {
                throw new NestedIOException("Connecting to server [" +
                        host + ":" + port + "] failed. Please check the connection.", ioe);
            }
            // check reply
            if (!FTPReply.isPositiveCompletion(client.getReplyCode())) {
                throw new MessagingException("Connecting to server [" +
                        host + ":" + port + "] failed. Please check the connection.");
            }
            LOG.debug("Connected to server [{}:{}]", host, port);
        }

        if (usingProxy && proxyUsername != null) {
            // login using proxy w/ auth
            LOG.debug("Login attempt to server [" + username + "@" +host + ":" + port + "] using proxy w/ authentication");
            try {
                if (!client.login(username + "@" + host + " " + proxyUsername, password, proxyPassword)) {
                    throw new IllegalStateException("Login to server [" + username + "@" + host + ":" + port + "] using proxy [" +
                            proxyUsername + "@" + proxyHost + ":" + proxyPort + "] failed. The response from the server is: " +
                            client.getReplyString());
                }
            } catch (IOException ioe) {
                throw new NestedIOException("Login to server [" + username + "@" + host + ":" + port + "] using proxy [" +
                        proxyUsername + "@" + proxyHost + ":" + proxyPort + "] failed.", ioe);
            }
        }
        else if (usingProxy) {
            // login using proxy
            LOG.debug("Login attempt to server [" + username + "@" +host + ":" + port + "] using proxy");
            try {
                if (!client.login(username + "@" + host, password)) {
                    throw new IllegalStateException("Login to server [" + username + "@" + host + ":" + port +
                            "] failed. The response from the server is: " +
                            client.getReplyString());
                }
            } catch (IOException ioe) {
                throw new NestedIOException("Login to server [" + username + "@" + host + ":" + port +
                        "] failed.", ioe);
            }
        } else {
            // direct login
            LOG.debug("Login attempt to server [" + username + "@" +host + ":" + port + "]");
            try {
                if (!client.login(username, password)) {
                    throw new IllegalStateException("Login to server [" + username + "@" + host + ":" + port +
                            "] failed. The response from the server is: " +
                            client.getReplyString());
                }
            } catch (IOException ioe) {
                throw new NestedIOException("Login to server [" + username + "@" + host + ":" + port +
                        "] failed.", ioe);
            }
        }
        LOG.debug("Connected to server [{}:{}]: {}", new Object[]{host, port, client.getReplyString()});

        this.postProcessClientAfterConnect(client);

        this.updateClientMode(client);
        client.setFileType(fileType);
        client.setBufferSize(bufferSize);
        return client;
    }

    private void updateClientMode(FTPClient client) {
        switch (this.clientMode) {
            case FTPClient.ACTIVE_LOCAL_DATA_CONNECTION_MODE:
                client.enterLocalActiveMode();
                break;
            case FTPClient.PASSIVE_LOCAL_DATA_CONNECTION_MODE:
                client.enterLocalPassiveMode();
                break;
            default:
                break;
        }
    }

    @Override
    protected FTPClient createClientInstance() {
        return new FTPClient();
    }

    @Required
    public void setProxyHost(String proxyHost) {
        this.proxyHost = proxyHost;
    }

    @Required
    public void setProxyPort(int proxyPort) {
        this.proxyPort = proxyPort;
    }

    public void setProxyUsername(String proxyUsername) {
        this.proxyUsername = proxyUsername;
    }

    public void setProxyPassword(String proxyPassword) {
        this.proxyPassword = proxyPassword;
    }

}

[spring-operator]

Oleg Zhurakousky commented

I am going to push this issue as I am discovering a lot of inconsistencies while testing with other Proxy servers (e.g., Squid) or without proxy servers at all. Here is what worries me the most

client.connect(((InetSocketAddress) new Proxy(Proxy.Type.HTTP,
    new InetSocketAddress(proxyHost, proxyPort)).address()).getAddress());

First I can't find any references from Apache showing an example of connecting via proxy in this way. Everything points to using a System properties. But that's OK However what's not OK is that this code always works no matter if I run proxy or not. . if i specify the correct proxyHost/proxyPort or not. I always get 220 - Service ready for new user.

I am finding a lot of questions about this with not a whole lot of answers, so at this point I am not comfortable with pushing a functionality that can not be adequately tested.

Also I see a lot of developers getting their answers using ftp4j-1.5.1 library, so we may need to investigate and possible switch to that as well.

For now I am moving it to the 2.2 backlog. I'll also push the branch with the current state of the code so you can try (I'll post the link momentarily)

[spring-operator]

Oleg Zhurakousky commented

Here is the link to my branch so you can test it out https://github.com/olegz/spring-integration/tree/INT-2505

[spring-operator]

Sloan Seaman commented

Thanks! I'll attempt to test it in the next few days

[spring-operator]

Sloan Seaman commented

The test code uses: ProxiedFtpSessionFactory but that code is not in the branch (at least, not in spring-integration / spring-integration-ftp / src / main / java / org / springframework / integration / ftp / session) I'm probably looking in the wrong place though....

[spring-operator]

Oleg Zhurakousky commented

Sorry I forgot to add it, i'll do it later but it wasn't any different than yours and as I said the connect part is what worries me the most when I realize that the connect is successful even if proxy server was not running

[spring-operator]

Ivan Lagunov commented

Oleg,

I also need this feature. We have SOCKS4 proxy and need to establish FTP connections over it. At the same time we have SFTP connections that don't work via proxy (so setting proxy for the whole JVM is a bad idea). I hope to see this feature in 2.2.

[spring-operator]

Gary Russell commented

This did not make it into the 2.2 release.

However, to address a comment in the original submission, the FtpSession is no longer package protected; it is public in the upcoming 2.2 release, and in 2.1 since 2.1.1.RELEASE.

So, the original poster's suggestion should be a viable workaround.

[spring-operator]

Artem Bilan commented

Some external request on the matter: https://stackoverflow.com/questions/49997030/does-spring-integration-have-a-proxy-support-for-ftp

[spring-operator]

Artem Bilan commented

There is FTPClient.setProxy() for our consideration:

/**
  * Sets the proxy for use with all the connections.
  * The proxy is used for connections established after the
  * call to this method.
  *
  * @param proxy the new proxy for connections.
  * @since 3.2
  */
 public void setProxy(Proxy proxy) {

So, we can just expose this option on the AbstractFtpSessionFactory.

On the other hand there is an FTPHTTPClient for HTTP Proxy tunneling. We can consider to implement an FtpHttpSessionFactory as well.