Jsch - Opening multiple channels in a single session to SLES15 SP2 server is failing

When executing commands by opening multiple channels in a single session to a SLES server is failing. It is failing with the deafult ssh that is coming along with installation. Not seeing the issue when compiled the same version of SSH(8.1 & 8.4) from sources. Options used to compile:

./configure --with-md5-passwords --with-privsep-path=/var/lib/sshd/ --sysconfdir=/etc/ssh --with-pam --with-selinux

Details of the Libraries used:

JSCH version: 0.1.55

SSH Server Machine Details OS: SUSE Linux Enterprise Server 15 SP2 SSHD: OpenSSH_8.1p1, OpenSSL 1.1.1d

Errors from /var/log/messages: fatal: mm_answer_audit_server_key_free: buffer error: incomplete message fatal: mm_request_receive: read: Connection reset by peer fatal: mm_request_send: write: Broken pipe

Error in JSCH log: INFO: Connecting to 10.75.32.47 port 22 INFO: Connection established INFO: Remote version string: SSH-2.0-OpenSSH_8.1 INFO: Local version string: SSH-2.0-JSCH-0.1.54 INFO: CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256 INFO: CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521 INFO: CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 INFO: SSH_MSG_KEXINIT sent INFO: SSH_MSG_KEXINIT received INFO: kex: server: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 INFO: kex: server: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 INFO: kex: server: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com INFO: kex: server: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com INFO: kex: server: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 INFO: kex: server: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 INFO: kex: server: none,zlib@openssh.com INFO: kex: server: none,zlib@openssh.com INFO: kex: server: INFO: kex: server: INFO: kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1 INFO: kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 INFO: kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc INFO: kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc INFO: kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96 INFO: kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96 INFO: kex: client: none INFO: kex: client: none INFO: kex: client: INFO: kex: client: INFO: kex: server->client aes128-ctr hmac-sha1 none INFO: kex: client->server aes128-ctr hmac-sha1 none INFO: SSH_MSG_KEX_ECDH_INIT sent INFO: expecting SSH_MSG_KEX_ECDH_REPLY INFO: ssh_rsa_verify: signature true WARN: Permanently added '10.75.32.47' (RSA) to the list of known hosts. INFO: SSH_MSG_NEWKEYS sent INFO: SSH_MSG_NEWKEYS received INFO: SSH_MSG_SERVICE_REQUEST sent INFO: SSH_MSG_SERVICE_ACCEPT received INFO: Authentications that can continue: publickey,keyboard-interactive,password INFO: Next authentication method: publickey INFO: Authentications that can continue: keyboard-interactive,password INFO: Next authentication method: keyboard-interactive INFO: Authentication succeeded (keyboard-interactive). Waiting for all tasks to finish... INFO: Caught an exception, leaving main loop due to Connection reset INFO: Disconnecting from 10.75.32.47 port 22

Attaching the sample Java based code to reproduce the issue:

import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;

public class SSHTest {

    public static void main(String[] args) {
        try {
            JSch jsch = new JSch();
            Session session = jsch.getSession("userName", "hostAddress", 22);
            String command = "echo \"Hi\" && sleep 1";
            java.util.Properties config = new java.util.Properties();
            config.put("StrictHostKeyChecking", "no");
            config.put("PreferredAuthentications","publickey,keyboard-interactive,password");
            session.setConfig(config);
            session.setPassword("david");
            session.setTimeout(60000);
            session.connect();

            ExecutorService channelsPool = Executors.newFixedThreadPool(4);
            for (int i = 0; i < 4; i++) {
                channelsPool.execute(new TaskExecutor(command, session));
            }

            System.out.println("Waiting for all tasks to finish...");
            channelsPool.shutdown(); // Disable new tasks from being submitted
            try {
                // Wait a while for existing tasks to terminate
                if (!channelsPool.awaitTermination(60, TimeUnit.SECONDS)) {
                    channelsPool.shutdownNow(); // Cancel currently executing tasks
                    // Wait a while for tasks to respond to being cancelled
                    if (!channelsPool.awaitTermination(60, TimeUnit.SECONDS))
                        System.err.println("Pool did not terminate");
                } else {
                    System.out.println("All tasks finished successfully");
                }
            } catch (InterruptedException ie) {
                // (Re-)Cancel if current thread also interrupted
                channelsPool.shutdownNow();
                // Preserve interrupt status
                Thread.currentThread().interrupt();
            }

            session.disconnect();

            System.out.println("************************* Execution completed *************************");
        } catch (JSchException e) {
            e.printStackTrace();
        }
    }

}

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.util.ArrayList;

import com.jcraft.jsch.ChannelExec;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;

public class TaskExecutor implements Runnable {

    private String command;
    private Session session;

    public TaskExecutor(String command, Session session) {
        this.command = command;
        this.session = session;
    }

    @Override
    public void run() {
        try {
            execute();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public void execute() throws JSchException, IOException {
        ChannelExec channelExec = (ChannelExec) session.openChannel("exec");
        channelExec.setCommand(command);
        InputStream inputStream = channelExec.getInputStream();
        InputStream errorStream = channelExec.getErrStream();
        OutputStream outputStream = channelExec.getOutputStream();
        channelExec.connect();
        int exitCode = channelExec.getExitStatus();

        ArrayList<String> output = new ArrayList<>();
        ArrayList<String> error = new ArrayList<>();
        try (BufferedReader outputReader = new BufferedReader(new InputStreamReader(inputStream));
                BufferedReader errorReader = new BufferedReader(new InputStreamReader(errorStream))) {
            boolean isDataLeftInStream = true;
            // Read data in output stream
            while (isDataLeftInStream) {
                String line = outputReader.readLine();
                if (line != null) {
                    output.add(line);
                } else {
                    isDataLeftInStream = false;
                }
            }

            // Read data in error stream
            isDataLeftInStream = true;
            while (isDataLeftInStream) {
                String line = errorReader.readLine();
                if (line != null) {
                    error.add(line);
                } else {
                    isDataLeftInStream = false;
                }
            }
        }

        channelExec.disconnect();

        System.out.println("Execution Result: \n\tExitCode=" + exitCode + "\n\tOutput=" + String.join("\n", output)
                + "\n\tError=" + String.join("\n", error));
    }

}

SUSE-Developer-Community / explore.suse.dev

Jsch - Opening multiple channels in a single session to SLES15 SP2 server is failing #70