known_hosts file - duplicate entries

[avnerw74]

Hi,

First, many thanks a lot for taking care of this project!

I'm experiencing a strange issue (using JSCH 0.2.7) where for a specific remote host, JSCH is constantly adding a new entry in the known_hosts file, although it already exists there.

After few weeks of work, the known_hosts file grew up to ~7 MB and contained 10K entries (around 9500!! of them are duplicates...). I am not sure what is causing this issue. The strict hostkey checking configuration value is set to default (which I believe it is "ask").

Below is an example of one of the entries (I changed few characters in the key) mysftphostname.some-company.net ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCVQEGv7gMw0iOz+Wj0mOU42EXRWfWdFJaLOu0gvrZzk+eUbeXouknZ2IT0IVlJMYo3Wx0D/6sf1VpaJV2A7PFRAlJIk4GiYRqsGJsYEiFSACaqCW5BLzQI/UtGEJHqlx7G3PI5Glth+6YGaQRRSfqPj5wU1pXfeif/K6YTMvdmwXI0AwAy377+7ERx1+DJ8CwPrbxFJj2ZbaX15Lg7j4yu5gMtB2se8/fCoKyON7awnOtiICWpMeJPgNgNfouvrH8n02B5NdhU1zeH/0tReu9+Zl+IdHlaZPd0b/ZKFn7wUBdz12gJWCkOdvaUiWrYRkBaU+q8phF1faVLrbLcWbwkDmo+q0/+qDFRUD6LUTIPPvtX78EM27c6a8HjSPLr6Cxh+7El/YibcFLWx6HTyGZo9X8O7f0HzlnKhGwKzNlbgGPn5JEgMn9Vuxr9QLa2AyhM9qlXHPNv9Kdw+YgTgXbPI7FPC9IWHiKgibwG/QPB7dxvimGWfluyWqtjUS3blc2CnYSXONAluvdMwpNV3mQR539mo+JjYTzlxaz/OMnwtBrfuMwEVYjz4IksvQeeJE38Q6bydWpWlJ+zGCTmujsUCQIbMsYprN4aVA3wkbaDuo4ChpMgQHI0ob3VYV/3RMPvELC6I8rZl6bdTAPjN7wZ+mXklEwazWo5rf0XbrFBgQ==

If it may help: The length of the key in base64 is 716. The length of the whole entry is 749

Have you encountered this behavior before?

Thanks, Avner

[mwiede]

That sounds strange. Can you show us the code you are using? Do you call com.jcraft.jsch.KnownHosts somewhere and how?

[norrisjeremy]

Hi @mwiede,

I suspect this is because there is code commented out in the KnownHosts.add() method here.

I'm not sure why this code is commented out: the earliest source code release I can find on Maven Central is the 0.1.27 release from 2006 and it was commented out as far back as then. I also can't find any mention in the original ChangeLog indicating why either.

Thanks, Jeremy

[norrisjeremy]

Hi @mwiede,

I suspect this is because there is code commented out in the KnownHosts.add() method here.

I'm not sure why this code is commented out: the earliest source code release I can find on Maven Central is the 0.1.27 release from 2006 and it was commented out as far back as then. I also can't find any mention in the original ChangeLog indicating why either.

Thanks, Jeremy

FYI, I was able to find the first release that contained the KnownHosts class from SourceForge (0.0.12 from 2003) and this code appears to always have been commented out. I'm not sure if it would be safe to simply uncomment it, or if there was some other reasoning behind this decision to leave it this way.

[norrisjeremy]

Looking at this further, I'm not sure how we are hitting the condition in which KnownHosts.add() is called repeatedly in the first place for this host.

@avnerw74 Do you provide JSch an implementation of UserInfo (by calling Session.setUserInfo()) by any chance? Also, do you happen to launch multiple sessions to this host concurrently in your application?

[avnerw74]

Hi @norrisjeremy , @mwiede ,

Thanks for looking at this issue. The customer just updated me that they mistakenly added a space before the hostname (e.g.: " myhost" instead of "myhost"), and after removing the extra space, it didn't duplicate entries, so it might be that JSch is ignoring the space when connecting to the server (as it connected OK), but doesn't ignore(/trim the space) when comparing to existing hostkey entry in the known_hosts.

I'll check this in the lab and update.

Thanks, Avner

[avnerw74]

@norrisjeremy - short update: I couldn't reproduce this in my environment. Jsch failed to connect to host with leading space (UnknownHostException). I think the issue is:

1. On their end for some reason " host" (with a leading space) is somehow accepted (as mentioned, on my end it fails.
2. Once it is accepted, it adds the entry to the known hosts file. So for example is the host is " host2" , the known_hosts file will look like: host1 ssh-rsa (space)host2 ssh-rsa

I wrote (space) since I noticed it is not shown OK in github posts... but imagine it is a real space :)

3. Now, when an host-key of one of the hosts changes or when adding a new hostkey, I noticed that Jsch writes the file again, but without the leading spaces. So, the file may look like: host1 ssh-rsa host2 ssh-rsa host3 ssh-rsa (notice the 2nd line has no leading space now).
4. Now the connect again to the " host2" definition. It cannot find " host2" in the file, so it added it again. host1 ssh-rsa host2 ssh-rsa host3 ssh-rsa (space)host2 ssh-rsa

So, as they have many connections every day, after a 2-3 weeks, they had almost ~10K entries of that 'leading space host', and it looked like: host1 ssh-rsa host2 ssh-rsa host3 ssh-rsa host2 ssh-rsa host4 ssh-rsa host2 ssh-rsa host2 ssh-rsa host5 ssh-rsa (space)host2 ssh-rsa

So, I think the hostname should be trimmed before the hostkey checking, no?

[avnerw74]

Hi @norrisjeremy , @mwiede,

Indeed, that was the scenario, caused by the fact that they were able to connect to " hostname" with a leading space (probably their DNS is permissive for leading/trailing spaces).

• when connecting to " hostname", jsch is writing the leading space to the known_hosts file
• when reading the file, jsch is loading the hostname to the vector without the leading space
• when jsch then updating the known_hosts file, the original space is removed
• when connecting to " " again, another entry is added...

We solved it by trimming the hostname (in our app).

Would you like to consider trimming the hostname in jsch, to avoid such scenario?

Thanks, Avner

[norrisjeremy]

I can look into trying to add some trimming inside JSch, but it may be a bit until I have some free time to implement (especially since this is a pretty unusual scenario that is difficult to trigger and there is an easy workaround).

[Jlai21321]

Hi @norrisjeremy , @mwiede,

I've also a similar problem as well.

The same issue can easily be reproduced by using the special character comma in the Host Entry.

For example, host1,ip. or ip,host format

will also result in duplication of the known hosts on every call.

[preeghos]

Hi @norrisjeremy , @mwiede,

we have also encountered a similar problem as well. We noticed that Jsch writes the file again and again and the file keeps on growing:

avocet.xxxxxx.ibm.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKO5N1iB60alqct+klwiYfTa/eqcMyuSLaTNAyrHIZiu avocet.xxxxxx.ibm.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKO5N1iB60alqct+klwiYfTa/eqcMyuSLaTNAyrHIZiu

It results in duplication of the known hosts on every call it makes.

JSCH TRACE (000000A4-65FD0696-00000022) , 'WARN: ', 'Permanently added 'avocet.xxxxx.ibm.com' (EDDSA) to the list of known hosts.'

[preeghos]

Hi @norrisjeremy , @mwiede,

The workaround mentioned in the ticket did not work for us. Would it be possible to add some trimming inside JSch. Thanks

[norrisjeremy]

Hi @preeghos,

When I last investigated this awhile ago, I was unable to reproduce the problem, so I was loathe to make any changes. If you can provide a simple test case that reproduces the issue, that would help us develop a solution.

Thanks, Jeremy

[preeghos]

Hi @norrisjeremy , @mwiede, We could reproduce the problem with our simple java code and we could find that the known_hosts keeps on increasing with duplicate entries. Attaching the java program for reference.

*****Java Code Start**** import java.nio.file.Paths; import java.util.Properties;

import com.jcraft.jsch.*;

public class SftpTest_Password{

public static void main(String[] args){

try{

  JSch jsch=new JSch(); 
  jsch.setLogger(new JSchTracer());

  if(args.length < 3){
      System.out.println("Usage : java SftpTest hostname port username password");
      System.exit(0);
  }

  String host=args[0];
  int port=Integer.parseInt(args[1]);
  String user=args[2];
  String password= args[3];
  jsch.setKnownHosts("C:\\Users\\xyz\\known_hosts");

  Session session=jsch.getSession(user, host, port);
  Properties config = new Properties();

  config.put("StrictHostKeyChecking","no"); 
  config.put("PreferredAuthentications", "password,keyboard-interactive,gssapi-with-mic");  

  session.setConfig(config);      
  session.setPassword(password);
  System.out.println("Going to connect to " + host + " using id " + user + " and password " + password);
  session.connect();
  System.out.println("Connection succeeded");
  session.disconnect();

}catch(Exception e){
    System.out.println(e);
    e.printStackTrace();
}

} }

class JSchTracer implements com.jcraft.jsch.Logger { static java.util.Hashtable<Integer, String> props = new java.util.Hashtable<Integer, String>(); static { props.put(new Integer(DEBUG), "DEBUG: "); props.put(new Integer(INFO), "INFO: "); props.put(new Integer(WARN), "WARN: "); props.put(new Integer(ERROR), "ERROR: "); props.put(new Integer(FATAL), "FATAL: "); }

public boolean isEnabled(int level) {
        return true;
}

public void log(int level, String message) {         
        System.out.println(message);          
}

} ****Java Code ends***

Our known_hosts file entries: xyz.ibm.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKO5N1iB60alqct+klwiYfTa/eqcMyuSLaTNAyrHIZiu xyz.hursley.ibm.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKO5N1iB60alqct+klwiYfTa/eqcMyuSLaTNAyrHIZiu xyz.hursley.ibm.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKO5N1iB60alqct+klwiYfTa/eqcMyuSLaTNAyrHIZiu

Can you please look into the issue. We have customers chasing for the solution.

Thanks, Preetha