[JENKINS-55971] SSH Host key matches, Authentication failed, Slaves failed to reconnect

[timja]

SSHLauncher{host='myhost', port=22, credentialsId='aaa-bbb-ccc-myRealCredentials', jvmOptions='', javaPath='/home/bla/jenkins/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.ManuallyTrustedKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[02/04/19 22:09:16] [SSH] Opening SSH connection to ligenr10.swb.siemens.de:22.
[02/04/19 22:09:16] [SSH] SSH host key matches key seen previously for this host. Connection will be allowed.
[02/04/19 22:09:16] [SSH] Authentication failed.

Manage old data complained on startup - ( didn't take real notice of message )

Credentials, private key, was corrupted when starting Jenkins after updating to 2.163

Eventually found and resolved.

( Jenkins- Credentials->System->Global Credentials -> name )

( updated Private Key - entered directly )

Might not be a bug but user error. However, caused 8 hour loss of Jenkins until problem found.

 

---

Originally reported by sgjenkins, imported from: SSH Host key matches, Authentication failed, Slaves failed to reconnect

• status: In Review
• priority: Minor
• resolution: Unresolved
• imported: 2022/01/10

[timja]

dnusbaum:

Perhaps another case of a migration issue with https://jenkins.io/security/advisory/2018-06-25/#SECURITY-440? CC wfollonier, but AFAIR we have never been able to reproduce this from scratch (the migration is supposed to be handled automatically). What version of Jenkins and plugins did you update from/to?

[timja]

sgjenkins:

Thanks for looking into it.

I upgraded from 2.162 -> 2.163

All plugins are the latest available ( and when 2.163 was working I checked all plugins again ).

My test system with only 2 slaves (the two machines are also in my live system, same master) worked which is why I applied the update to my live system. The two systems are however not identical, I don't need all plugins on my test system, but are up to date regarding ssh and credentials plugins.

 

[timja]

dnusbaum:

If you didn't update any plugins, then I really don't know what could have happened. Are there lines in your logs at the time of the upgrade containing "SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource"?

[timja]

sgjenkins:

Yes - hundreds since at least Jun 27 2018, the last one :

Feb 03, 2019 4:52:13 PM com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey$UsersPrivateKeySource readResolve
INFO: SECURITY-440: Migrating UsersPrivateKeySource to DirectEntryPrivateKeySource

I can't see what causes these messages. It does not seem to be associated to a build, maybe only with a new start ?

Actually, there are no such messages in the log from yesterday ( 5th ) or today.  My upgrade was on the 4th. Finally worked at approx 22:30 ( CET ) 

 

[timja]

dnusbaum:

Well, I'm glad it's working for you now. See similar reports of issues with that security fix in JENKINS-54746 and JENKINS-52232. Looking at JENKINS-54746 I think the exception in this comment (this is probably what you saw in the old data monitor) is the root cause. The migration is gated by the RunScripts permission being active when that code runs, and that code is expected to be run as ACL.System, but for some reason it ran as ACL.anonymous. When the exception is thrown in readResolve, perhaps the effect is that the serialized data is totally lost and it is as if no private keys were ever entered. Perhaps this code should be modified to instead just return if that permission is not found to avoid the conversion exception.

[timja]

dnusbaum:

Closing as a dupe of JENKINS-52232 to keep the discussion in one place.

[timja]

sgjenkins:

Thanks for looking into it. 

[timja]

[Duplicates: JENKINS-52232]