SFTP Error Invalid packet: indicated length 1349281121 too large

[niedbalski]

Error acquiring SFTPClient() (attempt 2 of 7): Invalid packet: indicated length 1349281121 too large . Is this related to sshj or to jclouds directly ?

[jclouds]

did it fail on that attempt?

On Tue, Jul 24, 2012 at 8:41 AM, Jorge Niedbalski reply@reply.github.com wrote:

Error acquiring SFTPClient() (attempt 2 of 7): Invalid packet: indicated length 1349281121 too large . Is this related to sshj or to jclouds directly ?

---

Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/issues/748

[niedbalski]

Yep, it failed

[jclouds]

I'd pass the issue to sshj. Meanwhile, you can try jsch to ensure it works with an alternative (could be something server-side, for ex)

On Tue, Jul 24, 2012 at 11:17 AM, Jorge Niedbalski reply@reply.github.com wrote:

Yep, it failed

---

Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/issues/748#issuecomment-7218267

[niedbalski]

This was triggered running th jenkins-jclouds plugin, for some reason

INFO: Client identity string: SSH-2.0-SSHJ_0_8_1_SNAPSHOT Jul 24, 2012 3:38:25 PM net.schmizz.sshj.transport.TransportImpl init INFO: Server identity string: SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1 Jul 24, 2012 3:38:27 PM net.schmizz.sshj.connection.channel.direct.SessionChannel startSubsystem INFO: Will request sftp subsystem Jul 24, 2012 3:38:27 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect INFO: Disconnected - BY_APPLICATION Jul 24, 2012 3:38:27 PM org.jclouds.logging.jdk.JDKLogger logError SEVERE: << (root:rsa[fingerprint(0c:40:28:c3:76:fa:16:cd:e6:d0:99:9b:cd:2e:de:cb),sha1(a0:32:c7:eb:68:ee:5d:6b:86:7d:e1:e4:6b:78:81:0a:5c:a1:36:5b)]@172.25.0.191:22) error acquiring SFTPClient() (out of retries - max 7): Invalid packet: indicated length 1349281121 too large

[karel1980]

import struct print struct.pack('>I', 1349281121) Plea

The first four bytes from 'Please log in as {user} instead of root' are interpreted as a number,

Read the whole story here:

https://github.com/shikhar/sshj/issues/75 http://www.snailbook.com/faq/sftp-corruption.auto.html

I'm not entirely sure who really is at fault here. The 'obvious' solution would be to setup the target machine so it doesn't send that 'Pleas log in as {user}' message.

Specifically for EC2 you should hunt down the person who created the image you're using and kindly ask them to change this :).

[karel1980]

Forget that. The 'Please log in as {user}' message is more than a warning: the login is actively denied. In that case you really should use override the credentials with the correct ones.

[jclouds]

Agreed, and by override, this means either TemplateOptions.overrideLoginUser or the same method in RunScriptOptions

On Fri, Aug 10, 2012 at 6:58 AM, Karel Vervaeke notifications@github.comwrote:

Forget that. The 'Please log in as {user}' message is more than a warning: the login is actively denied. In that case you really should use override the credentials with the correct ones.

— Reply to this email directly or view it on GitHubhttps://github.com/jclouds/jclouds/issues/748#issuecomment-7642679.

[SteveJones]

I'm running into this problem. The weird part is that denying root connections appears to be part of the bootstrap script sent by JClouds. i.e. this bit (from my logs):

exec 3<> /etc/ssh/sshd_config && awk -v TEXT="PasswordAuthentication no
PermitRootLogin no
" 'BEGIN {print TEXT}{print}' /etc/ssh/sshd_config >&3
hash service 2>&- && service ssh reload 2>&- || /etc/init.d/ssh* reload

This is not part of the init script I have set in the config, it appears to be coming out of JClouds.

The logs seem to show success in running the bootstrap script, I get this line:

INFO: Will request to exec `/tmp/init-bootstrap start`

followed by this one multiple times:

INFO: Will request to exec `/tmp/init-bootstrap status`

Then this happens (when trying to send slave.jar?):

NFO: >> blocking on sockets [23.20.152.84:22, 10.46.186.246:22] for 600000 MILLISECONDS
Sep 7, 2012 2:48:05 PM org.jclouds.logging.jdk.JDKLogger logInfo
INFO: << socket 10.46.186.246:22 opened
Sep 7, 2012 2:48:05 PM net.schmizz.sshj.transport.TransportImpl init
INFO: Client identity string: SSH-2.0-SSHJ_0_8_1_SNAPSHOT
Sep 7, 2012 2:48:05 PM net.schmizz.sshj.transport.TransportImpl init
INFO: Server identity string: SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
Sep 7, 2012 2:48:05 PM net.schmizz.sshj.connection.channel.direct.SessionChannel exec
INFO: Will request to exec `/tmp/init-bootstrap status`
Sep 7, 2012 2:48:08 PM net.schmizz.sshj.connection.channel.direct.SessionChannel startSubsystem
INFO: Will request `sftp` subsystem
Sep 7, 2012 2:48:08 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect
INFO: Disconnected - BY_APPLICATION
Sep 7, 2012 2:48:08 PM org.jclouds.logging.jdk.JDKLogger logInfo
INFO: << (root:rsa[fingerprint(25:e0:34:f2:e8:a6:34:a4:9f:a7:2d:0b:77:d9:55:14),sha1(f4:ed:91:d7:b6:00:62:17:1e:ee:93:93:6e:3b:1f:98:92:be:9e:58)]@10.46.186.246:22) error acquiring SFTPClient() (attempt 1 of 7): Invalid packet: indicated length 1349281121 too large

It looks like the JClouds disables root logins then attempts to send slave.jar as root. The jenkins user is set and I've tried with and without the admin user set to ubuntu (the default user on the machine).

[codefromthecrypt]

Hmm sounds weird. @abayer any insight On Sep 7, 2012 8:08 AM, "Steve Jones" notifications@github.com wrote:

I'm running into this problem. The weird part is that denying root connections appears to be part of the bootstrap script sent by JClouds. i.e. this bit (from my logs):

exec 3<> /etc/ssh/sshd_config && awk -v TEXT="PasswordAuthentication noPermitRootLogin no" 'BEGIN {print TEXT}{print}' /etc/ssh/sshd_config >&3hash service 2>&- && service ssh reload 2>&- || /etc/init.d/ssh* reload

This is not part of the init script I have set in the config, it appears to be coming out of JClouds.

The logs seem to show success in running the bootstrap script, I get this line:

INFO: Will request to exec /tmp/init-bootstrap start

followed by this one multiple times:

INFO: Will request to exec /tmp/init-bootstrap status

Then this happens (when trying to send slave.jar?):

NFO: >> blocking on sockets [23.20.152.84:22, 10.46.186.246:22] for 600000 MILLISECONDS Sep 7, 2012 2:48:05 PM org.jclouds.logging.jdk.JDKLogger logInfo INFO: << socket 10.46.186.246:22 opened Sep 7, 2012 2:48:05 PM net.schmizz.sshj.transport.TransportImpl init INFO: Client identity string: SSH-2.0-SSHJ_0_8_1_SNAPSHOT Sep 7, 2012 2:48:05 PM net.schmizz.sshj.transport.TransportImpl init INFO: Server identity string: SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1 Sep 7, 2012 2:48:05 PM net.schmizz.sshj.connection.channel.direct.SessionChannel exec INFO: Will request to exec /tmp/init-bootstrap status Sep 7, 2012 2:48:08 PM net.schmizz.sshj.connection.channel.direct.SessionChannel startSubsystem INFO: Will request sftp subsystem Sep 7, 2012 2:48:08 PM net.schmizz.sshj.transport.TransportImpl$1 notifyDisconnect INFO: Disconnected - BY_APPLICATION Sep 7, 2012 2:48:08 PM org.jclouds.logging.jdk.JDKLogger logInfo INFO: << (root:rsa[fingerprint(25:e0:34:f2:e8:a6:34:a4:9f:a7:2d:0b:77:d9:55:14),sha1(f4:ed:91:d7:b6:00:62:17:1e:ee:93:93:6e:3b:1f:98:92:be:9e:58)]@10.46.186.246:22) error acquiring SFTPClient() (attempt 1 of 7): Invalid packet: indicated length 1349281121 too large

It looks like the JClouds disables root logins then attempts to send slave.jar as root. The jenkins user is set and I've tried with and without the admin user set to ubuntu (the default user on the machine).

— Reply to this email directly or view it on GitHubhttps://github.com/jclouds/jclouds/issues/748#issuecomment-8368078.

[karel1980]

The snippet is produced by scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/ssh/SshStatements.java and scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/ssh/SshdConfig.java

There is a 'lockSshd' static method there which is responsible. This is in turn called from scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/login/AdminAccess.java AdminAccess has javadocs explaining its purpose. I don't know why it would be called in your case.

Other than that, thank you for making my day by bringing up this lovely construct:

exec 3<> /etc/ssh/sshd_config && awk -v TEXT="PasswordAuthentication no
PermitRootLogin no
" 'BEGIN {print TEXT}{print}' /etc/ssh/sshd_config >&3
hash service 2>&- && service ssh reload 2>&- || /etc/init.d/ssh* reload

First time I see the 'hash' builtin & wtf does >&- even mean?

[codefromthecrypt]

Haha, so there is a flag to AdminAccess to turn off lock ssh, so we can look at that.

This is the most interesting awk I've used :) hash checks that a command resolves (similar to which) >&- turns off a file descriptor similar to

/dev/null On Sep 7, 2012 8:36 AM, "Karel Vervaeke" notifications@github.com wrote:

The snippet is produced by

scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/ssh/SshStatements.java

There is a 'lockSshd' static method there which is responsible. This is in turn called from

scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/login/AdminAccess.java AdminAccess has javadocs explaining its purpose. I don't know why it would be called in your case.

Other than that, thank you for making my day by bringing up this lovely construct:

exec 3<> /etc/ssh/sshd_config && awk -v TEXT="PasswordAuthentication noPermitRootLogin no" 'BEGIN {print TEXT}{print}' /etc/ssh/sshd_config >&3hash service 2>&- && service ssh reload 2>&- || /etc/init.d/ssh* reload

First time I see the 'hash' builtin & wtf does >&- even mean?

— Reply to this email directly or view it on GitHubhttps://github.com/jclouds/jclouds/issues/748#issuecomment-8368915.

[abayer]

What confuses me is that I've never seen this happen with my own slaves with root as the Jenkins user...and we're definitely not passing any such flag to AdminAccess.

[karel1980]

Thanks a lot. Awk I can cope with ;-)

On Fri, Sep 7, 2012 at 5:40 PM, Adrian Cole notifications@github.comwrote:

Haha, so there is a flag to AdminAccess to turn off lock ssh, so we can look at that.

This is the most interesting awk I've used :) hash checks that a command resolves (similar to which) >&- turns off a file descriptor similar to

/dev/null On Sep 7, 2012 8:36 AM, "Karel Vervaeke" notifications@github.com wrote:

The snippet is produced by

scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/ssh/SshStatements.java

There is a 'lockSshd' static method there which is responsible. This is in turn called from

scriptbuilder/src/main/java/org/jclouds/scriptbuilder/statements/login/AdminAccess.java

AdminAccess has javadocs explaining its purpose. I don't know why it would be called in your case.

Other than that, thank you for making my day by bringing up this lovely construct:

exec 3<> /etc/ssh/sshd_config && awk -v TEXT="PasswordAuthentication noPermitRootLogin no" 'BEGIN {print TEXT}{print}' /etc/ssh/sshd_config &3hash service 2>&- && service ssh reload 2>&- || /etc/init.d/ssh* reload

First time I see the 'hash' builtin & wtf does >&- even mean?

— Reply to this email directly or view it on GitHub< https://github.com/jclouds/jclouds/issues/748#issuecomment-8368915>.

— Reply to this email directly or view it on GitHubhttps://github.com/jclouds/jclouds/issues/748#issuecomment-8369058.

[SteveJones]

So, new issue. I tried adding this to my init script:

mv /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
grep -v 'PermitRootLogin no' /etc/ssh/sshd_config.bak > /etc/ssh/sshd_config
reload ssh

But the log output doesn't show this in the init script when it attempts to launch.