Breaks scp - Githubissues

flaxton commented 12 years ago

I had been beating my head against the wall, trying to find out why scp was no longer working. Then I read that "echo" in your .bash initailization scripts is known to break it.

A little background. I use a Mac running OS X 10.7 Lion, and administer a Linux server running RHES 5, and have for 10 years. I keep my ssh keys on my Mac and have the public key copied to authorized_keys on the Linux server. ssh continues to work normally. I seldom scp, but when I tried today, everything seemed to work, except it didn't actually transfer a file (local to remote), it just hangs. Sending from remote to local spits out a line giving the file permissions, size and file name, but it doesn't actually transfer the file.

I tried lots of things - turned off the firewall, even went so far as to install the Mac Ports version of openssh, but no go.

Then I remembered that I had added .bash_ps1 to the end of my .bash_profile. I commented it out on both ends and voila scp works again.

I am an intermediate bash user. I'm not sure what could be done to see if you're using scp, and disable the echo. I suppose checking parameter $0 might work.

Anyway, I have .bash_ps1 disabled for now.

Thanks!

Fred

emilis commented 12 years ago

Hi,

I am very sorry that .bash_ps1 frustrated you so much.

It is frustrating for me that I cannot repeat your error. I also use scp with ssh keys frequently and it works for me both ways. It may be because I use GNU/Linux (mostly Ubuntu and Debian) on both ends. The closest environment I can come up with is check one CentOS server I use.

Just to be sure: do you use Bash on Mac? What version ($ bash --version)?

flaxton commented 12 years ago

Emilis,

I'm using bash 3.2.48 on Mac OS X 10.7 Lion, and bash 3.2.25 on CentOS 5.

I didn't think to try disabling it only on Mac OS X or Linux. I had a need to transfer a file, so I disabled it on both sides and confirmed it works now.

I did use the Mac version of .bash_ps1 from http://lifehacker.com/5840450/add-a-handy-separator-between-commands-in-your-terminal-on-mac-os-x-and-linux?utm_medium=referral&utm_source=pulsenews

Okay, I see that Rob Johnson changed it for Mac.

Let me try it both ways, and if only the Mac side causes a problem, I will contact Rob.

Thanks!

Fred

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 4:21 AM, Emilis Dambauskas wrote:

Hi,

I am very sorry that .bash_ps1 frustrated you so much.

It is frustrating for me that I cannot repeat your error. I also use scp with ssh keys frequently and it works for me both ways. It may be because I use GNU/Linux (mostly Ubuntu and Debian) on both ends. The closest environment I can come up with is check one CentOS server I use.

Just to be sure: do you use Bash on Mac? What version ($ bash --version)?

Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3029925

flaxton commented 12 years ago

Okay, I tried enabling .bash_ps1 on CentOS, and confirmed that scp is broken both ways. This is true whether I have .bash_ps1 enabled or disabled on Mac OS X.

Also note that this is running the scp command from Mac OS X. I'm doing this because the Mac is at my home office and I am too lazy to set up a public IP and modify my router ;-)

Here's an example - command on Mac OS X:

fredlaxton@Fred-Laxtons-MacBook:~/Downloads$ scp root@xxxxxxxx.xxx:tmpwork/weekly_offsite_backups.txt . C0644 0 weekly_offsite_backups.txt

The message shows the file permission, file size and filename

Just to anticipate a possible question, the file size doesn't matter, I was doing it with a 1 MB file yesterday with the same results, and I obfuscated the server name.

When I go in the reverse order (sending a file from Mac OS X to CentOS Linux) I get no message, and sep hangs.

So a problem appears to be in .bash_ps1 on Linux, although I am not saying that me being on a Mac has nothing to do with it ;-) so maybe as you say it only affects Mac & Linux transfers? But it doesn't matter whether I use .bash_ps1 on Mac or not in my testing.

Fred

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 10:16 AM, Fred Laxton wrote:

Emilis,

I'm using bash 3.2.48 on Mac OS X 10.7 Lion, and bash 3.2.25 on CentOS 5.

I didn't think to try disabling it only on Mac OS X or Linux. I had a need to transfer a file, so I disabled it on both sides and confirmed it works now.

I did use the Mac version of .bash_ps1 from http://lifehacker.com/5840450/add-a-handy-separator-between-commands-in-your-terminal-on-mac-os-x-and-linux?utm_medium=referral&utm_source=pulsenews

Okay, I see that Rob Johnson changed it for Mac.

Let me try it both ways, and if only the Mac side causes a problem, I will contact Rob.

Thanks!

Fred

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 4:21 AM, Emilis Dambauskas wrote:

Hi,

I am very sorry that .bash_ps1 frustrated you so much.

It is frustrating for me that I cannot repeat your error. I also use scp with ssh keys frequently and it works for me both ways. It may be because I use GNU/Linux (mostly Ubuntu and Debian) on both ends. The closest environment I can come up with is check one CentOS server I use.

Just to be sure: do you use Bash on Mac? What version ($ bash --version)?

Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3029925

emilis commented 12 years ago

Thank you for the time spent trying to debug this!

I was able to reproduce the error (see output below).

The error disappeared and the file was scp'd when I removed .bash_ps1 on my CentOS server, but left it on my local laptop.

I am going to fix this (although I cannot give a promise on how much it will take).

I guess that this error may be related to bash version or scripts used on RH/CentOS. Well... I will have to learn some more bash... ewww ;-)

Some debugging:

remote-centos$ bash --version
GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.
remote-centos$ date > test_file
remote-centos$ ls -l test_file
-rw-rw-r-- 1 emilis emilis 29 Dec  6 20:21 test_file

local-ubuntu$ bash --version
GNU bash, version 4.2.10(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
local-ubuntu$ scp remote-centos:test_file .
C0664 29 test_file
local-ubuntu$ ls -l test_file
ls: cannot access test_file: No such file or directory

$ scp -v remote-centos:test_file .
Executing: program /usr/bin/ssh host remote-centos, user (unspecified), command scp -v -f -- test_file
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /home/emilis/.ssh/config
debug1: Applying options for remote-centos
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 123.234.123.234 [123.234.123.234] port 22.
debug1: Connection established.
debug1: identity file /home/emilis/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/emilis/.ssh/id_rsa-cert type -1
debug1: identity file /home/emilis/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/emilis/.ssh/id_dsa-cert type -1
debug1: identity file /home/emilis/.ssh/id_ecdsa type -1
debug1: identity file /home/emilis/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 8a:d3:b4:fb:d3:dc:bd:a4:85:de:0b:22:ee:c6:73:23
debug1: Host '123.234.123.234' is known and matches the RSA host key.
debug1: Found key in /home/emilis/.ssh/known_hosts:53
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: An invalid name was supplied
Cannot determine realm for numeric host address

debug1: An invalid name was supplied
Cannot determine realm for numeric host address

debug1: An invalid name was supplied

debug1: An invalid name was supplied

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/emilis/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Authentication succeeded (publickey).
Authenticated to 123.234.123.234 ([123.234.123.234]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_TIME = en_DK.utf8
debug1: Sending command: scp -v -f -- test_file
Sending file modes: C0664 29 test_file
Sink: C0664 29 test_file
C0664 29 test_file
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2768, received 2328 bytes, in 0.1 seconds
Bytes per second: sent 27381.5, received 23028.9
debug1: Exit status 1

flaxton commented 12 years ago

No problem. I was just flummoxed there for a while, thinking that Mac OS X 10.7 Lion had broken scp ;-)

So while it would be nice to have it on my server, just having it on the Mac is nice.

Again, what I read was the an "echo" can disrupt scp, not sure why, so that would be my guess. Here's one place where I say this mentioned:

http://apple.stackexchange.com/questions/13802/why-does-echo-in-etc-bashrc-cause-scp-to-hang

Thanks!

Fred

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 1:39 PM, Emilis Dambauskas wrote:

Thank you for the time spent trying to debug this!

I was able to reproduce the error (see output below).

The error disappeared and the file was scp'd when I removed .bash_ps1 on my CentOS server, but left it on my local laptop.

I am going to fix this (although I cannot give a promise on how much it will take).

I guess that this error may be related to bash version or scripts used on RH/CentOS. Well... I will have to learn some more bash... ewww ;-)

Some debugging:

remote-centos$ bash --version
GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2005 Free Software Foundation, Inc.
remote-centos$ date > test_file
remote-centos$ ls -l test_file
-rw-rw-r-- 1 emilis emilis 29 Dec  6 20:21 test_file

local-ubuntu$ bash --version
GNU bash, version 4.2.10(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
local-ubuntu$ scp remote-centos:test_file .
C0664 29 test_file
local-ubuntu$ ls -l test_file
ls: cannot access test_file: No such file or directory

$ scp -v remote-centos:test_file .
Executing: program /usr/bin/ssh host remote-centos, user (unspecified), command scp -v -f -- test_file
OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /home/emilis/.ssh/config
debug1: Applying options for remote-centos
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 123.234.123.234 [123.234.123.234] port 22.
debug1: Connection established.
debug1: identity file /home/emilis/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/emilis/.ssh/id_rsa-cert type -1
debug1: identity file /home/emilis/.ssh/id_dsa type 2
debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
debug1: identity file /home/emilis/.ssh/id_dsa-cert type -1
debug1: identity file /home/emilis/.ssh/id_ecdsa type -1
debug1: identity file /home/emilis/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 8a:d3:b4:fb:d3:dc:bd:a4:85:de:0b:22:ee:c6:73:23
debug1: Host '123.234.123.234' is known and matches the RSA host key.
debug1: Found key in /home/emilis/.ssh/known_hosts:53
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: An invalid name was supplied
Cannot determine realm for numeric host address

debug1: An invalid name was supplied
Cannot determine realm for numeric host address

debug1: An invalid name was supplied

debug1: An invalid name was supplied

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/emilis/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Authentication succeeded (publickey).
Authenticated to 123.234.123.234 ([123.234.123.234]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_TIME = en_DK.utf8
debug1: Sending command: scp -v -f -- test_file
Sending file modes: C0664 29 test_file
Sink: C0664 29 test_file
C0664 29 test_file
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2768, received 2328 bytes, in 0.1 seconds
Bytes per second: sent 27381.5, received 23028.9
debug1: Exit status 1

Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3036593

emilis commented 12 years ago

Thank you for the link! Very helpful.

emilis commented 12 years ago

@flaxton: Can you test the new code and see if it fixes your problems?

I changed a line of code in .bashrc

from this:

if [ -f "$HOME/.bash_ps1" ]; then
    . "$HOME/.bash_ps1"
fi

to this:

if [ -n "$PS1" -a -f "$HOME/.bash_ps1" ]; then
    . "$HOME/.bash_ps1"
fi

If it works, I will send the fix to Whitson Gordon @ Lifehacker to ask to update this article: http://lifehacker.com/5840450/add-a-handy-separator-between-commands-in-your-terminal-on-mac-os-x-and-linux/

flaxton commented 12 years ago

The problem still happens with the change. When I remove it from .bashrc then scp again works normally.

Fred

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 3:07 PM, Emilis Dambauskas wrote:

@flaxton: Can you test the new code and see if it fixes your problems?

I changed a line of code in .bashrc

from this:
if [ -f "$HOME/.bash_ps1" ]; then
   . "$HOME/.bash_ps1"
fi
to this:
if [ -n "$PS1" -a -f "$HOME/.bash_ps1" ]; then
   . "$HOME/.bash_ps1"
fi
If it works, I will send the fix to Whitson Gordon @ Lifehacker to ask to update this article: http://lifehacker.com/5840450/add-a-handy-separator-between-commands-in-your-terminal-on-mac-os-x-and-linux/

Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3037883

emilis commented 12 years ago

Now I am stuck again, because it worked for me :-/

Did you change .bashrc on your server or your laptop?

There is one more trick we can do, but you'll have to test it yourself. Add this line to top of .bash_ps1:

[ -z "$PS1" ] && return

flaxton commented 12 years ago

I changed it on the server. Should I do it on the laptop?

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 4:13 PM, Emilis Dambauskas wrote:

Now I am stuck again, because it worked for me :-/

Did you change .bashrc on your server or your laptop?

There is one more trick we can do, but you'll have to test it yourself. Add this line to top of .bash_ps1:
[ -z "$PS1" ] && return
Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3038840

flaxton commented 12 years ago

I made both changes on both sides, to .bashrc or .bash_profile and .bash_ps1, then re-sourced .bashrc or .bash_profile to pick them up.

Still see the same problem.

Fred Laxton InfoTechDesign.net

"Your Website, Done Right"

email: fred@infotechdesign.net web: http://infotechdesign.net and http://about.me/fredonline phone: 864.724.9112

On Dec 6, 2011, at 4:13 PM, Emilis Dambauskas wrote:

Now I am stuck again, because it worked for me :-/

Did you change .bashrc on your server or your laptop?

There is one more trick we can do, but you'll have to test it yourself. Add this line to top of .bash_ps1:
[ -z "$PS1" ] && return
Reply to this email directly or view it on GitHub: https://github.com/emilis/emilis-config/issues/2#issuecomment-3038840

emilis commented 12 years ago

Maybe the test -n "$PS1" fails?

I mean: maybe your PS1 is not empty for non-interactive shells, because it has has been modified by your personal or server-wide bashrc?

You can test this by running this command:

laptop$ ssh server 'echo "$PS1"'

A correct PS1 value should print an empty line.

tdussmann commented 9 years ago

I see this issue thread is rather old, however I have just faced the same problem and solved it. Why it worked for years with my old ubuntu setup, I don't know, however now it broke scp / sftp access for me too.

So yeah, basically all additional not-expected output when scp/sftp gets it's shell on the remote system breaks the connection... To be able to keep using the bash_ps1 script in my .bashrc while also being able to sftp to the same machine, I changed my .bashrc entry from:

# Tweaked prompt.
if [ -n "$PS1" -a -f "$HOME/.bash_ps1" ]; then
    . "$HOME/.bash_ps1"
fi

to:

if [ -f "$HOME/.bash_ps1" ] && [[ $- == *i* ]]; then
   . "$HOME/.bash_ps1"
fi

which basically uses $- to check if the session is interactive or not and only sources .bash_ps1 if it is. Source: http://unix.stackexchange.com/questions/26676/how-to-check-if-a-shell-is-login-interactive-batch

@emilis maybe you could update your blogpost (http://emilis.github.io/2011/09/12/customized-bash-prompt.html) so people like me who stumple upon it without looking for any issues in your github repo will still benefit from it. I mean it doesn't hurt, does it?

P.S: Thanks for this awesome little helper!

emilis commented 9 years ago

@tdussmann Thank you for the illuminating comment!

I will patch the code and the blog post later today.

emilis / emilis-config

Breaks scp #2