search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.78k stars 494 forks source link

SCP with Dropbear not working #3296

Closed Gill-Bates closed 4 years ago

Gill-Bates commented 4 years ago

ADMIN EDIT

Solution

apt install openssh-client

The OpenSSH client package includes the scp binary invoked by Dropbear when SCP clients access. The same way, SFTP can be enabled:

apt install openssh-sftp-server

This installs the /usr/lib/sftp-server invoked by Dropbear on SFTP client access. Only take care to not install the openssh-server package, which enables the conflicting OpenSSH server service (sshd), which blocks port 22 by default, used by Dropbear already for initial SSH access.

Dropbear is the primary SSH-Server on the Dietpi-Ecosystem. Transferring files with WinSCP failed with following error code:

"Cannot execute SCP to start transfer. Please make sure that SCP is installed on the server and path to it is included in PATH. You may also try SFTP instead of SCP. Command failed with return code 127.".

Switching to OpenSSH will solve this issue. But it would be great, when I get rid of this annoying error message with Dropbear.

A short edit of the default.cfg will solve this issue:

cp /etc/profile /var/tmp echo "export PATH=$PATH:/var/tmp/bin" >> /var/tmp/profile mount -o bind /var/tmp/profile /etc/profile mkdir /var/tmp/bin ln -s /var/tmp/dropbear /var/tmp/bin/scp

MichaIng commented 4 years ago

@Gill-Bates Yes, Dropbear does not support SFTP natively.

Switching to OpenSSH will solve this issue. But it would be great, when I get rid of this annoying error message with Dropbear.

This is a very clear and informational message, isn't it? Why do you want to hide it?

Since /var/tmp/dropbear does not exist by default, I am not sure what your solution does? Also it adds a permanent (system-wide) bind mount. If you really want scp command to do something, why not create/link it from the local admin binary path, which is intended for this? ln -s /var/tmp/dropbear /usr/local/bin/scp

Gill-Bates commented 4 years ago

@MichaIng Connect & Login works fine even with Dropbear and WinSCP. Try to edit & save a file to get the error message.

Joulinar commented 4 years ago

@Gill-Bates the challenge is, that you don't have any SCP executable on your system if Dropbear is installed. 

root@DietPiVM1:~# which scp
root@DietPiVM1:~#

therefore you would need to switch to OpenSSH if you like to use SCP

root@DietPiVM1:~# which scp
/usr/bin/scp
root@DietPiVM1:~#

It should not take more than 2-3 minutes to switch the SSH Server using dietpi-software. Or are there any blockers switching to OpenSSH??

Gill-Bates commented 4 years ago

@MichaIng No! Its not a blocker to switch. But Dropbear is more lightweight. So I was thinking to modify Dropbear to get it running with WinSCP

MichaIng commented 4 years ago

I am confused now. scp is the SCP client, not the server, right? Yeah, it also comes with the openssh-client package: https://packages.debian.org/buster/amd64/openssh-client/filelist

SCP for what I found is just a non-interactive SFTP client implementation, so instead of having an interactive SFTP session which allows shell-like file manipulation besides upload of course, SCP does a single upload and exists immediately. But it requires an SFTP server regardless.

For SFTP/SCP, the SSH server needs to have the SFTP protocol implemented/invoked, and this should be not the case with Dropbear, I am not totally mistaken. But one can simply install the OpenSSH SFTP server as standalone module: https://packages.debian.org/buster/openssh-sftp-server I am not sure if there are other modifications required to Dropbear to invoke the sftp-server binary.

Joulinar commented 4 years ago

@MichaIng I checked which package is needed and openssh-client is shipping SCP executable. No need for the server package.

@Gill-Bates simple installation of openssh-client package should fix your issue already and you can stay with Dropbear.

apt-get install openssh-client

afterwards you should be able to use SCP protocol with your WinSCP client

MichaIng commented 4 years ago

@Joulinar

I checked which package is needed and openssh-client is shipping SCP executable. No need for the server package.

Yes, but this is the client, not the server, hence it will not allow you to upload something via WinSCP to your DietPi. This is why I tried to clarify with above, as the scp binary is not required and has nothing to do with WinSCP being able to connect or not. What you need on the server is the sftp-server binary and it needs to be invoked by the SSH server when a SFTP/SCP client connects.

Joulinar commented 4 years ago

@MichaIng nope. I checked it again on an empty VM. As soon as the openssh-client package is installed, the SCP executable is available. No need for any further configuration. Once done I opened my WinSCP client on my Windows desktop and connect to my VM using SCP protocol. Once connected I can download and upload files 😄

MichaIng commented 4 years ago

@Joulinar Very strange. Hmm, perhaps I misunderstood something and sftp-server is only required for interactive SFTP session. But still strange that scp binary makes a difference, since really, it is just a client... https://manpages.debian.org/scp

EDIT: Okay verified: SCP works without SFTP server and requires scp binary.

I use PuTTY SCP client (https://the.earth.li/~sgtatham/putty/latest/w64/pscp.exe) and it always failed on my server, but I recognised the -scp cmd option now which forces SCP protocol and finally it works with scp binary in place, while by default it uses SFTP protocol which requires sftp-server then (which must be placed at /usr/lib/sftp-server). Also good to know all of this works indeed without any further setup. And btw I simply copied the binary in place only, hence didn't install all the package overhead.

MichaIng commented 4 years ago

Okay, I mark this issue as closed. Basically:

apt install openssh-sftp-server openssh-client

And SCP + SFTP via Dropbear works OOTB :+1:.

ksz16 commented 2 years ago

I know it's old issue, but I have simple solution (maybe it will be helpful for someone). Just put the following code to Automation_Custom_Script.sh:

wget -O /tmp/openssh-client.deb http://mirror.netcologne.de/raspbian/raspbian/pool/main/o/openssh/openssh-client_8.4p1-5+b1_armhf.deb
dpkg-deb -xv /tmp/openssh-client.deb /tmp/openssh-client
sudo cp /tmp/openssh-client/usr/bin/scp /usr/bin/scp
sudo chmod 755 /usr/bin/scp
sudo chown root:root /usr/bin/scp
rm -rf /tmp/openssh-client /tmp/openssh-client.deb

This allows us to copy files using scp or pscp right after installation. To be able to do that we only need a single binary file in /usr/bin. So there is no need to install full openssh-client package (unless one needs all its features - personally, dbclient is enough for me).

MichaIng commented 2 years ago

And you think this is simpler than installing the OpenSSH client package? 😄 Yes only the binary is required, but installing it alone that way means that it is never updated with a regular apt upgrade, so for security reasons I wouldn't recommend it that way.

Another note when someone lands here: The above link is only valid on Raspbian Bullseye based images (DietPi ARMv6 RPi image) and it is the currently latest package, but may not be available anymore when a new package was shipped. The newest one can be found here, for Bullseye the second-latest one (the latest one is for Raspbian Bookworm): http://raspbian.raspberrypi.org/raspbian/pool/main/o/openssh/?C=M;O=D

Joulinar commented 2 years ago

yap going with dietpi-software install 0 is fare more simpler

ksz16 commented 2 years ago

Yes, I realize that this solution is not perfect and it is not automatic (occasionally you need to update the download link). But it works for my usage scenario: I need the immediate ability to copy files using spc right after automatic installation before even logging into the server. And those few lines of code get the job done.

ksz16 commented 2 years ago

You are probably right, and maybe it's better to install the whole openssh-client package (after all, this can also be done via Automation_Custom_Script.sh) By the way, lack of scp in Dropbear is supposedly a Debian issue. I'm also using OpenWrt where Dropbear is also installed and this problem does not occur there.

Joulinar commented 2 years ago

you could automatically install OpenSSH Cient automatically during first inital setup. This is how I do it for all my systems.

##### Non-interactive Firstrun Setup #####
# On first login, run update, initial setup and software installs without any user input
# - Setting this to "1" is required for below settings to take effect
# - It is HIGHLY recommended to also set CONFIG_BOOT_WAIT_FOR_NETWORK=2, to force infinite wait for network connection during boot and prevent connection timeout errors.
AUTO_SETUP_AUTOMATED=1

# Software to automatically install
# - List of available software IDs: https://github.com/MichaIng/DietPi/wiki/DietPi-Software-list
# - Add as many entries as you wish, one each line.
# - DietPi will automatically install all pre-reqs (e.g. ALSA/X11 for desktops etc)
# - E.g. the following (without the leading "#") will install the LXDE desktop automatically on first boot:
AUTO_SETUP_INSTALL_SOFTWARE_ID=0
ksz16 commented 2 years ago

@Joulinar Thanks a lot. Somehow I hadn't noticed this package on list before. I was probably confused by the fact that it has the number 0 :smile: This actually makes things easier.

MichaIng commented 2 years ago

A manual install works just the same way on DietPi an non-DietPi systems, btw: https://github.com/MichaIng/DietPi/issues/3296#issuecomment-598818924

splatch commented 2 years ago

Maybe a small and late advice, you can always enable "legacy" mode for ssh client (-O for openssh command line client) to go. No need to rush with package updates.

MichaIng commented 2 years ago

-O is for controlling multiplexing connections and requires an argument :think:: https://manpages.debian.org/bullseye/openssh-client/ssh.1.en.html#O

For something sensitive like SSH clients and server's I'd always keep them updated. Stable Debian package upgrades are always either bug fixes or security patches, carefully tested to not cause breaking changes, so for SSH I'd call it irresponsible to knowingly delay the update 😉. Also I'm not sure how package upgrades are related to this topic?

splatch commented 2 years ago

You're right that -O has this meaning for ssh client. According to man pages for scp in openssh 9+ it plays other role:

-O Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a ‘~’ prefix for older SFTP servers.

So you can expect more people coming here as for me reason for finding your issue was moving from openssh 8.9 to 9. :-)

MichaIng commented 2 years ago

Ah yes, scp is a different command indeed. If I'm not mistaken it uses SFTP by default only with Debian Bookworm and above, hence not relevant for current stable Bullseye: https://manpages.debian.org/testing/manpages-de/scp.1.de.html#O Also the SCP protocol is deprecated and not developed in favour of SFTP since a long time, so if there is really a client not supporting it, if there is any chance, the client should be replaced with a modern one with SFTP support.

The issue here however was not a client without SFTP support, but that the Dropbear SSH server does not come with SFTP and SCP support, and hence requires openssh-client (for scp) or openssh-sftp-server (for SFTP) or another SFTP server like Green End SFTP Server to be installed additionally 🙂. I use the latter since a long time btw, it is lighter and works very well with WinSCP, PuTTY, curl, sftp and other SFTP clients. In some cases, where the SFTP server path cannot be defined, one might want to create a symlink to the default OpenSSH SFTP server path:

apt install gesftpserver
ln -s /usr/libexec/gesftpserver /usr/lib/sftp-server
isarrider commented 1 year ago

how about an option in DietPi-Software, where Green End comes in:

None Dropbear (SSH only) Dropbear + Green End (SSH + SFTP - recommended) OpenSSH Server (Feature rich ...)

...?

MichaIng commented 1 year ago

Makes sense. I use the Dropbear + Green End combination successfully for a while now on a few systems. I mean also the OpenSSH SFTP server works well fine, not sure why one could be preferred over the other 🤔.

isarrider commented 1 year ago

I would say smaller footprint: https://packages.debian.org/bullseye/arm64/gesftpserver/filelist https://packages.debian.org/bullseye/arm64/openssh-client/filelist and a newer sftp version: https://www.greenend.org.uk/rjk/sftpserver/

MichaIng commented 1 year ago

Wrong comparison 😉: https://packages.debian.org/bullseye/arm64/openssh-sftp-server/filelist Ah, but the SFTP server pull the SSH client as dependency, indeed.

Newer SFTP protocol version is an argument, but since OpenSSH uses version 3, I guess most clients do not support newer versions either and most admins don't know about the new syntax, if they use the SFTP console manually anyway.

One downside of Green End is that it doesn't install to a default path, so clients do not find it unless you pass /usr/libexec/gesftpserver explicitly (possible in WinSCP). Can be solved with a symlink to /usr/lib/sftp-server or /usr/lib/openssh/sftp-server (ah I mentioned this above already).

isarrider commented 1 year ago

and somehow Green End is much slower for me for large filetransfers: OpenSSH (with SCP) ~ 850 MBit/s OpenSSH (with SFTP) ~ 650 MBit/s Green End (obv SFTP) ~ 200 MBit/s and connection drops... but WinSCP states it uses protocol version 6...

MichaIng commented 1 year ago

Did you compare with OpenSSH SSH server + Green End SFTP server, respectively Dropbear + OpenSSH SFTP server? Probably it's Dropbear which slows down transfers, not Green End.

AFAIK, the protocol version does not affect speed but only refines/defines SFTP syntax/commands. But I might be wrong. Interesting that SCP is faster. Quite nasty since SCP is deprecated and will probably be removed in the future, in favour of SFTP only.

isarrider commented 1 year ago

Speed: https://www.jscape.com/blog/scp-vs-sftp if we believe this source, I imagine it would be similar to TCP with SACK...

regarding you hint for comparison - good point... I think I can do a test tmr...

isarrider commented 1 year ago

from what I can see so far it seems Dropbear doesn't use AES Acceleration (https://github.com/mkj/dropbear/issues/166) or the OpenSSL implementation is much faster... Ill digg in more later...

isarrider commented 1 year ago 
root@Rock5-4GB:~# openssl speed -elapsed aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128 cbc for 3s on 16 size blocks: 33583349 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 64 size blocks: 9086473 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 256 size blocks: 2298588 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 1024 size blocks: 580377 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 8192 size blocks: 72749 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 16384 size blocks: 36389 aes-128 cbc's in 3.00s
OpenSSL 1.1.1n  15 Mar 2022
built on: Fri Jun 24 20:22:19 2022 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-rqLv6p/openssl-1.1.1n=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128 cbc     179111.19k   193844.76k   196146.18k   198102.02k   198653.27k   198732.46k

root@Rock5-4GB:~# openssl speed -elapsed -evp aes-128-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-128-cbc for 3s on 16 size blocks: 120753717 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 62404556 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 19951169 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 5315248 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 680927 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 16384 size blocks: 341512 aes-128-cbc's in 3.00s
OpenSSL 1.1.1n  15 Mar 2022
built on: Fri Jun 24 20:22:19 2022 UTC
options:bn(64,64) rc4(char) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-rqLv6p/openssl-1.1.1n=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DVPAES_ASM -DECP_NISTZ256_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-cbc     644019.82k  1331297.19k  1702499.75k  1814271.32k  1859384.66k  1865110.87k

ok, e.g. taking type 16384 is almost 10 times faster accelerated...

MichaIng commented 1 year ago

Can this be compared since Dropbear uses libtomcrypt?

isarrider commented 1 year ago

yes, as libtomcrypt does not (yet) have AES acceleration... clear that it is much slower then (it maxes out 1 A76 core fully)

MichaIng commented 1 year ago

Okay makes sense. What would be still interesting if OpenSSH SFTP server or Green End performs better (or equal) with Dropbear and/or OpenSSH SSH server. I'll find time for testing in a few days when back home from vacation.

isarrider commented 1 year ago

I hope I find time before, but the first week is more busy than expected...

isarrider commented 1 year ago

and somehow Green End is much slower for me for large filetransfers: OpenSSH (with SCP) ~ 850 MBit/s OpenSSH (with SFTP) ~ 650 MBit/s Green End (obv SFTP) ~ 200 MBit/s and connection drops... but WinSCP states it uses protocol version 6...

ok, new finding - it gets weird... with WinSCP, the upload speed to my Rock5 can be increased, if I use the background transfer function, the new table is:

OpenSSH (with SCP) ~ 850 MBit/s OpenSSH (with SFTP) ~ 890 MBit/s to, 720 MBit/s from Rock5 using the background transfer function of WinSCP OpenSSH (with SFTP) ~ 650 MBit/s using the normal transfer functon of WinSCP Green End (obv SFTP) ~ 200 MBit/s and connection drops...

(just to be clear, the 2GB file goes from RAMDisk to RAMDisk...)

MichaIng commented 1 year ago

Bad test with NanoPi R6S Wi-Fi 5 dongle, other system GiB Ethernet attached, transferring a 1 GiB file to and from RAM disk (ImDisk on Windows): All combinations (Dropbear+OpenSSH SFTP, Dropbear+GeSFTP, OpenSSH+GeSFTP) maxxed at 12.5 MiB/s (=100 MBit/s). Not sure why so slow, as it's 434.0 MBit/s link, however the speed remains quite consistently. Only OpenSSH+OpenSSH SFTP was slightly faster with 12.8 MiB/s max.

Interestingly, in my case SCP (with OpenSSH server) was significantly slower at max 8.65 MiB/s. With Dropbear + SCP however it maxxed at 12.8 MiB/s. All results can be replicated consistently, I rotated back and forth quite a few times.

I also tried with OpenSSH's "internal-sftp" module, which again maxxes at 12.5 MiB/s, i.e. (repeatedly) a little slower than the dedicated /usr/lib/sftp-server binary.

With "Transfer on background by default" indeed all combinations' transfer speeds are higher at up to 13.5 MiB/s, which is very strange. I wonder whether it is a dump visual error, like showing "MB" (powers of 1000) instead of "MiB" (powers of 1024) or such, as to me it doesn't make any sense why there would be a difference, or something is badly programmed 😄.

However, when I find time I'll do some cable-connected R6S from/to Odroid N2 tmpfs.

Your tests were all done on OpenSSH SSH server, or on Dropbear or mixed? I still do not see whether Green End is slower in your case or Dropbear. We have a theory why Dropbear may be slower, but in your results I do not see which SSH server was used.

What I faced with Dropbear is a dropping connection at every ~450 MiB. This happens with both: Green End as well as OpenSSH SFTP server (Dropbear SSH server!), so it is indeed Dropbear responsible for this. Is this what you mean with your dropping connection, indicating that you always combined Dropbear with Green End and OpenSSH only with its own SFTP server?

isarrider commented 1 year ago

Hi MichaIng,

ok, i did not mention this but very good that you pointed it out: Yes, I either had DropBear and GreenEnd together OR OpenSSH with included ftp...

I still think that the missing AES accel is responsible, looking at the CPU load during transfer... I had never a con drop with OpenSSH btw... (Ah and just to be sure, all my test are on cable with a switch and a FritzBox in between)