search

gridcf / gct

Grid Community Toolkit
Apache License 2.0
48 stars 30 forks source link

HPN-SSH not present in EPEL builds #108

Open asdorsey opened 5 years ago

asdorsey commented 5 years ago

Apologies in advance if you guys don't do the builds for EPEL. I have a question that I hope you can answer.

We recently deployed some data transfer nodes on CentOS 7.6 using the (at the time) latest available GCT packages in EPEL. It appears the gsi-openssh-server package that was installed (gsi-openssh-7.4p1-4.el7.x86_64) doesn't include the HPN patch - attempting to enable the feature in /etc/gsissh/sshd_config results in an error from gsisshd and the service refusing to start.

Oct 24 19:09:03 hdtn1 systemd: Starting Cluster Controlled gsisshd...
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config line 47: Deprecated option RSAAuthentication
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config line 56: Deprecated option RhostsRSAAuthentication
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config line 149: Unsupported option DisableUsageStats
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config: line 189: Bad configuration option: HPNDisabled
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config: line 196: Bad configuration option: HPNBufferSize
Oct 24 19:09:03 hdtn1 gsisshd: /etc/gsissh/sshd_config: terminating, 2 bad configuration options
Oct 24 19:09:03 hdtn1 systemd: gsisshd.service: main process exited, code=exited, status=255/n/a
Oct 24 19:09:03 hdtn1 systemd: Failed to start Cluster Controlled gsisshd.
Oct 24 19:09:03 hdtn1 systemd: Unit gsisshd.service entered failed state.
Oct 24 19:09:03 hdtn1 systemd: gsisshd.service failed.

I found #72 that references a discussion about dropping the HPN patch, but I can't find anything stating that this change was definitely made, and on what date that change was implemented.

Do you have any information on when HPN was removed from the EPEL package builds, and/or what was the last version that included HPN support?

rapier1 commented 4 years ago

Okay, I want to apologize for how late everything is on replying to this. I've been slammed with other projects that have litteraly soaked up all of my time. I'm starting to dedicated time back to HPN-SSH (the good news is that I have funding for this now).

@rapier1 https://github.com/rapier1 The HPN patch 14v15 for OpenSSH 7.6p1 doesn't work with OpenSSL 1.1.0, right?

I apologize if all of this has already been resolved. In answer to the direct question; no 7.6 does not work with OpenSSL 1.1. I have a really overly complex version of twisty ifdefs for 7.7 and 7.8 that does but that was an experiment that I'm not going to be supporting at this time.

On 6/26/20 1:18 PM, fscheiner wrote:

Geez, spoken to soon. :-/ I have a compile error for GSI-OpenSSH 7.6p1 w/HPN for SLES 15:

from the build log https://build.opensuse.org/package/live_build_log/home:frank_scheiner:gct/gsi-openssh/SLE_15/x86_64:

|[...] [ 80s] gcc -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -fpie -fstack-protector -pipe -Wall -Wpointer-arith -Wuninitialized -Wsign-compare -Wformat-security -Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -ftrapv -fno-builtin-memset -fstack-protector-strong -fPIE -I. -I.. -I. -I./.. -D_XOPEN_SOURCE=600 -D_BSD_SOURCE -D_DEFAULT_SOURCE -I/usr/include/editline -DLDAP_DEPRECATED -DOPENSSL_LOAD_CONF -I/usr/include/globus -DHAVE_CONFIG_H -c bsd-snprintf.c [ 80s] cipher-ctr-mt.c: In function 'ssh_aes_ctr': [ 80s] cipher-ctr-mt.c:442:18: error: dereferencing pointer to incomplete type 'EVP_CIPHER_CTX {aka struct evp_cipher_ctx_st}' [ 80s] ssh_ctr_inc(ctx->iv, AES_BLOCK_SIZE); [ 80s] ^~ [ 80s] cipher-ctr-mt.c: In function 'evp_aes_ctr_mt': [ 80s] cipher-ctr-mt.c:652:20: error: storage size of 'aes_ctr' isn't known [ 80s] static EVP_CIPHER aes_ctr; [ 80s] ^~~ [ 80s] cipher-ctr-mt.c:654:29: error: invalid application of 'sizeof' to incomplete type 'EVP_CIPHER {aka struct evp_cipher_st}' [ 80s] memset(&aes_ctr, 0, sizeof(EVP_CIPHER)); [ 80s] ^~~~~~ [ 80s] cipher-ctr-mt.c:652:20: warning: unused variable 'aes_ctr' [-Wunused-variable] [ 80s] static EVP_CIPHER aes_ctr; [ 80s] ^~~ [ 80s] cipher-ctr-mt.c:667:1: warning: control reaches end of non-void function [-Wreturn-type] [ 80s] } [ 80s] ^ [ 80s] make: [Makefile:171: cipher-ctr-mt.o] Error 1 [ 80s] make: Waiting for unfinished jobs.... [...] |

Maybe an |include| is missing, but could also be due to the build using OpenSSL 1.1.0. I see conditionals for different OpenSSL versions there in later versions of the HPN patch.

@rapier1 https://github.com/rapier1 The HPN patch 14v15 for OpenSSH 7.6p1 doesn't work with OpenSSL 1.1.0, right?

I used this RPM spec file https://build.opensuse.org/package/view_file/home:frank_scheiner:gct/gsi-openssh/gsi-openssh-SLE_15.spec?expand=1 and this adapted HPN patch https://build.opensuse.org/package/view_file/home:frank_scheiner:gct/gsi-openssh/openssh-7.6p1-hpn-14.15-modified.patch?expand=1 which is based on https://sourceforge.net/projects/hpnssh/files/Patches/HPN-SSH%2014v15%207.6p1/openssh-7_6_P1-hpn-14.15.diff/download https://sourceforge.net/projects/hpnssh/files/Patches/HPN-SSH%2014v15%207.6p1/openssh-7_6_P1-hpn-14.15.diff/download.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-650297555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66FXYWKVFKHWHG7DN7LRYTKAFANCNFSM4JEZMT3Q.

rapier1 commented 4 years ago

On 9/3/20 1:31 PM, adorsey-NOAA wrote:

@fscheiner https://github.com/fscheiner Thanks, if I can get some test nodes/VMs with CentOS 8 spun up I'll run some tests in between sites.

@rapier1 https://github.com/rapier1 Any ideas or progress on the segfault issues with the CentOS 7 packages?

I'll start looking into this now assuming that you are still facing this issues. My guess is that when I fixed issues with the aes-ctr cipher I didn't back port it all the way back to 7.4 or I somehow messed up the back port. That said, in most cases the default aes-ctr cipher is actually going to be faster that aes-ctr-mt. This is because when we developed the MT version CPUs didn't have AES instructions on die. With on die AES default version is faster. We hope to address that in the next 6 to 9 months (after we finish doing a threaded version of chacha20). You may want to test this with -oDisableMTAES=yes on the command line on the server side.

Just to confirm - this is 7.4 on Centos 7 using openssl 1.0?

Chris

rapier1 commented 4 years ago

You wouldn't happen to be using lustre on the file system with a lot of small files? Anyway, that the problem was limited to rsync makes me think the issue is within the interaction between rsync and ssh. I'll need to look into that. I'll try to recreate the problem here but I'm in the annoying place of trying to do this all in my home lab. With the CV19 issues it's more difficult to set things up on the PSC production infrastructure. I do have a 10g network here so that helps. Could you tell me a little more about the local setup?

Chris

On 9/29/20 1:24 PM, adorsey-NOAA wrote:

@rapier1 https://github.com/rapier1 another issue to report, though this may be related to the existing issues I've already mentioned.

I've been running down a performance problem on our DTNs; specifically, when users try to transfer files using rsync, performance is awful (~20MB/s). With scp, we can get 250MB/s from the same remote site. This is accompanied by 100% CPU usage of the gsisshd child process for that rsync process on the DTN.

I moved our DTNs to the gsi-openssh-server packages provided by XSEDE (https://software.xsede.org/production/gsi-openssh-server/7.5p1b-1/XSEDE-GSI-OpenSSH-install.html https://software.xsede.org/production/gsi-openssh-server/7.5p1b-1/XSEDE-GSI-OpenSSH-install.html) and that fixed the performance issue for us, so it appears to be related to the gsi-openssh-7.6p1 packages we had been previously running.

Sorry, I know I don't have any more information other than "it's slow". I still have the test DTNs, and I can use those to test any updated packages for CentOS 7 that you or the GCT team can provide.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-700862362, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66EGP3FCFMORADRRGHDSIIJ5NANCNFSM4JEZMT3Q.

asdorsey commented 4 years ago

Just to confirm - this is 7.4 on Centos 7 using openssl 1.0?

Correct on CentOS 7 and openssl 1.0 (1.0.2k to be exact), but the OpenSSH version is actually 7.6p1. These are the packages you built for me in November 2019. We were using those for a while, but we ran into the following issues (as a recap):

@fscheiner made an attempt to compile packages based off of the EPEL 7 gsissh source, but ran into a segfault issue. I ran into the same issue in September or October of 2019 attempting the same thing.

You wouldn't happen to be using lustre on the file system with a lot of small files?

Good guess! The transfers where we were having problems are for data sets coming from the NOAA production clusters to our R&D cluster; these data sets are usually a large number of relatively small files (up to a few MB each). We are using Lustre; specifically we're using LNet routers to isolate the DTNs from the cluster's IB network - all Lustre traffic to and from the DTNs goes over Ethernet and through a firewall to a pair of LNet routers.

Anyway, that the problem was limited to rsync makes me think the issue is within the interaction between rsync and ssh.

I agree; enabling compression in rsync improved performance and decreased load from the gsissh process on the server side, which seemed to point towards the gsissh server causing the bottleneck. That's why I switched to the XSEDE packages, which resolved the issue for us.

Could you tell me a little more about the local setup?

Our production and test DTNs are set up the same; they just mount different file systems.

Each DTN has a single 40Gb Ethernet interface which handles both internet and Lustre LNet traffic. Internet traffic comes in through N-Wave via two 40Gb WAN connections to the DTNs, through a Juniper SRX5800 firewall. Lustre LNet traffic goes back through the firewall to the LNet routers, which each have one 40Gb Ethernet connection and one HDR100 IB connection.

We're working with DDN on IB performance, but some recent tuning changes have improved performance enough there that I no longer believe filesystem access is our bottleneck (unless there's something peculiar about transferring lots of large files on Lustre - I'm still getting up to speed on Lustre, but I know that lots of small files have been a concern for performance in the past).

Sorry for the essay, I tried to recap everything I could from my perspective. If I've gotten something wrong, or if you need more information, please feel free to ask.

rapier1 commented 4 years ago

I've come to really dislike lustre lately. :)

(I'm trying to apply for a grant to develop a method to use automatically mounted and managed sparse file systems as an overlay to improve performance related to file stat. I have a prototype but it's still early stages)

I'll see if I can figure anything out. I'm installed some centos VMs right now to do some base level tests. I'll set up a 10G connected system after that.

On 10/5/20 12:17 PM, adorsey-NOAA wrote:

Just to confirm - this is 7.4 on Centos 7 using openssl 1.0?

Correct on CentOS 7 and openssl 1.0 (1.0.2k to be exact), but the OpenSSH version is actually 7.6p1. These are the packages you built for me in November 2019. We were using those for a while, but we ran into the following issues (as a recap):

@fscheiner https://github.com/fscheiner made an attempt https://github.com/gridcf/gct/issues/108#issuecomment-674199057 to compile packages based off of the EPEL 7 gsissh source, but ran into a segfault issue https://github.com/gridcf/gct/issues/108#issuecomment-677775673. I ran into the same issue in September or October of 2019 attempting the same thing.

You wouldn't happen to be using lustre on the file system with a lot
of small files?

Good guess! The transfers where we were having problems are for data sets coming from the NOAA production clusters to our R&D cluster; these data sets are usually a large number of relatively small files (up to a few MB each). We are using Lustre; specifically we're using LNet routers to isolate the DTNs from the cluster's IB network - all Lustre traffic to and from the DTNs goes over Ethernet and through a firewall to a pair of LNet routers.

Anyway, that the problem was limited to rsync makes me think the
issue is within the interaction between rsync and ssh.

I agree; enabling compression in rsync improved performance and decreased load from the gsissh process on the server side, which seemed to point towards the gsissh server causing the bottleneck. That's why I switched to the XSEDE packages, which resolved the issue for us.

Could you tell me a little more about the local setup?

Our production and test DTNs are set up the same; they just mount different file systems.

Each DTN has a single 40Gb Ethernet interface which handles both internet and Lustre LNet traffic. Internet traffic comes in through N-Wave via two 40Gb WAN connections to the DTNs, through a Juniper SRX5800 firewall. Lustre LNet traffic goes back through the firewall to the LNet routers, which each have one 40Gb Ethernet connection and one HDR100 IB connection.

We're working with DDN on IB performance, but some recent tuning changes have improved performance enough there that I no longer believe filesystem access is our bottleneck (unless there's something peculiar about transferring lots of large files on Lustre - I'm still getting up to speed on Lustre, but I know that lots of small files have been a concern for performance in the past).

Sorry for the essay, I tried to recap everything I could from my perspective. If I've gotten something wrong, or if you need more information, please feel free to ask.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-703736503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66FGBHF5QK4XQN5MR6LSJHWTBANCNFSM4JEZMT3Q.

rapier1 commented 4 years ago

By the way - are you using anything like parsyncfp as well or is this a straight rsync process?

On 10/5/20 12:17 PM, adorsey-NOAA wrote:

Just to confirm - this is 7.4 on Centos 7 using openssl 1.0?

Correct on CentOS 7 and openssl 1.0 (1.0.2k to be exact), but the OpenSSH version is actually 7.6p1. These are the packages you built for me in November 2019. We were using those for a while, but we ran into the following issues (as a recap):

@fscheiner https://github.com/fscheiner made an attempt https://github.com/gridcf/gct/issues/108#issuecomment-674199057 to compile packages based off of the EPEL 7 gsissh source, but ran into a segfault issue https://github.com/gridcf/gct/issues/108#issuecomment-677775673. I ran into the same issue in September or October of 2019 attempting the same thing.

You wouldn't happen to be using lustre on the file system with a lot
of small files?

Good guess! The transfers where we were having problems are for data sets coming from the NOAA production clusters to our R&D cluster; these data sets are usually a large number of relatively small files (up to a few MB each). We are using Lustre; specifically we're using LNet routers to isolate the DTNs from the cluster's IB network - all Lustre traffic to and from the DTNs goes over Ethernet and through a firewall to a pair of LNet routers.

Anyway, that the problem was limited to rsync makes me think the
issue is within the interaction between rsync and ssh.

I agree; enabling compression in rsync improved performance and decreased load from the gsissh process on the server side, which seemed to point towards the gsissh server causing the bottleneck. That's why I switched to the XSEDE packages, which resolved the issue for us.

Could you tell me a little more about the local setup?

Our production and test DTNs are set up the same; they just mount different file systems.

Each DTN has a single 40Gb Ethernet interface which handles both internet and Lustre LNet traffic. Internet traffic comes in through N-Wave via two 40Gb WAN connections to the DTNs, through a Juniper SRX5800 firewall. Lustre LNet traffic goes back through the firewall to the LNet routers, which each have one 40Gb Ethernet connection and one HDR100 IB connection.

We're working with DDN on IB performance, but some recent tuning changes have improved performance enough there that I no longer believe filesystem access is our bottleneck (unless there's something peculiar about transferring lots of large files on Lustre - I'm still getting up to speed on Lustre, but I know that lots of small files have been a concern for performance in the past).

Sorry for the essay, I tried to recap everything I could from my perspective. If I've gotten something wrong, or if you need more information, please feel free to ask.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-703736503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66FGBHF5QK4XQN5MR6LSJHWTBANCNFSM4JEZMT3Q.

asdorsey commented 4 years ago

I've come to really dislike lustre lately. :)

No comment. :-D You don't need to worry about the peculiarities of Lustre regarding the information I provided earlier - we see the same rsync performance issues transferring to /dev/null, which takes Lustre completely out of the equation.

By the way - are you using anything like parsyncfp as well or is this a straight rsync process?

Straight rsync via HPN-enabled ssh or gsissh client.

rapier1 commented 4 years ago

Hey, nothing about lustre here but I accidentally built HPN-SSH 8.3 on Centos 7 with openssh 1.0.2k. I meant to build 7.6 but 8.3 built without a problem on stock. Do you think you could give that a try on your system? It may not work but if it does we can remove issues associated with the older versions.

Chris

On 10/5/20 12:34 PM, adorsey-NOAA wrote:

I've come to really dislike lustre lately. :)

No comment. :-D You don't need to worry about the peculiarities of Lustre regarding the information I provided earlier - we see the same rsync performance issues transferring to /dev/null, which takes Lustre completely out of the equation.

By the way - are you using anything like parsyncfp as well or is this a
straight rsync process?

Straight rsync via HPN-enabled ssh or gsissh client.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-703746310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66BEOAGPCQM2AK4ANMLSJHYSPANCNFSM4JEZMT3Q.

asdorsey commented 4 years ago

I'll give it a try on the test DTNs today or tomorrow as I have time, hopefully by this evening.

rapier1 commented 4 years ago

This doesn't build in the in the GSSI stuff but it if it works then it makes life a lot easier in the long run.

On 10/7/20 10:32 AM, adorsey-NOAA wrote:

I'll give it a try on the test DTNs today or tomorrow as I have time, hopefully by this evening.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-704977630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66DUNUUH3RPCY7ATPKDSJR3ZDANCNFSM4JEZMT3Q.

ellert commented 4 years ago

EPEL 8 pull request merged. Update submitted to epel-testing: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-ab204de5da

asdorsey commented 4 years ago

@rapier1 Can I get a link to the OpenSSH 8.3 packages you mentioned?

rapier1 commented 4 years ago

I had just built from source. I'll try to get a package for 8.3p1 on Centos 7 built as soon as possible.

On 10/8/20 11:30 AM, adorsey-NOAA wrote:

@rapier1 https://github.com/rapier1 Can I get a link to the OpenSSH 8.3 packages you mentioned?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-705648360, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66BNL373HJEP5JUXJU3SJXLJPANCNFSM4JEZMT3Q.

rapier1 commented 4 years ago

I just got the 8.4p1 gsissh from ellert. I'm going to look into create an rpm for that for Centos7. I'll let you know if it's possible.

On 10/8/20 11:30 AM, adorsey-NOAA wrote:

@rapier1 https://github.com/rapier1 Can I get a link to the OpenSSH 8.3 packages you mentioned?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-705648360, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66BNL373HJEP5JUXJU3SJXLJPANCNFSM4JEZMT3Q.

rapier1 commented 4 years ago

I just built the rpms for HPN14v23 for OpenSSH 8.4 for CentOS7. They seem to work on my system but I don't have a way to test the gsi authentication at this time.

You can get a tar files with rpms and the source rpm here https://sourceforge.net/projects/hpnssh/files/GSI-Openssh-HPN-SSH/CentOS%207/

Let me know if that works. I'm really curious to know if it helps with some of the performance problems you've seen.

Next step is to get this working for CentOS8 if I can.

On 10/8/20 11:30 AM, adorsey-NOAA wrote:

@rapier1 https://github.com/rapier1 Can I get a link to the OpenSSH 8.3 packages you mentioned?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-705648360, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66BNL373HJEP5JUXJU3SJXLJPANCNFSM4JEZMT3Q.

rapier1 commented 4 years ago

As an aisde - to get this working for CentOS7 I had to disable some of the functionality that was in the original files provided by @ellert. This includes crypto-policies and the USB key sign in that depends on libfido2. I don't think either were supported under CentOS7 but just so you know.

asdorsey commented 4 years ago

@rapier1 sorry for delayed response, having major issues on the cluster FS so my focus is on that at the moment. Hope to get back to you with test results next week.

fscheiner commented 4 years ago

Sorry for not replying earlier, I was off-duty on vacation.

On 9/3/20 1:31 PM, adorsey-NOAA wrote: @rapier1 https://github.com/rapier1 Any ideas or progress on the segfault issues with the CentOS 7 packages?

I'll start looking into this now assuming that you are still facing this issues. My guess is that when I fixed issues with the aes-ctr cipher I didn't back port it all the way back to 7.4 or I somehow messed up the back port. That said, in most cases the default aes-ctr cipher is actually going to be faster that aes-ctr-mt. This is because when we developed the MT version CPUs didn't have AES instructions on die. With on die AES default version is faster. We hope to address that in the next 6 to 9 months (after we finish doing a threaded version of chacha20). You may want to test this with -oDisableMTAES=yes on the command line on the server side.

@rapier1 @adorsey-NOAA That switch is good to know. This should also added to the HPN-README. I'm going to do that in the HPN patch I use for the EPEL/Fedora GSI-OpenSSH w/HPN packages. And indeed it solves/works around the problems described in https://github.com/gridcf/gct/issues/108#issuecomment-674199057 and https://github.com/gridcf/gct/issues/108#issuecomment-677775673. Using it for the gsisshd makes transfers working for me when selecting the AES-CTR (which actually uses the AES-CTR-MT cipher on HPN enabled SSH clients) from the client. Tested successfully with:

...against:

...in both directions (put/get).

I only now recognized another issue with the GSI-OpenSSH 7.4p1 w/HPN based gsisshd: It doesn't detect the 8.0p1 and 7.9p1 clients as HPN aware:

## server debug output for 8.3p1 client
[...]
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_8.3p1c-GSI GSI-hpn14v22
SSH: Server;Ltype: Version;Remote: 172.16.1.35-39686;Protocol: 2.0;Client: OpenSSH_8.3p1c-GSI GSI-hpn14v22
debug1: match: OpenSSH_8.3p1c-GSI GSI-hpn14v22 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.4p1c-GSI GSI-hpn14v13-OpenSSH_7.4
[...]

## server debug output for 7.9p1 client
[...]
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_7.9
SSH: Server;Ltype: Version;Remote: 172.16.1.9-52414;Protocol: 2.0;Client: OpenSSH_7.9
debug1: match: OpenSSH_7.9 pat OpenSSH* compat 0x04000000
debug1: Remote is NON-HPN aware
debug1: Local version string SSH-2.0-OpenSSH_7.4p1c-GSI GSI-hpn14v13-OpenSSH_7.4
[...]

## server debug output for 8.0p1 client
[...]
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_8.0
SSH: Server;Ltype: Version;Remote: 172.16.1.34-35174;Protocol: 2.0;Client: OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Remote is NON-HPN aware
debug1: Local version string SSH-2.0-OpenSSH_7.4p1c-GSI GSI-hpn14v13-OpenSSH_7.4
[...]

I'm not yet sure why this happens, but when checking the built-in strings in the 7.9p1 and 8.0p1 based gsissh binaries I see the version string (e.g. OpenSSH_7.9) two times, e.g.:

$ strings /usr/bin/gsissh | grep 'OpenSSH_7.9'
OpenSSH_7.9p1b-GSI GSI-hpn14v18
OpenSSH_7.9

...whereas the 7.4p1 or 8.3p1 based gsissh binaries only contain one version string for the actual OpenSSH version with the additional GSI and HPN information, e.g.:

$ strings /usr/bin/gsissh | grep 'OpenSSH_8.3'
OpenSSH_8.3p1c-GSI GSI-hpn14v22

Checking the server debug output, it seems to receive the "wrong" version string when using the 7.9p1 and 8.0p1 based clients.

@rapier1 UPDATE: Ok, this also happens for newer gsisshds (e.g. 8.0p1 based). I see, this is due to https://github.com/rapier1/openssh-portable/issues/22. As that issue is already closed, what is the solution, as I didn't find a commit regarding that issue?

Just to confirm - this is 7.4 on Centos 7 using openssl 1.0? Chris

@rapier1 That's correct.

fscheiner commented 4 years ago

@ellert @adorsey-NOAA If the stock AES-CTR cipher is anyhow faster than the AES-CTR-MT cipher on "modern" CPUs with AES acceleration, I think GSI-OpenSSH w/HPN for EPEL/Fedora/SUSE should be shipped with DisableMTAES yes pre-configured in /etc/gsissh/sshd_config per default. I'll come up with patches for this. Dito for GSI-OpenSSH 7.4 w/HPN for EPEL7 now that we have a solution/workaround for the segfaults when selecting the AES-CTR-MT cipher from the client.

fscheiner commented 4 years ago

@rapier1 UPDATE: Ok, this also happens for newer gsisshds (e.g. 8.0p1 based). I see, this is due to rapier1/openssh-portable#22. As that issue is already closed, what is the solution, as I didn't find a commit regarding that issue?

@rapier1 Ok, found it. I'll include that change for 7.9p1 and 8.0p1.

fscheiner commented 4 years ago

@adorsey-NOAA GSI-OpenSSH w/HPN packages for EPEL7 are available now from https://koji.fedoraproject.org/koji/buildinfo?buildID=1635832

Details here: https://src.fedoraproject.org/rpms/gsi-openssh/pull-request/3

Update for EPEL8 is on the way.

Details here: https://src.fedoraproject.org/rpms/gsi-openssh/pull-request/4

ellert commented 4 years ago

EPEL 7 update: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-7e2b65b63e EPEL 8 update: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-8f4051fdc0

asdorsey commented 4 years ago

@fscheiner @ellert Thanks for that, I'll test those packages ASAP.

asdorsey commented 4 years ago

@fscheiner @ellert @rapier1 I tested the GSI-OpenSSH packages provided at https://koji.fedoraproject.org/koji/buildinfo?buildID=1635832 on our test DTN systems.

I immediately ran into the previously mentioned issue with the segfaults, but setting DisableMTAES yes and restarting gsisshd fixed the issue as previously described.

Transfer speeds with gsiscp are around 195MB/s, which is what I'm expecting from this remote site. Transfer speeds with rsync are worse (~30MB/s) than gsiscp, but also within the range we've come to expect. Every so often rsync performance will spike up to par with gsiscp, I'm not sure why but I don't think it's an issue with gsi-openssh because it happens with normal SSH servers as well.

The issue I described in my comment from April 7 is no longer present. With these packages, I no longer see the corrupted log messages when users disconnect.

I'm also no longer seeing the issue where child gsisshd processes are hanging around after a user disconnects.

The packages that @fscheiner provided fix all of the issues I was seeing. I'm excited to see these get into EPEL (and hopefully get this ticket closed finally).

fscheiner commented 4 years ago

@adorsey-NOAA

I immediately ran into the previously mentioned issue with the segfaults, but setting DisableMTAES yes and restarting gsisshd fixed the issue as previously described.

Actually that should be already active after installation because this setting is included in the shipped sshd_config AFAIK. But I think rpm by default does not overwrite existing config files from a previous package installation.

And just to be sure, you saw these segfaults only for the 7.4p1 version, right?

Transfer speeds with gsiscp are around 195MB/s, which is what I'm expecting from this remote site. Transfer speeds with rsync are worse (~30MB/s) than gsiscp, but also within the range we've come to expect. Every so often rsync performance will spike up to par with gsiscp, I'm not sure why but I don't think it's an issue with gsi-openssh because it happens with normal SSH servers as well.

I wonder if it could make a difference if you would also disable the AES-CTR-MT cipher for the client (in ssh_config) if you have processors with AES-NI instructions in use and an AES cipher is used during a transfer. E.g. this page recommends the following (with arcfour replaced with aes128-gcm@openssh.com as per https://gist.github.com/KartikTalwar/4393116#gistcomment-1707049):

rsync -a -P -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" [...]

Maybe this makes a difference for your use case.

asdorsey commented 4 years ago

Actually that should be already active after installation because this setting is included in the shipped sshd_config AFAIK. But I think rpm by default does not overwrite existing config files from a previous package installation.

Correct; I had an existing customized config file so I had to add that entry directly. It was present in /etc/gsissh/sshd_config.rpmnew.

And just to be sure, you saw these segfaults only for the 7.4p1 version, right?

Also correct.

I wonder if it could make a difference if you would also disable the AES-CTR-MT cipher for the client (in ssh_config) if you have processors with AES-NI instructions in use and an AES cipher is used during a transfer.

Thanks, I'll give that a try. That issue doesn't have to hold up resolving this issue, but I really appreciate the pointer.

EDIT: Looks like the performance issues with rsync were on the client, related to checksumming; adding the -W (whole file) parameter increased performance to match gsiscp. Thanks for the pointers, they got me looking in the right direction.

e-sakane commented 4 years ago

Hi, @fscheiner

May I please ask you a question about the HPN-SSH ? We are pleased that the GCT has released the gsi-openssh with the HPN-SSH. Will the GCT continue to include the HPN-SSH patch in the gsi-openssh package in the future release ?

Thank you in advance.

fscheiner commented 4 years ago

@e-sakane

We are pleased that the GCT has released the gsi-openssh with the HPN-SSH.

You're welcome! Also many thanks go to @adorsey-NOAA for creating that very issue here - which actually kicked all that off - and even more thanks go to @rapier1 for keeping the HPN patches and @ellert for keeping the GSI patch going the whole time.

Will the GCT continue to include the HPN-SSH patch in the gsi-openssh package in the future release ?

I can't speak for all of the GridCF, but I'd say, as long as @rapier1 can provide us with updated HPN patches and - again not to forget - also @ellert can provide us with updated GSI patches, GSI-OpenSSH w/HPN can exist in EPEL/Fedora and for SUSE.

fscheiner commented 4 years ago

@adorsey-NOAA

EDIT: Looks like the performance issues with rsync were on the client, related to checksumming; adding the -W (whole file) parameter increased performance to match gsiscp. Thanks for the pointers, they got me looking in the right direction.

Great that you found a solution. I only wonder, why not using -W didn't hinder performance with the older GSI-OpenSSH from XSEDE you mentioned in an earlier comment.

asdorsey commented 4 years ago

Great that you found a solution. I only wonder, why not using -W didn't hinder performance with the older GSI-OpenSSH from XSEDE you mentioned in an earlier comment.

Honestly @fscheiner I'm not sure. It might have been my test method and whether I cleaned up the destination filesystem first - supposedly rsync will use -W implicitly if the remote file doesn't exist. However, I didn't pay that much attention to the destination filesystem while I was testing before.

e-sakane commented 4 years ago

@fscheiner

Thank you for your reply ! The HPCI project in Japan uses the GSI-OpenSSH with HPN-SSH, so I would like to provide feedback as possible.

rapier1 commented 4 years ago

Hey all, just wanted to report that a new version of HPN-SSH (15v1) is now available. There are two major changes in this one. 1) The internal buffer sizes have all been normalized to 32k. This reduces the number of memcopys and read/write syscalls. My tests indicate that this gives a small but noticeable improvement. On my systems that was around 5%. 2) For those using the None cipher switch you'll now have the option of disabling HMAC as well. On my testbed this resulted, under optimal conditions, a 50% improvement in throughput in comparison to None with HMAC (500MB/s to 900MB/s). Obviously this is a use at your own risk thing, don't use for sensitive data, etc.

I don't have a patch for GSI yet but it shouldn't be overly difficult. No promises on when I will get to it but I will.

fscheiner commented 4 years ago

Hey all, just wanted to report that a new version of HPN-SSH (15v1) is now available.

Great news, saw that already last week on SourceForge. :-)

I don't have a patch for GSI yet but it shouldn't be overly difficult. No promises on when I will get to it but I will.

I'm already on it and prepared GSI-OpenSSH 8.4p1 w/HPN packages for Fedora 33 . But there seems to be an issue with at least GSI-OpenSSH 8.3p1 and 8.4p1 (w/ and w/o HPN!) that break a successful GSI authentication for me. I'll look at 8.2p1 and 8.1p1 to see if these are also affected. At least the clients are working - assumingly - up to 8.3p1 according to my earlier testing.

asdorsey commented 4 years ago

I just noticed that the updated packages have landed in EPEL, and I am seeing those now in our local EPEL mirror.

From my perspective, this issue can be closed, unless you want to keep it open to handle the newest update.

Thank you all again for the hard work to get HPN-SSH back into the EPEL.version of gsi-openssh.

rapier1 commented 4 years ago

Thanks for all the work to get HPN into gsi-openssh! I'm more than happy to know that my work is helping other people. If any issues come up again please contact me and I'll do what is necessary to get things working. Also, please join the hpnssh community mailing list at hpnssh-community@psc.edu. We'll also have a dedicated website "real soon now"

fscheiner commented 3 years ago

I'm referencing https://github.com/openssh-gsskex/openssh-gsskex/issues/18 here.

To make it short: The GSS key exchange (KEX) - and therefore also GSI authentication - is broken since (GSI-)OpenSSH 8.0p1 with the exception of the SHA1 based GSS group exchange (GEX). Until now I wasn't able to find the exact error and so far this was not taken up by the maintainers of the GSS KEX patches for OpenSSH (used by both Fedora and Debian). So I'd be happy for any support or "publicity". E.g. if you consider that as important as I do, comment on the above referenced issue appropriately.

More details in the above mentioned issue.

I detected this when integrating the HPN patches into the more recent GSI-OpenSSH versions for currently supported Fedora distributions and testing the resulting binaries.

CCing to @msalle @ellert @matyasselmeci @rapier1 @adorsey-NOAA @e-sakane just to be sure you're informed.

e-sakane commented 3 years ago

Hi @fscheiner -san,

We encountered the same issue the other day. Taking the past gsi-openssh implementation into consideration, we have addressed the issue and modified the following patches included in the gsi-openssh-8.0p1-6.el8.src.rpm:

I hasten to share the modified patches. Please see the attached. We hope those help you. But, we are sorry for no detail description of the modification.

openssh-7.7p1-fips.patch.txt openssh-8.0p1-crypto-policies.patch.txt openssh-8.0p1-gsissh.patch.txt openssh-8.0p1-gssapi-keyex.patch.txt

fscheiner commented 3 years ago

@e-sakane Awesome, that's really great news, thanks a lot! I'll build and test the results asap and report back.

fscheiner commented 3 years ago

@e-sakane Small update: I now went through all patches you provided and compared them to the original ones. The modified patches primarily remove the SHA256 based GSS KEX methods from the list of default GSS KEX methods (GSS_KEX_DEFAULT_KEX in ssh-gss.h) and put the SHA1 based GSS GEX method to the top. This GSS GEX method definitely works from my testing, but the also SHA1 based GSS Group14 KEX does not IIRC, because it is affected by the same issue that also breaks the other SHA256 based GSS KEX methods. But as the GSS GEX method is used first because of its position in the defaults list, GSI auth should work again with the provided patches (as long as the local crypto policies do not enforce other methods via command line arguments). So I think you can also get your result by configuring the GSS KEX methods via command line arguments as I did for my testing here for example.

But the SHA256 based GSS KEX methods are not broken per se - they were working perfectly with GSI-OpenSSH 7.9p1. And I actually would like to make that working again for the future as the SHA1 based methods are going to be phased out sooner or later.

My current plan is to patch kexgss_server() and its dependencies back to a working state. And maybe the easiest way would be to exchange it with the functionality used in GSI-OpenSSH 7.9p1.

rapier1 commented 3 years ago

Sort but not entirely related question - has anyone done any work on porting GSI to OpenSSH 8.4p1 as a of yet? I just built a better package for CentOS 8.3 (it installs in /opt and uses it's own systemd routines now) but I'm using the 8.4p1 code base.

Chris

On 2/15/21 8:36 AM, fscheiner wrote:

@e-sakane https://github.com/e-sakane Small update: I now went through all patches you provided and compared them to the original ones. The modified patches primarily remove the SHA256 based GSS KEX methods from the list of default GSS KEX methods (|GSS_KEX_DEFAULT_KEX| in |ssh-gss.h|) and put the SHA1 based GSS GEX method to the top. This GSS GEX method definitely works from my testing, but the also SHA1 based GSS Group14 KEX does not IIRC, because it is affected by the same issue that also breaks the other SHA256 based GSS KEX methods. But as the GSS GEX method is used first because of its position in the defaults list, GSI auth should work again with the provided patches (as long as the local crypto policies do not enforce other methods via command line arguments). So I think you can also get your result by configuring the GSS KEX methods via command line arguments as I did for my testing here https://github.com/openssh-gsskex/openssh-gsskex/issues/18#issue-786223010 for example.

But the SHA256 based GSS KEX methods are not broken per se - they were working perfectly with GSI-OpenSSH 7.9p1. And I actually would like to make that working again for the future as the SHA1 based methods are going to be phased out sooner or later.

My current plan is to patch |kexgss_server()| and its dependencies back to a working state. And maybe the easiest way would be to exchange it with the functionality used in GSI-OpenSSH 7.9p1.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-779228879, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66EYX36L4ZGMSUCVRF3S7EPNXANCNFSM4JEZMT3Q.

fscheiner commented 3 years ago

Hi Chris,

Sort but not entirely related question - has anyone done any work on porting GSI to OpenSSH 8.4p1 as a of yet?

Yes, thanks to @ellert GSI-OpenSSH 8.4p1 is available in Fedora 33, see https://src.fedoraproject.org/rpms/gsi-openssh. I assume it still misses the fixes for https://github.com/openssh-gsskex/openssh-gsskex/issues/18 and I still need to adapt your patches for GSI-OpenSSH 8.3p1 and 8.4p1.

Cheers, Frank

rapier1 commented 3 years ago

Thank Frank!

Is the gsissh patch limited to the openssh-8.4p1-gsissh.patch file or are there more dependencies? My goal is to wrap it into the Centos 8.3 package I'm building. That said, before I rush into this I'm thinking it would be better if I got you patches for HPNSSH for 8.3p1 and 8.4p1 rather than trying to roll my own. Would that work better for you or are you good with incorporating the newer versions of HPNSSH when you have time?

On 3/5/21 2:37 AM, fscheiner wrote:

Hi Chris,

Sort but not entirely related question - has anyone done any work on
porting GSI to OpenSSH 8.4p1 as a of yet?

Yes, thanks to @ellert https://github.com/ellert GSI-OpenSSH 8.4p1 is available in Fedora 33, see https://src.fedoraproject.org/rpms/gsi-openssh https://src.fedoraproject.org/rpms/gsi-openssh. I assume it still misses the fixes https://github.com/openssh-gsskex/openssh-gsskex/pulls/19 for openssh-gsskex/openssh-gsskex#18 https://github.com/openssh-gsskex/openssh-gsskex/issues/18 and I still need to adapt your patches for GSI-OpenSSH 8.3p1 and 8.4p1.

Cheers, Frank

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gridcf/gct/issues/108#issuecomment-791222335, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66GHAISHT2W4IXHCBKLTCCC2ZANCNFSM4JEZMT3Q.

fscheiner commented 3 years ago

Thank Frank! Is the gsissh patch limited to the openssh-8.4p1-gsissh.patch file or are there more dependencies?

It also needs everthing related to GSS IIC.

My goal is to wrap it into the Centos 8.3 package I'm building. That said, before I rush into this I'm thinking it would be better if I got you patches for HPNSSH for 8.3p1 and 8.4p1 rather than trying to roll my own. Would that work better for you or are you good with incorporating the newer versions of HPNSSH when you have time?

The latter would work perfectly for me, as I did it that way already for the GSI-OpenSSH versions in EPEL 7 and EPEL 8. I was just held back for Fedora by the GSS KEX problems mentioned earlier - which are fortunately solved now. I usually took the HPN patches from Sourceforge and adapted them - which became easier with every new version - to GSI-OpenSSH. So if you continue to provide them there, I'll pick them up next week or so. There's no need to to adapt them yourself.

rapier1 commented 3 years ago

On 3/5/21 11:09 AM, fscheiner wrote:

The latter would work perfectly for me, as I did it that way already for the GSI-OpenSSH versions in EPEL 7 and EPEL 8. I was just held back for Fedora by the GSS KEX problems mentioned earlier - which are fortunately solved now. I usually took the HPN patches from Sourceforge and adapted them - which became easier with every new version - to GSI-OpenSSH. So if you continue to provide them there, I'll pick them up next week or so. There's no need to to adapt them yourself. So there is source RPM up there for 8.4p1 (HPNSSH 15v1) at https://sourceforge.net/projects/hpnssh/files/RPMS/HPN-SSH%2015v1%208.4p1/SRC%20RPM/ I don't have the one for 8.3p1 up there but give me a few and I'll try to find it and build it for you.

We also have the patchfiles for 8.4 and 8.3 at https://sourceforge.net/projects/hpnssh/files/Patches/

Alternatively there are the github sources but I can understand that being a pain in the butt.

Let me know if you have any issues and when you get your rpms built. I'd like to roll them out to PSC soon.

Thanks!

Chris

fscheiner commented 3 years ago

@rapier1

So there is source RPM up there for 8.4p1 (HPNSSH 15v1) at https://sourceforge.net/projects/hpnssh/files/RPMS/HPN-SSH%2015v1%208.4p1/SRC%20RPM/ I don't have the one for 8.3p1 up there but give me a few and I'll try to find it and build it for you.

No worries, it's OK for me to just use the patches, as I have to "adapt" (piece of cake since a while) each HPN patch anyhow to the GSI-OpenSSH sources no matter if I download them from Sourceforge or fetch them from the OpenSSH source RPMs on SourceForge.

Alternatively there are the github sources but I can understand that being a pain in the butt.

Indeed, I would prefer to use the patches instead of the GitHub sources. :-)

Let me know if you have any issues and when you get your rpms built. I'd like to roll them out to PSC soon. Thanks! Chris

I have GSI-OpenSSH 8.3p1 (for Fedora 32) and GSI-OpenSSH 8.4p1 (for Fedora 33) w/HPN ready in the meantime - 8.3p1 got patched already last year, but because of https://github.com/openssh-gsskex/openssh-gsskex/issues/18 it didn't work until I applied the patches from https://github.com/openssh-gsskex/openssh-gsskex/pull/19. With these patches both of them work successfully with all available GSS KEX/GEX methods:

[johndoe@host ~]$ sudo bin/test-gss-kex-for-gsi-openssh.bash host.domain.tld johndoe2
gsisshd: OpenSSH_8.4p1c-GSI GSI-hpn15v1, OpenSSL 1.1.1j  FIPS 16 Feb 2021
gsissh: OpenSSH_8.4p1c-GSI GSI-hpn15v1, OpenSSL 1.1.1j  FIPS 16 Feb 2021

Wait 3 seconds for startup of gsisshd ...

gss-gex-sha1- OK
gss-group1-sha1- OK
gss-group14-sha256- OK
gss-nistp256-sha256- OK
gss-curve25519-sha256- OK
gss-group16-sha512- OK

~Hm, maybe I should also print the GSI-OpenSSH version number in the test script.~ Fixed.

I still need to create the PR for GSI-OpenSSH 8.4p1 over at Fedora and @ellert needs to decide (1) if we wait for Fedora to pick up the patches from https://github.com/openssh-gsskex/openssh-gsskex/pull/19 or (2) if we create separate bugs for GSI-OpenSSH and patch them on our own until Fedora updates their OpenSSH packages. Either way, IIC it will take at least two weeks in addition until the resulting RPMs will reach the "standard" package sources, but they will be available from "[...][-updates]-testing" package sources in the meantime.

For the time being here are the source RPMs I created w/HPN and fixes for https://github.com/openssh-gsskex/openssh-gsskex/issues/18 included (zipped to allow upload to GitHub): gsi-openssh-8.3p1-5.fc32.src.rpm.zip (SHA256: 3f49c77d7fa00fec9e3e93e7c69c675f0e79dcfdf20b93a456d8b46d4664e88f) gsi-openssh-8.4p1-5.fc33.src.rpm.zip (SHA256: f1423aced1c1aa57907f42c1354bba7ba9ed8b660adca1a8a53267b23a4aa250)

ellert commented 3 years ago

I have added the proposed changes from https://github.com/openssh-gsskex/openssh-gsskex/pull/19 to the Fedora/EPEL builds:

Please test and give karma if they work.

Mattias.

fscheiner commented 3 years ago

Hi @ellert, already on it. :-) Will report the results here later on.

fscheiner commented 3 years ago

I have added the proposed changes from openssh-gsskex/openssh-gsskex#19 to the Fedora/EPEL builds:

* EPEL 8 - gsi-openssh-8.0p1-7.el8
  https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-5392fab667

Works for me:

[johndoe@host ~]$ sudo bin/test-gss-kex-for-gsi-openssh.bash host.domain.tld johndoe2
gsisshd: OpenSSH_8.0p1c-GSI GSI-hpn14v19, OpenSSL 1.1.1g FIPS  21 Apr 2020
gsissh: OpenSSH_8.0p1c-GSI GSI-hpn14v19, OpenSSL 1.1.1g FIPS  21 Apr 2020

Wait 3 seconds for startup of gsisshd ...

gss-gex-sha1- OK
gss-group1-sha1- OK
gss-group14-sha256- OK
gss-nistp256-sha256- OK
gss-curve25519-sha256- OK
gss-group16-sha512- OK

[johndoe@host ~]$ yum info gsi-openssh
[...]
Name         : gsi-openssh
Version      : 8.0p1
Release      : 7.el8
[...]
Source       : gsi-openssh-8.0p1-7.el8.src.rpm
[...]
* Fedora 32 - gsi-openssh-8.3p1-5.fc32
  https://bodhi.fedoraproject.org/updates/FEDORA-2021-81c8581192

Works for me:

[johndoe@host ~]$ sudo bin/test-gss-kex-for-gsi-openssh.bash host.domain.tld johndoe2
gsisshd: OpenSSH_8.3p1c-GSI GSI-hpn14v22, OpenSSL 1.1.1i FIPS  8 Dec 2020
gsissh: OpenSSH_8.3p1c-GSI GSI-hpn14v22, OpenSSL 1.1.1i FIPS  8 Dec 2020

Wait 3 seconds for startup of gsisshd ...

gss-gex-sha1- OK
gss-group1-sha1- OK
gss-group14-sha256- OK
gss-nistp256-sha256- OK
gss-curve25519-sha256- OK
gss-group16-sha512- OK

[johndoe@host ~]$ yum info gsi-openssh
[...]
Name         : gsi-openssh
Version      : 8.3p1
Release      : 5.fc32
[...]
Source       : gsi-openssh-8.3p1-5.fc32.src.rpm
[...]
* Fedora 33 - gsi-openssh-8.4p1-6.fc33
  https://bodhi.fedoraproject.org/updates/FEDORA-2021-fa267d8125

Works for me:

[johndoe@host ~]$ sudo bin/test-gss-kex-for-gsi-openssh.bash host.domain.tld johndoe2
gsisshd: OpenSSH_8.4p1c-GSI GSI-hpn15v1, OpenSSL 1.1.1j  FIPS 16 Feb 2021
gsissh: OpenSSH_8.4p1c-GSI GSI-hpn15v1, OpenSSL 1.1.1j  FIPS 16 Feb 2021

Wait 3 seconds for startup of gsisshd ...

gss-gex-sha1- OK
gss-group1-sha1- OK
gss-group14-sha256- OK
gss-nistp256-sha256- OK
gss-curve25519-sha256- OK
gss-group16-sha512- OK

[johndoe@host ~]$ yum info gsi-openssh
[...]
Name         : gsi-openssh
Version      : 8.4p1
Release      : 6.fc33
[...]
Source       : gsi-openssh-8.4p1-6.fc33.src.rpm
[...]

* Fedora 34 - gsi-openssh-8.5p1-1.fc34
  https://bodhi.fedoraproject.org/updates/FEDORA-2021-b09f187229

~This will take a little longer, as I don't have a Fedora 34 VM yet.~ Finished.

Works for me:

[johndoe@host ~]$ sudo bin/test-gss-kex-for-gsi-openssh.bash host.domain.tld johndoe2
gsisshd: OpenSSH_8.5p1c-GSI GSI-hpn15v2, OpenSSL 1.1.1i FIPS  8 Dec 2020
gsissh: OpenSSH_8.5p1c-GSI GSI-hpn15v2, OpenSSL 1.1.1i FIPS  8 Dec 2020

Wait 3 seconds for startup of gsisshd ...

gss-gex-sha1- OK
gss-group1-sha1- OK
gss-group14-sha256- OK
gss-nistp256-sha256- OK
gss-curve25519-sha256- OK
gss-group16-sha512- OK

[johndoe@host ~]$ yum info gsi-openssh
[...]
Name         : gsi-openssh
Version      : 8.5p1
Release      : 1.fc34
[...]
Source       : gsi-openssh-8.5p1-1.fc34.src.rpm
[...]

Test script source at https://gist.github.com/fscheiner/92ea125c72cd70283a712585206c1015.

fscheiner commented 3 years ago

@ellert: Thanks BTW for including the HPN patches into GSI-OpenSSH 8.5p1 for Fedora 34 and Rawhide.

@rapier1 @adorsey-NOAA @e-sakane This also means that as soon as the latest GSI-OpenSSH RPMs leave the "(updates-)testing" package sources, every current distribution using EPEL/Fedora package sources will have HPN included in their GSI-OpenSSH versions! :-)