search

rapier1 / hpn-ssh

HPN-SSH based on OpenSSH
https://psc.edu/hpn-ssh-home
Other
302 stars 41 forks source link

Ubuntu 22.04 Openssl 3.0 and 1.1.1t #47

Closed jimthedj65 closed 10 months ago

jimthedj65 commented 1 year ago

Hi All

Installed successfully on 22.04 server but for some reason my second build errors out at make -j16

I get undefined errors as below

/usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in functionchachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(sshkey.o): in functionsshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:./libssh.a(digest-openssl.o)3464: in function: undefined reference to ssh_digest_blocksizeEVP_PKEY_base_id': ' /home/myhost/openssh-portable/digest-openssl.c:111/usr/bin/ld: undefined reference to: EVP_MD_block_size' /home/myhost/openssh-portable/sshkey.c:3476: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in functionchachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /usr/bin/ld: ./libssh.a(cipher.o): in functioncipher_init': /home/myhost/openssh-portable/cipher.c:418: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: ./libssh.a(cipher.o): in functioncipher_get_keyiv': /home/myhost/openssh-portable/cipher.c:603: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(cipher.o): in functioncipher_set_keyiv./libssh.a(digest-openssl.o)': : in function /home/myhost/openssh-portable/cipher.c:634ssh_digest_blocksize: undefined reference to': EVP_CIPHER_CTX_iv_length' /home/myhost/openssh-portable/digest-openssl.c:/usr/bin/ld111: : undefined reference to EVP_MD_block_size./libssh.a(cipher.o)' : in functioncipher_get_keyiv_len': /home/myhost/openssh-portable/cipher.c:574: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in functionEVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in functionEVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:collect2: error: ld returned 1 exit status 343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' ./libssh.a(cipher-chachapoly-libcrypto.o): in functionchachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' collect2: error: ld returned 1 exit status /usr/bin/ld: make: *** [Makefile:220: hpnssh-agent] Error 1 make: *** Waiting for unfinished jobs.... ./libssh.a(digest-openssl.o): in functionssh_digest_blocksize': /home/myhost/openssh-portable/digest-openssl.c:111: undefined reference to EVP_MD_block_size' /usr/bin/ld: make: *** [Makefile:229: hpnssh-pkcs11-helper] Error 1 ./libssh.a(cipher-ctr-mt.o): in functionssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference toEVP_CIPHER_CTX_key_length' /usr/bin/ld: /usr/bin/ld: ./libssh.a(cipher-ctr-mt.o): in function ssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference toEVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in functionchachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(digest-openssl.o): in functionssh_digest_blocksize': /home/myhost/openssh-portable/digest-openssl.c:111: undefined reference to EVP_MD_block_size' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in functionEVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in functionEVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference toEVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' ssh-keygen.o: in functiondo_convert_from_pkcs8': /home/myhost/openssh-portable/ssh-keygen.c:702: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/ssh-keygen.c:725: undefined reference toEVP_PKEY_base_id' /usr/bin/ld: /usr/bin/ld: collect2: error: ld returned 1 exit status collect2: error: ld returned 1 exit status make: [Makefile:217: hpnssh-add] Error 1 make: [Makefile:232: hpnssh-sk-helper] Error 1 ./libssh.a(sshkey.o): in function sshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference toEVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:3464: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:3476: undefined reference toEVP_PKEY_base_id' /usr/bin/ld: ./libssh.a(cipher-ctr-mt.o): in function ssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference toEVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference to EVP_CIPHER_CTX_key_length' ./libssh.a(sshkey.o): in functionsshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference to EVP_PKEY_base_id'

Any ideas, In have checked dependencies as best as I can, no errors on the ./configure stage.

Thanks for any guidance.

rapier1 commented 1 year ago

Hey there, I'll take a look in a couple of days. I'm out of town and stuck on my phone until Tuesday. What did you use as your configuration line?

On Sun, Jun 18, 2023, 12:55 jimthedj65 @.***> wrote:

Hi All

Installed successfully on 22.04 server but for some reason my second build errors out at make -j16

I get undefined errors as below

/usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in function chachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(sshkey.o): in function sshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:./libssh.a(digest-openssl.o)3464: in function : undefined reference to ssh_digest_blocksizeEVP_PKEY_base_id': ' /home/myhost/openssh-portable/digest-openssl.c:111/usr/bin/ld: undefined reference to : EVP_MD_block_size' /home/myhost/openssh-portable/sshkey.c:3476: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in function chachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /usr/bin/ld: ./libssh.a(cipher.o): in function cipher_init': /home/myhost/openssh-portable/cipher.c:418: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: ./libssh.a(cipher.o): in function cipher_get_keyiv': /home/myhost/openssh-portable/cipher.c:603: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(cipher.o): in function cipher_set_keyiv./libssh.a(digest-openssl.o)': : in function /home/myhost/openssh-portable/cipher.c:634ssh_digest_blocksize: undefined reference to ': EVP_CIPHER_CTX_iv_length' /home/myhost/openssh-portable/digest-openssl.c:/usr/bin/ld111: : undefined reference to EVP_MD_block_size./libssh.a(cipher.o)' : in function cipher_get_keyiv_len': /home/myhost/openssh-portable/cipher.c:574: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:collect2: error: ld returned 1 exit status 343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' ./libssh.a(cipher-chachapoly-libcrypto.o): in function chachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' collect2: error: ld returned 1 exit status /usr/bin/ld: make: [Makefile:220: hpnssh-agent] Error 1 make: Waiting for unfinished jobs.... ./libssh.a(digest-openssl.o): in function ssh_digest_blocksize': /home/myhost/openssh-portable/digest-openssl.c:111: undefined reference to EVP_MD_block_size' /usr/bin/ld: make: [Makefile:229: hpnssh-pkcs11-helper] Error 1 ./libssh.a(cipher-ctr-mt.o): in function ssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: /usr/bin/ld: ./libssh.a(cipher-ctr-mt.o): in function ssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: ./libssh.a(cipher-chachapoly-libcrypto.o): in function chachapoly_new': /home/myhost/openssh-portable/cipher-chachapoly-libcrypto.c:68: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: ./libssh.a(digest-openssl.o): in function ssh_digest_blocksize': /home/myhost/openssh-portable/digest-openssl.c:111: undefined reference to EVP_MD_block_size' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_get_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:343: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:345: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: openbsd-compat//libopenbsd-compat.a(libressl-api-compat.o): in function EVP_CIPHER_CTX_set_iv': /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:372: undefined reference to EVP_CIPHER_CTX_iv_length' /usr/bin/ld: /home/myhost/openssh-portable/openbsd-compat/libressl-api-compat.c:374: undefined reference to EVP_CIPHER_CTX_iv_length' ssh-keygen.o: in function do_convert_from_pkcs8': /home/myhost/openssh-portable/ssh-keygen.c:702: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/ssh-keygen.c:725: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /usr/bin/ld: collect2: error: ld returned 1 exit status collect2: error: ld returned 1 exit status make: [Makefile:217: hpnssh-add] Error 1 make: *** [Makefile:232: hpnssh-sk-helper] Error 1 ./libssh.a(sshkey.o): in function sshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:3464: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: /home/myhost/openssh-portable/sshkey.c:3476: undefined reference to EVP_PKEY_base_id' /usr/bin/ld: ./libssh.a(cipher-ctr-mt.o): in function ssh_aes_ctr_init': /home/myhost/openssh-portable/cipher-ctr-mt.c:581: undefined reference to EVP_CIPHER_CTX_key_length' /usr/bin/ld: /home/myhost/openssh-portable/cipher-ctr-mt.c:584: undefined reference to EVP_CIPHER_CTX_key_length' ./libssh.a(sshkey.o): in function sshkey_parse_private_pem_fileblob': /home/myhost/openssh-portable/sshkey.c:3447: undefined reference to EVP_PKEY_base_id'

Any ideas, In have checked dependencies as best as I can, no errors on the ./configure stage.

Thanks for any guidance.

— Reply to this email directly, view it on GitHub https://github.com/rapier1/openssh-portable/issues/47, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66AGPZ2JC7EJ3BRO23TXL4XI7ANCNFSM6AAAAAAZK763CU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jimthedj65 commented 1 year ago

Hey thanks for such a fast response I used

autoreconf -f -i ./configure --without-openssl-header-check

jimthedj65 commented 1 year ago

I forgot to mention that I got a headers our of sync

configure: error: Your OpenSSL headers do not match your library. Check config.log for details. If you are sure your installation is consistent, you can disable the check by running "./configure --without-openssl-header-check". Also see contrib/findssl.sh for help identifying header/library mismatches.

I then ran ./configure --without-openssl-header-check

PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

rapier1 commented 1 year ago

That would likely be the problem. My guess is that it's getting the headers from the openssl 3.0 default installation and using the libraries from openssl 1.1. so there will be a mismatch due to differences in the API. There is a specifically addition to the configure line that you need to get the right headers. I'll get that to you when I'm back in front of my computer.

On Sun, Jun 18, 2023, 17:05 jimthedj65 @.***> wrote:

I forgot to mention that I got a headers our of sync

configure: error: Your OpenSSL headers do not match your library. Check config.log for details. If you are sure your installation is consistent, you can disable the check by running "./configure --without-openssl-header-check". Also see contrib/findssl.sh for help identifying header/library mismatches.

I then ran ./configure --without-openssl-header-check

— Reply to this email directly, view it on GitHub https://github.com/rapier1/openssh-portable/issues/47#issuecomment-1596269804, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKL66FF54PDPLTC5HK4QR3XL5US3ANCNFSM6AAAAAAZK763CU . You are receiving this because you commented.Message ID: @.***>

jimthedj65 commented 1 year ago

ok thanks is there a switch on configure to use specifically 1.1.1

this is exactly what comes back

checking whether snprintf correctly terminates long strings... yes checking whether snprintf understands %zu... yes checking whether vsnprintf returns correct values on overflow... yes checking whether snprintf can declare const char *fmt... yes checking whether system supports SO_PEERCRED getsockopt... yes checking if openpty correctly handles controlling tty... yes checking whether AI_NUMERICSERV is declared... yes checking if SA_RESTARTed signals interrupt select()... yes checking for getpgrp... yes checking if getpgrp accepts zero args... yes checking for openssl... /usr/local/bin/openssl checking for openssl/opensslv.h... yes checking OpenSSL header version... 1010114f (OpenSSL 1.1.1t 7 Feb 2023) checking for OpenSSL_version... yes checking for OpenSSL_version_num... yes checking OpenSSL library version... 1010114f (OpenSSL 3.0.8 7 Feb 2023) checking whether OpenSSL's headers match the library... no configure: error: Your OpenSSL headers do not match your library. Check config.log for details. If you are sure your installation is consistent, you can disable the check by running "./configure --without-openssl-header-check". Also see contrib/findssl.sh for help identifying header/library mismatches.

jimthedj65 commented 1 year ago

I tried running ./findssl.sh in contrib and get a permission denied, I changed it to chmod a+x to allow it to execute and got the following.

Searching for OpenSSL header files. OPENSSL_VERSION_NUMBER /home/myhost/QAT/OLD/openssl-1.1.1q/include/openssl/opensslv.h 0x1010113fL /home/myhost/QAT/OLD/qat2.0/quickassist/utilities/osal/src/linux/user_space/openssl/opensslv.h OPENSSL_VERSION_NUMBER /home/myhost/QAT/OLD/qat_driver/quickassist/utilities/osal/src/linux/user_space/openssl/opensslv.h OPENSSL_VERSION_NUMBER /home/myhost/QAT/openssl-1.1.1t/include/openssl/opensslv.h OPENSSL_VERSION_NUMBER /home/myhost/QAT/qat1.7/quickassist/utilities/osal/src/linux/user_space/openssl/opensslv.h 0x1010113fL /home/myhost/QAT/qat2.0/quickassist/utilities/osal/src/linux/user_space/openssl/opensslv.h OPENSSL_VERSION_NUMBEROPENSSL_VERSION_NUMBER /usr/include/openssl/opensslv.h OPENSSL_VERSION_NUMBER /usr/local/include/openssl/opensslv.h OPENSSL_VERSION_NUMBEROPENSSL_VERSION_NUMBER /usr/local/src/openssl-3.0.8/include/openssl/opensslv.h OPENSSL_VERSION_NUMBEROPENSSL_VERSION_NUMBER /usr/local/src/openssl-3.0.8/include/openssl/opensslv.h.in OPENSSL_VERSION_NUMBER /usr/local/ssl/include/openssl/opensslv.h

Searching for OpenSSL shared library files.

Searching for OpenSSL static library files.

Something a bit screwy with my openssl install. any hints or advice greatly appreciated.

jimthedj65 commented 1 year ago

I have reverted to the apt-get version for now and will revisit this. I have a project that will donate to this project if we can see significant uplift.

jimthedj65 commented 1 year ago

on one of my clients I have hpnscp stalling on a transfer scp runs fine. What would cause hpnscp to stall ?

rapier1 commented 1 year ago

Hey there,

So I would try to clean up your openssl install. Basically, the headers that describe what is actually in the OpenSSL libraries are different than the actual library that is going to be used. In this case it's finding OSSL 1.1 headers but the library is OSSL 3.0. OSSL 3.0 has a number of differences and we load different sections of the HPN-SSH code depending on the version we are told about. The end result is that we end up calling functions that are described in OSSL 1.1 but don't exist in OSSL 3.0 we are actually linking against.

My suggestion is to use the packaged version of OpenSSL. If you want to install from source I suggest building OpenSSL with a /opt installation prefix. For example, in the OpenSSL source directory you'd use "./Configure --prefix=/opt/openssl-3.0" to have the headers, libraries, and applications installed into /opt/openssl-3.0.

In HPN-SSH you'd then tell it to use that specific installation with "./configure -with-ssl-dir=/opt/openssl-3.0 --with-rpath=-Wl,-rpath," You need the --with-rpath statement for it to work properly.

As for hpnscp hanging - can you tell me more about what you are doing? The specific command line and anything from the debugging output would be helpful.

jimthedj65 commented 1 year ago

perfect thanks, I will give that a try. at the moment I am trying to see how I can get this to improve a low-bandwidth ADSL circuit? does it have a sweet spot in terms of minimum bandwidth to be effective?

rapier1 commented 1 year ago

You probably aren't going to see much improvement with that sort network path. A large part of the performance improvement comes from matching the internal SSH buffers to the TCP receive buffer. Basically, TCP receive buffers automatically grow to meet the outstanding data capacity of a path. That means how much data can be in flight at any one point without having been acknowledged by the receiver. The outstanding data capacity can be computed by what is called the bandwidth delay product. That's the bandwidth of the connection at the slowest point multiplied by the round trip time (or delay).

So if you have a 1Gb connection and you are transferring data to a host 75ms away you'd need a receive buffer of 8.94MB in order to fully fill the connection with data. Now, OpenSSH has its own flow control mechanism that rides on top of this. In OpenSSH this is limited to about 1.5MB. Being that this buffer is less than the TCP buffer it acts as a limit on data throughput. So on the 1GB path with 75ms of delay OpenSSH would be limited to just under 175Mb/s. Part of what HPN-SSH does is make the application aware of the current TCP receive buffer size and grows the internal flow control buffer to match it. This also means that you need HPN-SSH on whichever side is accepting the data to see this improvement as it's dealing with receiver side buffers.

ADSL lines are pretty slow so even if you have a high round trip time (rtt) the outstanding data capacity likely won't exceed the internal buffer limits of OpenSSH. In that case, you likely won't see an improvement from HPN-SSH's more advanced flow control. However, if OpenSSH is CPU limited (like you are hitting 100% every time you run it) HPN-SSH may be of help because we use more efficient threaded ciphers (use -caes256-ctr on the command line to try it).

gdevenyi commented 1 year ago

Chiming in here, as I had similar issues with OpenSSL+OpenSSH building from scratch, this is what fixed the mismatched versions for me.

  1. Configure OpenSSL with: ./config -fPIC shared --prefix=/opt/openssl --openssldir=/opt/openssl -Wl,-rpath=/opt/openssl/lib -Wl,--enable-new-dtags
  2. Configure OpenSSH with: PATH=/opt/openssl/bin:${PATH} ./configure --prefix=/opt/hpnssh --with-ssl-dir=/opt/openssl --with-rpath=-Wl,-rpath,

In particular here, its the RPATH configuration that avoids having to modify the system LD variables or /etc/ld configs, which is essential to avoid breaking other OpenSSL dependent code.