1.0.1g :: Missing symbols while linking libcrypto.a

skull-squadron commented 8 years ago

Looks like there maybe a dependency issue which is not present in vanilla 1.0.2g.

Edit: Fails in all tested configurations: default configure options, no-shared, no-fips, etc.

if [ -n "libcrypto.1.0.0.dylib libssl.1.0.0.dylib" ]; then \
        (cd ..; /Applications/Xcode.app/Contents/Developer/usr/bin/make libcrypto.1.0.0.dylib); \
    fi
[ -z "" ] || cc -fPIC -fno-common -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -arch x86_64 -O3 -DL_ENDIAN -Wall -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DCHAPOLY_x86_64_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -Iinclude \
        -DFINGERPRINT_PREMAIN_DSO_LOAD -o fips_premain_dso  \
        fips_premain.c fipscanister.o \
        libcrypto.a
Undefined symbols for architecture x86_64:
  "_chacha_20_core_avx", referenced from:
      _CRYPTO_chacha_20 in libcrypto.a(chacha20.o)
  "_chacha_20_core_avx2", referenced from:
      _CRYPTO_chacha_20 in libcrypto.a(chacha20.o)
  "_poly1305_finish_avx2", referenced from:
      _EVP_chacha20_poly1305_init_draft in libcrypto.a(e_chacha20poly1305.o)
      _EVP_chacha20_poly1305_init in libcrypto.a(e_chacha20poly1305.o)
  "_poly1305_update_avx2", referenced from:
      _EVP_chacha20_poly1305_init_draft in libcrypto.a(e_chacha20poly1305.o)
      _EVP_chacha20_poly1305_init in libcrypto.a(e_chacha20poly1305.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[4]: *** [link_a.darwin] Error 1
make[3]: *** [do_darwin-shared] Error 2
make[2]: *** [libcrypto.1.0.0.dylib] Error 2
make[1]: *** [shared] Error 2
make: *** [build_crypto] Error 1

More build logs here updated: minimal examples

skull-squadron commented 8 years ago

Will troubleshoot because there is some dependency differences between Homebrew and user-attended causing it to fail.

skull-squadron commented 8 years ago

Intel's avx-optimized asm scripts appear to output empty files with all of the following nasm versions on OS X:

0.98.40 (Apple Computer, Inc. build 11)
2.12
2.12 (head)

AVX Codegen Scripts

./crypto/chacha20poly1305/asm/chacha20_avx.pl &
./crypto/chacha20poly1305/asm/chacha20_avx2.pl &
./crypto/chacha20poly1305/asm/poly1305_avx.pl
./crypto/chacha20poly1305/asm/poly1305_avx2.pl &
./crypto/chacha20poly1305/asm/poly1305_x64.pl

& Generating empty .s files

Example (no output)

(cd crypto/chacha20poly130 && /usr/bin/perl asm/chacha20_avx2.pl macosx)

Perhaps non-avx platforms need dependencies altered to match fallbacks to non-avx/-avx2 implementations.

felixbuenemann commented 8 years ago

@steakknife The problem is due to newer versions of Apple's clang no longer showing the original clang version, so you need to add another avx check at the top of each perl file:

if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
  $avx = 2;
}

After that the files get generated, but I'm running into another error:

Undefined symbols for architecture x86_64:
  "poly1305_finish_x64", referenced from:
      _poly1305_finish_avx2 in libcrypto.a(poly1305_avx2.o)
     (maybe you meant: _poly1305_finish_x64)
  "poly1305_update_x64", referenced from:
      _poly1305_update_avx2 in libcrypto.a(poly1305_avx2.o)
     (maybe you meant: _poly1305_update_x64)
ld: symbol(s) not found for architecture x86_64

I fixed this manually by replacing jmp poly1305_update_x64with jmp _poly1305_update_x64 inside poly1305_avx2.s it seems this is a translation error by the x86_64-xlate.pl script, which should also mangle the symbol for the jump, because it references a global that is mangled.

felixbuenemann commented 8 years ago

After reading the code of x86_64-xlate.pl I noticed that those unmagled symbols should've been marked as extern, so the complete fix is:

diff --git a/crypto/chacha20poly1305/asm/chacha20_avx.pl b/crypto/chacha20poly1305/asm/chacha20_avx.pl
index bf3e3f0..4a8e9f3 100644
--- a/crypto/chacha20poly1305/asm/chacha20_avx.pl
+++ b/crypto/chacha20poly1305/asm/chacha20_avx.pl
@@ -68,6 +68,10 @@ if (`$ENV{CC} -v 2>&1` =~ /(^clang version|based on LLVM) ([3-9])\.([0-9]+)/) {
   $avx = ($ver>=3.0) + ($ver>=3.01);
 }

+if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
+  $avx = 2;
+}
+
 if ($avx>=1) {{

 my ($rol8, $rol16, $state_cdef, $tmp,
diff --git a/crypto/chacha20poly1305/asm/chacha20_avx2.pl b/crypto/chacha20poly1305/asm/chacha20_avx2.pl
index 9f31f86..f6dcb31 100644
--- a/crypto/chacha20poly1305/asm/chacha20_avx2.pl
+++ b/crypto/chacha20poly1305/asm/chacha20_avx2.pl
@@ -67,6 +67,10 @@ if (`$ENV{CC} -v 2>&1` =~ /(^clang version|based on LLVM) ([3-9])\.([0-9]+)/) {
   $avx = ($ver>=3.0) + ($ver>=3.01);
 }

+if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
+  $avx = 2;
+}
+
 if ($avx>=2) {{

 my ($state_4567, $state_89ab, $state_cdef, $tmp,
diff --git a/crypto/chacha20poly1305/asm/poly1305_avx.pl b/crypto/chacha20poly1305/asm/poly1305_avx.pl
index cedeca1..c30f6a1 100644
--- a/crypto/chacha20poly1305/asm/poly1305_avx.pl
+++ b/crypto/chacha20poly1305/asm/poly1305_avx.pl
@@ -74,6 +74,10 @@ if (`$ENV{CC} -v 2>&1` =~ /(^clang version|based on LLVM) ([3-9])\.([0-9]+)/) {
         $avx = ($ver>=3.0) + ($ver>=3.01);
 }

+if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
+        $avx = 2;
+}
+
 if ($avx>=1) {{

 my ($_r0_, $_r1_, $_r2_, $_r3_, $_r4_,
diff --git a/crypto/chacha20poly1305/asm/poly1305_avx2.pl b/crypto/chacha20poly1305/asm/poly1305_avx2.pl
index d2dd51f..90dc4a8 100644
--- a/crypto/chacha20poly1305/asm/poly1305_avx2.pl
+++ b/crypto/chacha20poly1305/asm/poly1305_avx2.pl
@@ -57,6 +57,10 @@ if (`$ENV{CC} -v 2>&1` =~ /(^clang version|based on LLVM) ([3-9])\.([0-9]+)/) {
         $avx = ($ver>=3.0) + ($ver>=3.01);
 }

+if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
+        $avx = 2;
+}
+
 if ($avx>=1) {{

 my ($_r0_, $_r1_, $_r2_, $_r3_, $_r4_,
@@ -395,6 +399,7 @@ my ($state, $in, $in_len, $hlp, $rsp_save)

 $code.=<<___;

+.extern poly1305_update_x64
 ###############################################################################
 # void poly1305_update_avx2(void* $state, void* in, uint64_t in_len)
 .globl poly1305_update_avx2
@@ -907,6 +912,7 @@ poly1305_update_avx2:
 6:
   ret
 .size poly1305_update_avx2,.-poly1305_update_avx2
+.extern poly1305_finish_x64
 ###############################################################################
 # void poly1305_finish_avx2(void* $state, uint8_t mac[16]);
 .type poly1305_finish_avx2,\@function,2
diff --git a/crypto/chacha20poly1305/asm/poly1305_x64.pl b/crypto/chacha20poly1305/asm/poly1305_x64.pl
index 31c4c47..478f707 100644
--- a/crypto/chacha20poly1305/asm/poly1305_x64.pl
+++ b/crypto/chacha20poly1305/asm/poly1305_x64.pl
@@ -55,6 +55,10 @@ if (`$ENV{CC} -v 2>&1` =~ /(^clang version|based on LLVM) ([3-9])\.([0-9]+)/) {
         $avx = ($ver>=3.0) + ($ver>=3.01);
 }

+if (`$ENV{CC} -v 2>&1` =~ /^Apple LLVM version [7-9]/) {
+        $avx = 2;
+}
+

 {
 {

After that compilation and make test on macosx with OpenSSL_1_0_2-stable is successful.

skull-squadron commented 7 years ago

Awesome, thanks Apple (sarcasm) while thanks (non-sarcasm) @felixbuenemann.

cloudflare / sslconfig

1.0.1g :: Missing symbols while linking libcrypto.a #31

AVX Codegen Scripts

Example (no output)