Open vext01 opened 7 years ago
@vext01 Fixed the dead link on the support web page. I cannot help with the main issue though...
@vext01: One workaround would be to download and compile OpenSSL, and link against OpenSSL instead of LibreSSL.
Does this work for you?
@triska Not really. I'm packaging for OpenBSD, so we would really like to link LibreSSL.
Failing that, I could always disable the SSL module, but ideally we could find a workaround.
Thanks
@vext01: Before we make changes in SWI-Prolog, please try to find out whether LibreSSL plans to support CRYPTO_ONCE
in the (near) future, if you need the SSL package to work.
I've just asked on our porting mailing list. Watch this space.
We won't be implementing that API in LibreSSL, but the good news is that you already have workaround code that can be used with some minor patching:
-#if OPENSSL_VERSION_NUMBER < 0x10100000L
+#if OPENSSL_VERSION_NUMBER < 0x10100000L || defined(LIBRESSL_VERSION_NUMBER)
I've applied this everywhere I see #if OPENSSL_VERSION_NUMBER < 0x10100000L
and SWI builds against LibreSSL. Does that sound right to you?
You can see my changes in my patch set: https://github.com/jasperla/openbsd-wip/tree/master/lang/swi-prolog/patches
Yes that sounds very good, thank you! Does make check
succeed for all test cases?
Please file a pull request against the SSL package, the commit starting with: PORT: ...
. We can discuss all further issues there.
No worries.
Someone informed me that LibreSSL is a mix of the OpenSSL 1.0 and 1.1 APIs. So I should try a more systematic approach to deciding which bits to patch.
Bear with me :)
Yes, please try to keep the patchset small. I will help with the port in every way I can, so please let me know if you have any questions, for example via issues you can file against the ssl package.
It seems that I only need to patch one of the version checks to get swi to build the SSL module, but I'm not sure how I should run the tests.
make check
in the top level does not seem to run any SSL tests.make check
in the ssl package dir does this:(cd tests && ./mkcerts.sh >& /dev/null)
/bin/sh: >&/dev/null : illegal file descriptor name
(The latter may be a bashism)
If the latter is what I'm supposed to be doing, I can start debugging the shell script. Please let me know.
Yes, the latter, i.e. running it in the ssl package, ought to run the tests.
Thanks, I will see what I can do.
Quick update.
I went and fixed each failure one at a time with a compilation/test cycle in between. As it turns out, it's not enough to get it to compile: I was getting lazy binding errors at runtime.
After fixing the tests runner target, I know have no symbol errors, but:
cd tests && /bin/sh mkcerts.sh
Generating a 2048 bit RSA private key
.........................................................................................................................+++
...............................................+++
writing new private key to 'test_certs/rootCA/private/cakey.pem'
-----
Using configuration from rootCA.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Jul 24 14:42:58 2017 GMT
Not After : Jul 24 14:42:58 2018 GMT
Subject:
countryName = UK
stateOrProvinceName = Scotland
localityName = Edinburgh
organizationName = SWI-Prolog
commonName = Testing Root CA
X509v3 extensions:
X509v3 Subject Key Identifier:
95:29:B0:EF:A3:EE:22:AE:94:F1:47:D5:32:3B:37:51:DD:07:41:07
X509v3 Authority Key Identifier:
keyid:95:29:B0:EF:A3:EE:22:AE:94:F1:47:D5:32:3B:37:51:DD:07:41:07
X509v3 Basic Constraints:
CA:TRUE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign, CRL Sign
Certificate is to be certified until Jul 24 14:42:58 2018 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
mkcerts.sh[20]: seq: not found
Generating a 2048 bit RSA private key
...........+++
..................+++
writing new private key to 'test_certs/14_CA/private/cakey.pem'
-----
Using configuration from 14.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4096 (0x1000)
Validity
Not Before: Jul 24 14:42:58 2017 GMT
Not After : Jul 24 14:42:58 2018 GMT
Subject:
countryName = UK
stateOrProvinceName = Scotland
localityName = Edinburgh
organizationName = SWI-Prolog
commonName = Some Untrusted CA
X509v3 extensions:
X509v3 Subject Key Identifier:
96:B0:61:9D:C0:D2:00:1E:8E:C1:AB:88:7A:52:A0:72:24:8F:20:C4
X509v3 Authority Key Identifier:
keyid:96:B0:61:9D:C0:D2:00:1E:8E:C1:AB:88:7A:52:A0:72:24:8F:20:C4
X509v3 Basic Constraints:
CA:TRUE
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign
Certificate is to be certified until Jul 24 14:42:58 2018 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
Generating a 2048 bit RSA private key
....................................................+++
.............................................+++
writing new private key to 'test_certs/14-key.pem'
-----
Using configuration from 14_tail.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4097 (0x1001)
Validity
Not Before: Jul 24 14:42:58 2017 GMT
Not After : Jul 24 14:42:58 2018 GMT
Subject:
countryName = UK
stateOrProvinceName = Scotland
localityName = Edinburgh
organizationName = SWI-Prolog
commonName = www.example.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
2E:49:0E:51:20:8B:1A:52:55:FA:96:5C:5C:05:D1:18:5A:77:69:DA
X509v3 Authority Key Identifier:
keyid:96:B0:61:9D:C0:D2:00:1E:8E:C1:AB:88:7A:52:A0:72:24:8F:20:C4
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Certificate is to be certified until Jul 24 14:42:58 2018 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
mkcerts.sh[37]: seq: not found
cat: test_certs/15-cert.pem: No such file or directory
11.csr: No such file or directory
5647068520160:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('11.csr', 'r')
5647068520160:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
mkcerts.sh[48]: xxd: not found
hexdump: 11.der: No such file or directory
Error parsing structure
12375821279264:error:0DFFF07B:asn1 encoding routines:CRYPTO_internal:header too long:/usr/src/lib/libcrypto/asn1/asn1_lib.c:152:
12375821279264:error:0DFFF066:asn1 encoding routines:CRYPTO_internal:bad object header:/usr/src/lib/libcrypto/asn1/tasn_dec.c:1113:
12375821279264:error:0DFFF03A:asn1 encoding routines:CRYPTO_internal:nested asn1 error:/usr/src/lib/libcrypto/asn1/tasn_dec.c:683:
Error opening key file test_certs/11-key.pem
10802722701600:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('test_certs/11-key.pem', 'r')
10802722701600:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
unable to load key file
head: unknown option -- c
usage: head [-count | -n count] [file ...]
unable to load X509 request
28625781432992:error:0DFFF07B:asn1 encoding routines:CRYPTO_internal:header too long:/usr/src/lib/libcrypto/asn1/asn1_lib.c:152:
Using configuration from 11.cnf
11.csr: No such file or directory
31537261335648:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('11.csr', 'r')
31537261335648:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
mkcerts.sh[78]: seq: not found
mkcerts.sh[84]: seq: not found
Using configuration from 24.cnf
Error opening test_certs/24-cert.pem test_certs/24-cert.pem
4679530772704:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('test_certs/24-cert.pem', 'r')
4679530772704:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
unable to load certificate
mkcerts.sh[106]: seq: not found
Using configuration from 27.cnf
Error opening test_certs/27_CA/cacert.pem test_certs/27_CA/cacert.pem
5138423779232:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('test_certs/27_CA/cacert.pem', 'r')
5138423779232:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
unable to load certificate
Using configuration from 26_tail.cnf
Error opening CA private key ./test_certs/26_CA/private/cakey.pem
2418909553120:error:02FFF002:system library:func(4095):No such file or directory:/usr/src/lib/libcrypto/bio/bss_file.c:255:fopen('./test_certs/26_CA/private/cakey.pem', 'r')
2418909553120:error:20FFF002:BIO routines:CRYPTO_internal:system lib:/usr/src/lib/libcrypto/bio/bss_file.c:257:
unable to load CA private key
Using configuration from 23.cnf
mkcerts.sh[120]: seq: not found
Generating a 2048 bit RSA private key
.........+++
.......................+++
writing new private key to 'test_certs/server-key.pem'
-----
Using configuration from server.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4097 (0x1001)
Validity
Not Before: Jul 24 14:42:59 2017 GMT
Not After : Jul 24 14:42:59 2018 GMT
Subject:
countryName = UK
stateOrProvinceName = Scotland
localityName = Edinburgh
organizationName = SWI-Prolog
commonName = localhost
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
2C:F2:E9:67:A5:41:3E:34:32:17:FE:F2:8F:6B:7B:4D:4D:0B:6D:A5
X509v3 Authority Key Identifier:
keyid:95:29:B0:EF:A3:EE:22:AE:94:F1:47:D5:32:3B:37:51:DD:07:41:07
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Certificate is to be certified until Jul 24 14:42:59 2018 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
Generating a 2048 bit RSA private key
.+++
...........+++
writing new private key to 'test_certs/client-key.pem'
-----
Using configuration from client.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4098 (0x1002)
Validity
Not Before: Jul 24 14:42:59 2017 GMT
Not After : Jul 24 14:42:59 2018 GMT
Subject:
countryName = NL
stateOrProvinceName = Noord-Holland
localityName = Amsterdam
organizationName = SWI-Prolog
commonName = J. Wielemaker
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
D2:1D:75:57:5D:8F:C0:71:7E:3F:E9:B9:A1:6C:16:FA:23:B0:64:65
X509v3 Authority Key Identifier:
keyid:95:29:B0:EF:A3:EE:22:AE:94:F1:47:D5:32:3B:37:51:DD:07:41:07
X509v3 Key Usage:
Digital Signature, Non Repudiation, Key Encipherment
Certificate is to be certified until Jul 24 14:42:59 2018 GMT (365 days)
Write out database with 1 new entries
Data Base Updated
../swipl.sh -q -f test_ssl.pl -F none -g test_ssl,halt -t 'halt(1)'
.......
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:219:
test server: received error: Socket error: Can't assign requested address
.....
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:81:
test readme: received error: SSL(14007086) CONNECT_CR_CERT: certificate verify failed
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:540:
test Valid certificate, correct hostname in CN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [verified,verified]:true
ERROR: Got: []:error(error(socket_error('Operation timed out'),A))
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:543:
test Valid certificate, correct hostname in SAN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [verified,verified]:true
ERROR: Got: []:error(error(socket_error('Operation timed out'),A))
...
Any thoughts on this, before I dive deeper?
This still seems to be a shell problem, in particular please check out this message:
mkcerts.sh[37]: seq: not found
I have added a shell function to mkcerts.sh
to avoid dependency on seq
.
We should investigate the necessary conditions and replace the version tests with feature tests in configure. I don't think libressl compatibility is a real must right now, but it is nice to have. Here is a list of systems that support it. Notably Alpine Linux is an interesting Docker and embedded system target.
Thanks Jan. And yes, feature checks are the proper way (tm).
I've locally patched my port to use either use jot(1)
(a bit like seq(1)
), or in cases with only 2 or 3 reps, just hard-coded the cert numbers:
https://github.com/jasperla/openbsd-wip/blob/master/lang/swi-prolog/patches/patch-packages_ssl_tests_mkcerts_sh
Now the tests at least run, but:
wilfred:ssl> gmake check
../swipl.sh -q -f test_ssl.pl -F none -g test_ssl,halt -t 'halt(1)'
.......
.
.....
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:81:
test readme: received error: SSL(14007086) CONNECT_CR_CERT: certificate verify failed
..
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:546:
test Valid certificate, incorrect hostname in CN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:549:
test Valid certificate, incorrect hostname in SAN and CN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
.
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:555:
test Valid certificate, incorrect wildcard hostname in SAN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:558:
test Valid certificate, wildcard hostname in SAN with wildcard too high, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:561:
test Valid certificate, wildcard hostname in SAN with wildcard too low, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
.
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:567:
test Valid certificate, illegal wildcard hostname in CN, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
ERROR: /usr/ports/pobj/swi-prolog-7.4.2/swipl-7.4.2/packages/ssl/test_ssl.pl:570:
test Hostname containing embedded NULL, signed by trusted CA: wrong answer (compared using ==)
ERROR: Expected: [hostname_mismatch,verified,verified]:true
ERROR: Got: [verified,verified]:true
.....................
.
gmake: *** [Makefile:117: check] Error 1
This is worrying. It seems that we are verifying certs which should be invalid! Any ideas?
Sorry about the slow turnaround with this -- life gets in the way.
Ah, ah. I've seen this kind of thing before. I think my home DSL router is likely causing the failures.
I'm going in to the University tomorrow, so I can try the tests there. Watch this space.
No, sadly my theory was wrong. These appear to be real failures.
To narrow down the cause of this issue, could you please try the following patch:
diff --git a/ssl4pl.c b/ssl4pl.c index a097942..2102c10 100644 --- a/ssl4pl.c +++ b/ssl4pl.c @@ -35,6 +35,7 @@ */ #include <config.h> +#undef HAVE_X509_CHECK_HOST #include <SWI-Stream.h> #include <SWI-Prolog.h> #include <assert.h>
This enables our custom implementation of hostname verification, including calling the callback if the verification fails.
Using the above we get a little further, but then it gets worrying. As we see below, a normal SSL query fails.
ERROR: /home/jan/src/swipl-devel/packages/ssl/test_ssl.pl:81:
test readme: received error: SSL(14007086) CONNECT_CR_CERT: certificate verify failed
We also get
?- [library(http/http_open)].
true.
?- http_open('https://eu.swi-prolog.org/', In, []).
ERROR: url `'https://eu.swi-prolog.org/'' does not exist (status(400,Bad Request))
ERROR: In:
ERROR: [11] throw(error(existence_error(url,'https://eu.swi-prolog.org/'),context(_4422,...)))
The test version of the ssl package is in a branch openbsd
. I plan to dig further, but not right now.
I think this is now fixed, right Jan?
On the swi-devel master branch:
$ gmake check
../swipl.sh -q -f test_ssl.pl -F none -g test_ssl -t halt
.......
.
.....
.
................................
.
:)
We have only "fixed" this by falling back to our internal implementation, which was originally only added for backwards compatibility with OpenSSL versions less than 1.0.2.
The fact that this was necessary indicates that LibreSSL behaves differently than recent OpenSSL versions in a core aspect of the library (host name validation!) and may in fact be a mistake in LibreSSL! In short, a verification callback seems to not be invoked when hostname validation fails if the library code is used for this purpose. I have self-assigned this issue in https://github.com/SWI-Prolog/packages-ssl/issues/115 because I want to look further into it when I get the chance. However, I would greatly appreciate any help I can get, if you want to look into this too.
I consider the immediate issue (port to OpenBSD) resolved to the extent that the library is usable on this platform, and suggest to take further discussion to the ssl issue I have filed for this, if you are interested.
OK.
Note that there were some other changes: https://github.com/SWI-Prolog/packages-ssl/commits/master
Namely picking up the root certs from the right path.
Namely picking up the root certs from the right path.
Yes. Work should start from the current master. Just to avoid problems I deleted the openbsd
branch on the server.
With the latest release of swi-prolog on OpenBSD-current:
It looks like the problematic code is:
LibreSSL does not define
CRYPTO_ONCE
and friends.What would be a suitable workaround?
Cheers
P.S. There is a dead link to a bugzilla on the SWI web site here: http://www.swi-prolog.org/Support.html