Finding TLS/SSL libraries and certificates

[lassik]

A HTTPS connection can fail with a -40 error:

> (http-get "repology.org" "/" (make-keyword "secure") #t)
*** ERROR: TLS handshake failed: -40*** ERROR: TLS handshake failed: -40

This comment may be relevant:

Okay, I figured out the cause of the problem. For certain web servers which have more than 1 hostname, the client has to tell the server the exact hostname the client is trying to connect to, so that the web server can present the right SSL certificate having the hostname the client is expecting. In case of blocking SSL, the client should use "SSL_set_tlsext_host_name(SSL ssl, char hostname)" function to set the server's hostname.

[shirok]

Hmm, axTLS does set SSL extension host name, so it doesn't seem to be a direct cause.

-40 error code is also generated with NO_CIPHER, meaning the available ciphers the server proposed aren't supported by the axTLS. I think it's a more likely cause. Did you try mbedtls?

[shirok]

mbedtls worked.

(use rfc.tls)
(default-tls-class <mbed-tls>)
(tls-ca-bundle-path "/etc/ssl/certs/ca-certificates.crt")
(http-get "repology.org" "/" :secure #t)

You need to set up an appropriate ca-bundle path if you didn't set the default path at configure time. We need to address that eventually so that it will "just work". See https://github.com/shirok/Gauche/issues/365

[shirok]

With HEAD, you can say (tls-ca-bundle-path 'system) instead. We'd like to make it default, but <ax-tls> has issues with default ca-bundle.

[lassik]

Sorry about the delay with this.

With HEAD on a mac, I get:

$ gosh
gosh> (use rfc.tls)
gosh> (use rfc.http)
gosh> (tls-ca-bundle-path 'system)
#f
gosh> (http-get "repology.org" "/" (make-keyword "secure") #t)
file '/etc/ssl/certs/ca-certificates.crt' does not exist
invalid digest: 2a 86 48 ce 3d 04 03 03
Error: Invalid X509 ASN.1 file (Unsupported digest)
invalid digest: 2a 86 48 ce 3d 04 03 02
Error: Invalid X509 ASN.1 file (Unsupported digest)
.
.
.
invalid digest: 2a 86 48 ce 3d 04 03 03
Error: Invalid X509 ASN.1 file (Unsupported digest)
*** ERROR: TLS handshake failed. Possibly no supported ciphers. (code=-40)

I have a /usr/local/etc/openssl/cert.pem file, but it's not clear from the above output whether Gauche is reading it. In ext/tls/load_system_cert.c function system_cert_loader() the first file in the search order is /etc/ssl/certs/ca-certificates.crt (which doesn't exist on my machine). The above output shows an error about that. Could it be that it stops before reading the other file (which exists on my computer)?

./configure shows that Gauche is using AxTLS as the backend.

[shirok]

Could you try mbedTLS? I get the same result as yours with axtls, and the cause is that repology.org uses ciphers axtls doesn't support. The messages (invalid digest etc.) are from axtls and you can ignore it for now (it merely meas some certs are not supported in axtls and skipped.)

mbedTLS supports more ciphers and digests. The only issue we don't make it default is that it is Apache license and, although BSD and Apache are compatible, the resulting binary would satisfy two licenses and it'll make things a bit complicated. But if AxTLS won't be updated, we might need to consider switching.

[lassik]

With mbedtls:

$ gosh
gosh> (use rfc.tls)
gosh> (use rfc.http)
gosh> (tls-ca-bundle-path 'system)
#f
gosh> (tls-ca-bundle-path)
system
gosh> (http-get "repology.org" "/" (make-keyword "secure") #t)
*** ERROR: unbound variable: tls-ca-bundle-path
Stack Trace:
_______________________________________
  0  (tls-ca-bundle-path)
        at "/Users/lassi/.local/share/gauche-0.97/0.9.10_pre2/lib/rfc/http.scm":745
  1  (tls-ca-bundle-path)
        at "/Users/lassi/.local/share/gauche-0.97/0.9.10_pre2/lib/rfc/http.scm":745
  2  (make-tls :server-name (~ conn 'server) :options (if (tls-ca- ...
        at "/Users/lassi/.local/share/gauche-0.97/0.9.10_pre2/lib/rfc/http.scm":744
  3  (start-connection conn)
        at "/Users/lassi/.local/share/gauche-0.97/0.9.10_pre2/lib/rfc/http.scm":757

[lassik]

Wait...

gosh> (default-tls-class)
#<class <ax-tls>>

[shirok]

You have to call (default-tls-class <mbed-tls>).

[lassik]

I didn't realize mbedtls needs to be installed separately and isn't bundled like axtls. Installing it, configure now detects it:

checking mbedtls/net_sockets.h usability... yes
checking mbedtls/net_sockets.h presence... yes
checking for mbedtls/net_sockets.h... yes
checking for library containing mbedtls_ctr_drbg_init... -lmbedcrypto
checking for library containing mbedtls_x509_crt_init... -lmbedx509
checking for library containing mbedtls_ssl_init... -lmbedtls
checking for openssl... openssl

This Gauche has been configured with the following parameters:
           version: 0.9.10_pre2
         multibyte: utf8
     documentation: Yes
              slib: /usr/local/slib
            thread: pthreads
           tls/ssl: mbedtls 
          CA store: none 
  optional modules: odbm ndbm gdbm zlib

But there's a build error:

gcc -DHAVE_CONFIG_H -I. -I. -I../../src -I../../src -I../../gc/include   -no-cpp-precomp  -g -O2 -Wall -Wextra -Wno-unused-label -no-cpp-precomp -fPIC -fno-common  -fomit-frame-pointer  -o tls-mbed.o -c tls-mbed.c
In file included from tls-mbed.c:84:
./load_system_cert.c:59:19: error: use of undeclared identifier 'SSL_OK'
        if (st == SSL_OK) return SCM_TRUE;

[lassik]

Is SSL_OK an axtls constant? MbedTls seems to use negative numbers for error returns.

[shirok]

Ah, right. I didn't catch it because I always compile with both axTLS and mbedTLS.

[lassik]

OK, with that commit everything now works fine! Tried with ./configure --with-tls=mbedtls which produces a Gauche with mbedtls only, no axtls. I had some build problems but they were solved by deleting all the files untracked by git.

This leads to the question whether axtls is still a relevant library. Many websites and browsers like to upgrade to recent TLS versions on a quite aggressive schedule, disabling support for old ciphers. If axtls doesn't keep up with the latest ciphers, it may be of limited use on today's internet.

[lassik]

Just made a pre-SRFI about TLS clients. https://github.com/pre-srfi/tls-client Comments and suggestions are welcome. It will probably take months to be posted as a real SRFI, since there are so many draft SRFIs already.

[shirok]

Yeah, users expect things just work, and don't want to bother to track down certificates, ciphers, etc...

If you have mbedtls installed in standard place, Gauche's configure automatically detects it and build with both axtls and mbedtls, with axtls as default. If we establish a pretty reliable way to get CA cert paths, I think it's reasonable to make mbedtls default if it is available. (axtls has an option to defer server cert check, allowing to connect without CA certs. mbedtls doesn't.)

[lassik]

Yeah, users expect things just work, and don't want to bother to track down certificates, ciphers, etc...

Agreed.

If you have mbedtls installed in standard place, Gauche's configure automatically detects it

This is good.

and build with both axtls and mbedtls, with axtls as default. If we establish a pretty reliable way to get CA cert paths, I think it's reasonable to make mbedtls default if it is available. (axtls has an option to defer server cert check, allowing to connect without CA certs. mbedtls doesn't.)

Does AxTLS implement enough ciphers that it continues to be useful?

[shirok]

Does AxTLS implement enough ciphers that it continues to be useful?

No, it occasionally fails on some sites. The only advantage of AxTLS is that I can include it in Gauche source distribution with BSDL, and having it guarantees Gauche to have some support of TLS out of the box.

[lassik]

True, small easy-to-include libraries are very nice. It's unfortunate that the development pace of TLS is so fast that smaller implementations struggle to keep up.

We started an issue in the pre-SRFI's repo (https://github.com/pre-srfi/tls-client/issues/3) about where to find certificates. If we do a survey, maybe we'll find a way to have good-enough defaults for everyone.

[shirok]

Latest commit checks availability of several well-known ca-bundle paths, and if it finds any, it favors <mbed-tls>. So, as far as you build Gauche on a system where mbedTLS is installed, you'll probably get it just work.

[lassik]

The configure script should probably show a warning if it uses axTLS as a fallback, since axTLS supports less ciphers than people expect.

Mbed TLS is widely available (https://repology.org/project/mbedtls/versions) though there seem to be quite a few vulnerable versions still around.

Can Gauche use OpenSSL/LibreSSL?

[shirok]

OpenSSL used to use 4-clause BSDL that put extra restriction in the license terms, and I believe LibreSSL still uses it. I want to avoid that.

OpenSSL v3 became Apache License v2, which is the same as MbedTLS, and also quite compatible to BSDL, so it is an option. Actually I had a prospect to support OpenSSL as well. As an external library we could write OpenSSL layer in the similar way as MbedTLS layer.

[lassik]

OpenSSL used to use 4-clause BSDL that put extra restriction in the license terms, and I believe LibreSSL still uses it. I want to avoid that.

Is that the infamous "advertising clause"?

OpenSSL v3 became Apache License v2, which is the same as MbedTLS, and also quite compatible to BSDL, so it is an option. Actually I had a prospect to support OpenSSL as well. As an external library we could write OpenSSL layer in the similar way as MbedTLS layer.

Sounds good! OpenSSL is probably the most widely installed library on *nix platforms? GnuTLS is probably quite popular as well.

[lassik]

@shirok I sent a PR to Homebrew to enable mbedtls for their Gauche recipe: https://github.com/Homebrew/homebrew-core/pull/62857

[APIPLM]

@shirok as I run (default-tls-class <mbed-tls>), the following message come out in docker with the image practicalscheme/gauche:latest

*** ERROR: unbound variable: <mbed-tls>
Stack Trace:
_______________________________________
  0  <mbed-tls>
        [unknown location]
  1  (eval expr env)
        at "/usr/share/gauche-0.97/0.9.9/lib/gauche/interactive.scm":269

Does that mean the image just was build with only axTLS implementation? Or that was build with both axTLS and MbedTLS, then there is an option, that we can set MbedTLS available for us.

[lassik]

$ docker run practicalscheme/gauche:latest gauche-config --reconfigure
./configure  '--prefix=/usr' '--enable-multibyte=utf-8'

shows that it's not built with mbedtls support.

[lassik]

Also this finds no mbedtls files: docker run practicalscheme/gauche:latest find / -iname "*m*tls*"

Does the configure script in git master auto-detect mbedtls?

Could we add something to gosh -V output to show which TLS it's built with?

[shirok]

Does that mean the image just was build with only axTLS implementation? Or that was build with both axTLS and MbedTLS, then there is an option, that we can set MbedTLS available for us.

@APIPLM @lassik The docker image doesn't include mbedtls. If mbedtls were installed before configure, it would've been autodetected.

If the docker image is where you're based on, I can rebuild it including mbedtls. It's licensing concern but I think Apache License won't cause any issue.

[lassik]

FWIW, I just added mbedtls to the docker run -it schemers/gauche:head container.

[APIPLM]

@shirok Seem like the whole process of building the docker image practicalscheme/gauche in the docker file does not have the configuration for the implementation mbedtls or axTLS. I guest that it is that configuration in the base image ADD file:308bd6a55b052f212f1259ebcc844a5ede95c27c7f61528e11513cc9bfb74e6d in /. Are you guys trying to build the new base image? But the image do have the script get-gauche.sh to update the patch.

[shirok]

@APIPLM I'm testing the updated Dockerfile including mbedtls now. (AxTLS is included in Gauche source and built-in by default). Once I see it works I'll push it.

[shirok]

It seems easier to work on the newest version, rather than modifying 0.9.9 docker build, for the CA-bundle handling is also extended in the recent commits. I'm almost ready to release 0.9.10... can you guys wait for it and the new 0.9.10 Docker image?

[lassik]

If 0.9.10 is right around the corner, we should wait for that release before enabling mbedtls in Homebrew as well. Then we'll avoid an unnecessary extra revision of the Homebrew 0.9.9 package.

[shirok]

I think this discussion is past enough to be closed. Supporting OpenSSL is still a viable option; open it as a new issue when demand comes up.

[karme]

@shirok does the original request work for you?

For me it still fails:

gosh$ (gauche-version)
"0.9.15_rc1"
gosh$ (use rfc.http)
gosh$ (http-get "repology.org" "/" :secure #t)
*** ERROR: TLS handshake failed: SSL - A fatal alert message was received from our peer (-30592)
Stack Trace:
_______________________________________
  0  (report-error e)
  1  (%tls-connect tls host p proto)
  2  (connect-socket)
        at "/usr/local/share/gauche-0.98/0.9.15_rc1/lib/rfc/http.scm":748
  3  (start-connection conn)
        at "/usr/local/share/gauche-0.98/0.9.15_rc1/lib/rfc/http.scm":753
  4  thunk
  5  (with-connection conn (^ (i o) (request-response i o method u ...
        at "/usr/local/share/gauche-0.98/0.9.15_rc1/lib/rfc/http.scm":303
  6  (eval expr env)
        at "/usr/local/share/gauche-0.98/0.9.15_rc1/lib/gauche/interactive.scm":359
  7  (evaluator exp (vm-current-module))
  8  (with-error-handler (^e (report-error e) #t) (^ () (let loop2 ...

(sid-amd64-sbuild)karme@amalthea:/tmp$ dpkg -l "*mbed*"|grep ^i
ii  libmbedcrypto7t64:amd64 2.28.8-1     amd64        lightweight crypto and SSL/TLS library - crypto library
ii  libmbedtls-dev:amd64    2.28.8-1     amd64        lightweight crypto and SSL/TLS library - development files
ii  libmbedtls14t64:amd64   2.28.8-1     amd64        lightweight crypto and SSL/TLS library - tls library
ii  libmbedx509-1t64:amd64  2.28.8-1     amd64        lightweight crypto and SSL/TLS library - x509 certificate library

maybe my mbedtls is too old?

[shirok]

Aah, right. Got distracted by discussions.

[shirok]

Hmm... MbedTLS 3.6.0 has the same issue. We do use mbedtls_ssl_set_hostname now, and indeed the error message is different.

[shirok]

Maybe this. https://forums.mbed.com/t/error-0x7780-during-handshake/6883