Provide an example of a tls client

[zoggy]

Hello,

It would be great to have an example of a client (not a server) connecting to a server and authenticating with certificates.

This is partly related to https://github.com/mirage/ocaml-cohttp/issues/471 .

cc @hannesm

[hannesm]

while I won't have time for this, here are some pointers (first, conduit would need support for certificates):

• https://github.com/mirage/ocaml-conduit/issues/7
• https://github.com/mirage/ocaml-conduit/issues/31

(or am I misguided and is there a cohttp API not involving conduit to do HTTP client connections? if so, you could use tls_lwt).

A simple example using OCaml-TLS and client certificates is available at https://github.com/mirleft/ocaml-tls/blob/master/lwt/examples/echo_client.ml -- just read private key and certificate chain (line 14-16) and pass them to the client config (line 19).

[zoggy]

Thanks for your answer. Indeed I found no way in cohttp API to plug TLS connection.

Maybe a simple way would be to add a connected_call function to Cohttp_lwt.Make_client which would be like call but would take (ic, oc) in parameter, rather than ctx and uri ?

[rgrinberg]

That definitely sounds like the way forward. I'd like to make cohttp usable without conduit anyway.

[zoggy]

I'll try to set up an example with such a change.

[zoggy]

@hannesm I'm starting from the https://github.com/mirleft/ocaml-tls/blob/master/lwt/examples/echo_client.ml example but I'm getting tls errors:

 (record-in (((content_type ALERT) (version (Supported TLS_1_2))) "\002("))

(alert-in (FATAL HANDSHAKE_FAILURE))

(record-out (ALERT "\001\000"))

(ok-alert-out HANDSHAKE_FAILURE)

TLS ALERT (remote end): HANDSHAKE_FAILURE
Fatal error: exception Tls_lwt.Tls_alert(6)

Since I don't known which files to use as server certificates, I commented out some code:

 ...
   | Some f      -> `Ca_file f) >>= fun authenticator ->
  (*X509_lwt.private_of_pems
    ~cert:server_cert
    ~priv_key:server_key >>= fun certificate ->*)
  Tls_lwt.connect_ext
    ~trace:eprint_sexp
    Tls.Config.(client ~authenticator (*~certificates:(`Single certificate)*) ~ciphers:Ciphers.supported ())
    (host, port) >>= fun (ic, oc) ->
...

but this does not solve the problem.

Are these server certificates required ?

[hannesm]

not sure what your other endpoint is. the client uses server_cert, defined in ex_common.ml in the same directory, which read the (nowadays invalid ./certificates/server.pem -- assuming your cwd is a ocaml-tls git checkout).

you can generate your own certificate and private key and either overwrite server.pem and server.key or change the server_cert / server_key in echo_client.ml with full paths to your custom ones.

[zoggy]

But are these server certificates required ? By now I just use

Tls_lwt.connect ~trace:eprint_sexp authenticator (host, port) >>= fun (ic, oc) ->

so no server certificate is involved. My problem seems to be that the PEM file I use as authentication certificate is not valid/recognized. This PEM file is exported from firefox, but as you may have guessed already I'm not a TLS specialist and I wonder if this is a correct certificate to use.

[hannesm]

I've no clue about your scenario, and your paste doesn't include the actual error.

Depending on your scenario, you need on the client:

• the set of trusted CA certificates (since you want to verify that the server you're talking to is good) -- this is configured via authenticator https://github.com/mirleft/ocaml-tls/blob/master/lwt/examples/echo_client.ml#L9-L13, depending on your command line arguments it might take all the certificates from https://github.com/mirleft/ocaml-tls/tree/master/certificates (you might want to specify one explicitly on the command line)
• your client certificate and key -- this is currently hardcoded to server.key and server.pem

if you provide only an authenticator (as mentionde above), there won't be any client certificate involved. You'll have to use Tls_lwt.connect_ext (and provide a Tls.Config.client manually).

[zoggy]

From what you say, the certificates/server* files are used to authenticate the client. I thought they were used to authenticate the server :-) And I thought the authenticator was used to authenticate the client, not the server... Thanks for the explanation.

Now when I use the server.* files from https://github.com/mirleft/ocaml-tls/tree/master/certificates, it seems that they are validated. Thanks, I will now struggle with openssl to create these server.key and server.pem files from my certificate stored in firefox... The aim is be able to retrieve some of my data using

 ./cohttp_lwt_client zoggy.databox.me 443

[zoggy]

In fact https://databox.me seems to be the only host for which the connection fails with these errors:

(record-in (((content_type ALERT) (version (Supported TLS_1_2))) "\002("))

(alert-in (FATAL HANDSHAKE_FAILURE))

(record-out (ALERT "\001\000"))

(ok-alert-out HANDSHAKE_FAILURE)

TLS ALERT (remote end): HANDSHAKE_FAILURE
Fatal error: exception Tls_lwt.Tls_alert(6)

Don't know why but it's no luck as it is the one I'm interested in.

[hannesm]

could you provide a bit more output please? the lines you pasted do not include the actual problem... otoh I can try to connect myself to databox and debug... maybe tomorrow..

[zoggy]

These lines are the only output I have :-/ Is there a way to get more ? I will post the code somewhere tomorrow if you need it.

[zoggy]

@rgrinberg The solution to add a connected_call function to Cohttp_lwt_s.Client was not a great idea, because other modules use this module signature, and some cannot have such a function in their interface (XHR modules, for example).

But exploring the Cohttp code, I came with another solution which does not require modifying Cohttp. I define a new Net_tls module, with same interface as Cohttp_lwt_unix_net. I also need a IO module not using Conduit:

module IO =
  struct
    type 'a t = 'a Lwt.t
    let (>>=) = Lwt.bind
    let return = Lwt.return

    type ic = Lwt_io.input_channel
    type oc = Lwt_io.output_channel
    type conn = ic * oc

    let read_line ic = Lwt_io.read_line_opt ic
    let read ic count = Lwt_io.read ~count ic
    let write oc buf = Lwt_io.write oc buf
    let flush oc = Lwt_io.flush oc
  end

module Tls_net =
  struct
    module IO = IO
    type ctx = Tls.Config.client Lwt.t
    let sexp_of_ctx _ = Sexplib.Sexp.Atom "tls ctx"
    let default_ctx =
      X509_lwt.authenticator `No_authentication_I'M_STUPID >>=
      fun authenticator ->
          Lwt.return
            (Tls.Config.(client
              ~authenticator ~ciphers:Ciphers.supported ()))

    let connect_uri ~ctx uri =
      let host = match Uri.host uri with None -> "" | Some s -> s in
      let port = match Uri.port uri with None -> 443 | Some n -> n in
      ctx >>= fun client ->
      Tls_lwt.connect_ext
        ~trace:eprint_sexp
        client (host, port)
        >>= fun (ic, oc) -> Lwt.return ((ic, oc), ic, oc)

    let close c = Lwt.catch (fun () -> Lwt_io.close c) (fun _ -> return_unit)
    let close_in ic = ignore_result (close ic)
    let close_out oc = ignore_result (close oc)
    let close ic oc = ignore_result (close ic >>= fun () -> close oc)
end

Type ctx uses Lwt.t because default_ctx require a default authenticator which cannot be obtained without Lwt. @hannesm Would it be possible to have a value like default_authenticator to avoid this ?

Then I get a Client module:

module Client = Cohttp_lwt.Make_client (IO) (Tls_net)

I can then define my own function call which uses the Client.call function provided by Cohttp:

let call meth ?ca ?body ?chunked ?headers iri =
  let%lwt authenticator = X509_lwt.authenticator
    (match ca with
     | None        -> `Ca_dir server_cert_dir
     | Some "NONE" -> `No_authentication_I'M_STUPID
     | Some f      -> `Ca_file f)
  in
  let%lwt certificate =
    X509_lwt.private_of_pems ~cert:server_cert ~priv_key:server_key
  in
  let ctx = Lwt.return
    (Tls.Config.client
     ~authenticator ~certificates:(`Single certificate)
       ~ciphers:Tls.Config.Ciphers.supported ()
    )
  in
  Client.call ~ctx ?body ?chunked ?headers meth
    (Uri.of_string (Iri.to_uri iri))

let get = call `GET
let delete = call `DELETE
let post = call `POST
let put = call `PUT
let patch = call `PATCH

This function can be defined differently according to your application context: here each connection creates a new context (i.e. a Tls.Config.client) by looking at the ?ca argument and certificate on disk.

Finally, I could tls-authenticate successfully to my account on https://rww.io to create a ressource by a POST request :-)

[zoggy]

Regarding the default authenticator, now I use X509.Authenticator.null so that type ctx can be defined without Lwt.t:

type ctx = Tls.Config.client [@@deriving sexp_of]
let default_ctx =
   let authenticator = X509.Authenticator.null in
   Tls.Config.(client ~authenticator ~ciphers:Ciphers.supported ())

[hannesm]

@zoggy both are not good ideas: the null authenticator always returns true, don't do that. both chain_of_trust (which takes a time and a list of trusted CA certificates) and server_key_fingerprint (look here) are not inside of Lwt.t (but those which read files etc. over here are inside of Lwt.t).

you shouldn't pass ~ciphers to client unless you are really sure what you are doing. the default list of ciphers in OCaml-TLS is well curated.

EDIT: and there is no such thing as a default authenticator: depending on your application you have to choose which strategy to use (and which CA certificates are trustworthy).

[zoggy]

Ok, thanks. So I could use a default authenticator built with chain_of_trust [] so that it will always fail, forcing the developer to specify a context (a default context is required to conform to Cohttp_lwt_unix_net interface).

Ok for the ciphers, I had just copy-pasted from your example.

[hannesm]

oh, thanks... I just removed the ~ciphers from our echo_client example.

providing a default context with an empty set of trust anchors in chain_of_trust sounds sensible for now

[zoggy]

Did you try your echo example on https://databox.me ?

[hannesm]

@zoggy I just tried echo_client, and https://www.ssllabs.com/ssltest/analyze.html?d=databox.me tells me that they only support 3 ciphersuites, all using ECDHE for the key exchange (and unfortunately neither nocrypto nor OCaml-TLS has EC support at the moment, it is planned)

[talex5]

I spent a bit of time last week investigating how to use cohttp with https in 0install. Here are my notes on how to get it working using cohttp_lwt_unix.

First, you must not use ocaml-tls in the default configuration, because conduit disables certificate validation in this mode and you cannot override it. See https://github.com/mirage/ocaml-conduit/blob/2aa5d06fc81cd74dc56553cd490a9c23cb538680/lwt-unix/conduit_lwt_tls_real.ml#L33

To avoid this, call Cohttp_lwt.Make_client with your own Net implementation that overrides connect_uri with one that forces the use of Conduit_lwt_unix_ssl, or invokes ocaml-tls itself.

By default, openssl uses CA certificate paths hard-coded at compile time. As these are different on every Linux system, if you want portable binaries then you will need to search for the right paths yourself. Here's the code I use for that:

https://github.com/0install/0install/blob/6c0f5c51bc099370a367102e48723a42cd352b3b/ocaml/zeroinstall/http.cohttp.ml#L4-L66

To use your custom context with the correct CAs you'll need to avoid Conduit_lwt_unix and go directly to Conduit_lwt_unix_ssl. Unfortunately, there is no way to turn the result of this into a Conduit_lwt_unix.flow as required by cohttp (since the flow type is private). You will hit the same problem if you used ocaml-tls directly.

Luckily, cohttp doesn't actually use the flow for anything, so you can override the IO module with your own implementation that doesn't use it. See https://github.com/0install/0install/blob/6c0f5c51bc099370a367102e48723a42cd352b3b/ocaml/zeroinstall/http.cohttp.ml#L68-L111 for some suitable Net and IO modules.

[mseri]

@samoht did your change to use ca-cert somewhat solve this issue?