Usage examples?

[hugusmaximus]

I wonder if it would be possible to have some documentation about usage examples (beyond the "tlstunnel --help"). This software looks wonderful. Would be nice to have some doc on how to use it. Even (general public) early adopters need some help.... :-)

[hannesm]

Dear @hugusmaximus, thanks for your interest.

Does https://github.com/robur-coop/tlstunnel/pull/16 satisfy your needs, or is there something unclear / missing? Indeed, this project is vastly underdocumented, and currently build around a specific use-case (where we use a wildcard certificate from let's encrypt, and store that certificate in DNS).

If you have a different use-case / a favourite way to pass your X.509 certificate, please let us know. We could also revise/adapt this project to e.g. provision and store the X.509 certificate directly with let's encrypt (but we'll need a DNS server with dynamic updates since wildcard certificates are only available via the DNS challenge).

[hugusmaximus]

Definitely yes, this satisfy my needs. Thanks a lot! Anyway, I would add few more bytes of text explaining what "solo5-hvt" is (I had to google it...) may be with a link to this "hypervisor middleware ??". Also, I don't have "tlstunnel.hvt" in my "unikernel/dist" directory, I just have "tlstunnel", so I guess this should be the target build for KVM..? And last (but not least important) regarding the "once tlstunnel managed to get a certificate via DNS"... which is, sorry for my ignorance, totally new to me (I'm used to deploy certs manually, was not aware it was possible to do it via DNS...) and looks like I need to use "dns-primary-git" and/or "dns-letsencrypt-secondary"... my question is: is it possible to deploy certificate without using those dns? (maybe loading it from file). Sorry for all the questions and thanks for the lighting fast answer.

[hannesm]

Definitely yes, this satisfy my needs. Thanks a lot! Anyway, I would add few more bytes of text explaining what "solo5-hvt" is (I had to google it...) may be with a link to this "hypervisor middleware ??".

I think this is slightly deeper, to avoid re-iteration of documentation (how to deploy a MirageOS unikernel), we should centralise this somewhere (including network configuration, ...) -- i.e. finally get this operator documentation / book out.

Also, I don't have "tlstunnel.hvt" in my "unikernel/dist" directory, I just have "tlstunnel", so I guess this should be the target build for KVM..?

Yes, so the extension is indicating the target -- being it hvt (KVM), spt (seccomp), virtio, xen, or no ending (unix executable). From a security perspective, I still prefer KVM (due to attack surface).

And last (but not least important) regarding the "once tlstunnel managed to get a certificate via DNS"... which is, sorry for my ignorance, totally new to me (I'm used to deploy certs manually, was not aware it was possible to do it via DNS...) and looks like I need to use "dns-primary-git" and/or "dns-letsencrypt-secondary"... my question is: is it possible to deploy certificate without using those dns? (maybe loading it from file). Sorry for all the questions and thanks for the lighting fast answer.

A MirageOS unikernel does not have access to the file system (in the general case) -- and together with the goal of providing generic reporoducible builds (i.e. shipping the identical binary to all customers), we need some way to push the certificate to the unikernel. Using a file and including it in the unikernel binary is the old way, but contradicts the generic reproducible builds concept.

Another path could be to use a block device of the certificate chain & private key - maybe loosening the requirements to be sector-aligned (or pre-process the files appending comment markers). But then the update process (certificate renewal) will be slighlty cumbersome [something needs to update the files, and the unikernel needs to reload them]. But I think this is doable, and would be an easy way to load the certificates.

From my viewpoint, we developed DNS services and a let's encrypt client, and just use DNS as a key-value store with the certificates present. While this is suitable if all you want is MirageOS unikernels on your domain, it may be inconvenient for a migration from legacy systems to MirageOS unikernels. I think I understand your requirements, but will need some time to provide a suitable solution. Thanks for the discussion.

[hugusmaximus]

From a security perspective, I still prefer KVM (due to attack surface).

Sure, I'm just trying on a standard Linux box to test usability. My (mid-term) intention is to run it virtualized on top of seL4. Not sure if this is possible as have no idea if MirageOS is using GICv2 or GICv3 and if anyone ever tried this... but if it can run on seL4 it will run on seL4, which is formally verified (so attack surface is the smaller possible, moreover if seL4 runs on bare metal).

Using a file and including it in the unikernel binary is the old way

Ok, this is exactly was I was looking for, just embedding the certs at compile time...

but contradicts the generic reproducible builds concept

Yes, I understand your point of view, and it makes sense on environments where you want generic reproducible concept. But it makes this software dependent on "third party" solutions (such dns servers...). I like more the concept of just a binary with everything it requires built in, but of course, this is just a personal opinion and I get the value of reproducible builds.

But then the update process (certificate renewal) will be slighlty cumbersome [something needs to update the files, and the unikernel needs to reload them].

How much it takes this unikernel to boot...? If you use some block device maybe you can make it load the certs only at boot time. Whenever a reload of certs is required just reboot the unikernel. If you make it at compile time (my preferred option) then you just need to recompile the binary. To me this is not a problem, I'm from the old school and used to compile custom Linux Kernels on very old hardware... so an unikernel looks like an amazingly fast compilation to me and certs only change once a year and is a single server, not a distributed environment (where DNS option looks very clever).

I have no idea what will be best option, but I think the easier you make it possible, the wider audience you will reach. Still you will have the option of DNS deployment, but I bet many people would love to just load certificates in some trivial way, either at compile time either from a block device... whatever.

While this is suitable if all you want is MirageOS unikernels on your domain, it may be inconvenient for a migration from legacy systems to MirageOS unikernels.

That's it...

but will need some time to provide a suitable solution. Thanks for the discussion

My pleasure.

[hannesm]

Using a file and including it in the unikernel binary is the old way

Ok, this is exactly was I was looking for, just embedding the certs at compile time...

I created a branch named "mirage-kv" on this repository for exactly this use-case. You have to put your server.pem (certificate chain) and server.key (private key) in the unikernel/tls subdirectory before doing make build, and this will then be used. No dependency onto DNS etc.

Let me know if this is useful for you, or if you struggle with getting it to work. Thanks.

This is not merged into the main branch, that will take some more thoughts (enabling multiple use cases).

How much it takes this unikernel to boot...? If you use some block device maybe you can make it load the certs only at boot time. Whenever a reload of certs is required just reboot the unikernel.

It boots in milliseconds. The underlying issue are potentially active connections that are then dropped (in the current state of MirageOS without notifying the other side(s)).

[hugusmaximus]

Wow... thanks a lot! I'll give a try ASAP and will come back with feedback!

[hugusmaximus]

I tried to compile the new branch but I got:

hugo@anakin:~/tlstunnel/unikernel$ mirage configure -t unix
hugo@anakin:~/tlstunnel/unikernel$ mirage build
File "dune.build", line 16, characters 3-16:
16 |    mirage-kv-mem mirage-logs mirage-random mirage-runtime mirage-time
        ^^^^^^^^^^^^^
Error: Library "mirage-kv-mem" not found.
-> required by _build/default/main.exe
-> required by alias all
-> required by alias default
Generating static_tls.ml        
Generating static_tls.mli
hugo@anakin:~/tlstunnel/unikernel$   

[hannesm]

could you try, as written in the mirage docs, https://mirage.io/docs/hello-world, a "make depend" after your configure (or a "mirage configure" followed by "make")?

[hugusmaximus]

Ops... I'm really sorry. I was asleep for too long and missed the "make depend". Yes, it compiles perfectly. Anyway, now I'm having this error: "connect disk.img: failed to open file":

hugo@anakin:~/tlstunnel/unikernel$ sudo ./dist/tlstunnel --ipv4=10.0.5.10/24 --private-ipv4=10.0.5.3/24 --key=12345
2024-01-16T17:33:20Z: [ERROR] [mirage-block-unix] connect disk.img: failed to open file
2024-01-16T17:33:20Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
2024-01-16T17:33:20Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
Fatal error: exception Failure("connect disk.img: failed to open file")
Raised at Lwt.Miscellaneous.poll in file "duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29
Called from Unix_os__Main.run in file "duniverse/mirage-unix/lib/main.ml", line 5, characters 8-18
Called from Dune__exe__Main.run in file "main.ml", line 3, characters 12-30
Called from Dune__exe__Main in file "main.ml", line 187, characters 5-10

not sure what the program expects as I can't see any command line argument option to solve this nor in the usage example... may this issue be specific to the unix build?

[hannesm]

there's a default value for the configuration store, which is named "disk.img". A truncate -s 1m disk.img should get you ready.

[hugusmaximus]

Yes, now it doesn't complain anymore about "disk.img", but I get a bind() error:

hugo@anakin:~/tlstunnel/unikernel$ sudo ./dist/tlstunnel --ipv4=10.0.5.1/24 --private-ipv4=10.0.5.3/24 --key=12345
2024-01-16T18:20:03Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
2024-01-16T18:20:03Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
2024-01-16T18:20:03Z: [INFO] [application] read from 2024-01-16T18:12:50-00:00 (counter 0) 0 bytes data
2024-01-16T18:20:03Z: [INFO] [application] SNI map has 0 entries
Fatal error: exception Unix.Unix_error(Unix.EADDRNOTAVAIL, "bind", "")
Raised by primitive operation at Tcpv4v6_socket.listen.(fun) in file "duniverse/mirage-tcpip/src/stack-unix/tcpv4v6_socket.ml", line 149, characters 6-50
Called from Stdlib__List.iter in file "list.ml", line 112, characters 12-15
Called from Dune__exe__Unikernel.Main.start.(fun) in file "unikernel.ml", line 399, characters 4-151
Called from Lwt.Sequential_composition.bind.create_result_promise_and_callback_if_deferred.callback in file "duniverse/lwt/src/core/lwt.ml", line 1844, characters 16-19
Re-raised at Lwt.Miscellaneous.poll in file "duniverse/lwt/src/core/lwt.ml", line 3123, characters 20-29
Called from Unix_os__Main.run in file "duniverse/mirage-unix/lib/main.ml", line 5, characters 8-18
Called from Dune__exe__Main.run in file "main.ml", line 3, characters 12-30
Called from Dune__exe__Main in file "main.ml", line 187, characters 5-10

I manually checked I can bind to this address:

hugo@anakin:~/tlstunnel/unikernel$ sudo nc -l -v -p 443 -s 10.0.5.1
Listening on anakin 443
^C

I tried with different interfaces/IPs but I got the same error, so, I'm lost here...

[hannesm]

Thanks for testing, and sorry that you encounter these issues.

Since a sudo nc -l -v -p 443 -s 10.0.5.1 works fine for you, this means that IP address is configured on your machine. Does a sudo ./dist/tlstunnel --ipv4=10.0.5.1/24 --private-ipv4=10.0.5.1/24 --key=12345 work fine as well?

The EADDRNOTAVAIL is an exception raised by the call to bind -- and bind is called three times (on TCP): once for the public listener on port 443 (with the IP provided by --ipv4=) - again port 80, same IP address (for redirecting to https), and once for port 1234 (on the --private-ipv4=) for the control interface.

You can also not specify --ipv4 and --private-ipv4 -> this will then bind to all IP addresses [since it defaults to 0.0.0.0] (you will receive errors if port 80 / 443 / 1234 is already in used by another program).

But have a try for sudo ./dist/tlstunnel --ipv4=10.0.5.1/24 --private-ipv4=10.0.5.1/24 --key=12345 --ipv4-only=true --private-ipv4-only=true.

[hugusmaximus]

It's working!

hugo@anakin:~/tlstunnel/unikernel$ sudo ./dist/tlstunnel --ipv4=10.0.5.1/24 --private-ipv4=10.0.5.1/24 --key=12345 --ipv4-only=true --private-ipv4-only=true
2024-01-17T18:57:57Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
2024-01-17T18:57:57Z: [INFO] [tcpip-stack-socket] Dual IPv4 and IPv6 socket stack: connect
2024-01-17T18:57:57Z: [INFO] [application] read from 2024-01-16T18:12:50-00:00 (counter 0) 0 bytes data
2024-01-17T18:57:57Z: [INFO] [application] SNI map has 0 entries
2024-01-17T18:57:57Z: [INFO] [application] now reading certificates from KV
2024-01-17T18:57:57Z: [INFO] [application] read certificates from KV
2024-01-17T18:57:57Z: [INFO] [application] certificate valid until 2024-10-11T23:59:59-00:00, good true
2024-01-17T19:08:59Z: [INFO] [application] xxxxxxxxxxxx.com is now redirected to 10.0.5.3:80
2024-01-17T19:11:50Z: [INFO] [application] xxxxxxxxxxxx.com is no longer redirected
2024-01-17T19:12:02Z: [INFO] [application] xxxxxxxxxxxxxx.com is now redirected to 10.0.5.3:80
2024-01-17T19:15:11Z: [ERROR] [application] TCP read error Unix.Unix_error(Unix.EBADF, "check_descriptor", "")

I was setting a wrong private address. I have been able to run the tlstunnel and to add, remove and list routes and also to test with a real environment (backend web server). It is wonderful. I'm just getting this error some times:

[ERROR] [application] TCP read error Unix.Unix_error(Unix.EBADF, "check_descriptor", "")

if you are interested I can try to debug. The config utility works fine, anyway, maybe some static file loaded at compile time (same as certificates) would make it even easier to use. Overall, I'm pretty happy with the result. I'm going to stress it a bit to check how it behaves under load.

[hannesm]

Great that it works for you!

[ERROR] [application] TCP read error Unix.Unix_error(Unix.EBADF, "check_descriptor", "")

That's likely rather innocent -> when a write() fails, this may (currently) result in the file descriptor being closed -- while still there's a read ongoing. There's some ongoing work to revise the "FLOW close" API (starting at https://github.com/mirage/mirage-flow/pull/48, with documentation in https://github.com/mirage/mirage-flow/pull/49 and https://github.com/mirage/mirage-flow/pull/51). Then, on the TLS layer there's https://github.com/mirleft/ocaml-tls/issues/485 and also work in progress to include shutdown (and at the same time have a stable semantics for close and whether the underlying TCP FLOW is closed as well -- see https://github.com/mirleft/ocaml-tls/pull/488). I hope to have these changes finalized and merged and released before the end of this month.

maybe some static file loaded at compile time (same as certificates) would make it even easier to use

And here, similar to our previous discussion about TLS certificates, my point is that clearly you can "simplify" this unikernel by avoiding (a) block device (b) any management channel and just hardcode the host -> IP, port map in the OCaml code. My take is that this is very fine, but is a separate unikernel project (of course, code can be reused). And on my way for deployment (and again, reproducible builds), I really like to push the configuration out of compile time.