How do I force TLSDomain to accept self signed certificate

What Operating System(s) are you seeing this problem on?

Linux X11

Which Wayland compositor or X11 Window manager(s) are you using?

WezTerm version

wezterm 20230612-063953-baf9d970

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

When I run the wezterm-mux-server inside a docker container, I setup a TLS server running on an exposed port, and I generate a set of ssl files (certs and keys relevant to the container). I copy those files out of the container and start wezterm from my host, attempting to connect to the wezterm-mux-server inside the container. I get a failure with the error the handshake failed: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:: self-signed certificate. Which of course they are, I just made them. How do I make wezterm accept these self signed certs?

Or maybe a better question. What is the "expected" way to connect TLS Domains as the doc is quite unclear on the way to utilize them (aside from the nebulous "remote_ssh bootstrap" which I don't want to opt for in a container).

To Reproduce

Honestly you can probably recreate this by simply starting the mux server on one machine, generating certificates and using them on another to connect. I have a bit of a complex docker file for all this (and more) which can be found here.

The specific steps I am running into are

Setup a container that has the nightly wezterm-mux-server build in it
Expose a port on said container, and set a configuration inside the container (for this example, I exposed 1234)
Start the mux server in the container
Connect to the TLS domain from the container host

Experience failure

Configuration

-- server (container) wezterm.lua
local wezterm = require("wezterm")
local config = wezterm.config_builder and wezterm.config_builder() or {}
config.tls_servers = {{
    bind_address = '0.0.0.0:1234',
    pem_private_key = '/root/.local/share/wezterm/ssl/ca.key',
    pem_ca = '/root/.local/share/wezterm/ssl/ca.crt',
    pem_cert = '/root/.local/share/wezterm/ssl/server.crt',
}}
return config

-- client (host) wezterm.lua
local wezterm = require("wezterm")
local config = wezterm.config_builder and wezterm.config_builder() or {}

config.tls_clients = {{
    name = "dummy",
    remote_address = "0.0.0.0:1234",
    pem_private_key = '/tmp/wezterm-ssl/ca.key',
    pem_ca = '/tmp/wezterm-ssl/ca.crt',
    pem_cert = '/tmp/wezterm-ssl/server.crt',
    remote_wezterm_path = '/usr/bin/wezterm'
}}

return config

Expected Behavior

I would expect a way to tell wezterm to accept self signed certs, though its also possible that I am performing this connection all wrong.

Logs

Client Logs

22:05:00.756 ERROR mux::connui > Failed: SslConnector for 0.0.0.0:1234 with host name 0.0.0.0

Caused by: 0: the handshake failed: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:: self-signed certificate 1: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889: 2: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:

22:05:01.228 ERROR env_bootstrap > panic at /home/miversen/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.38/src/error.rs:223:40 - !? 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: __libc_start_main 35:

thread 'main' panicked at 'called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 1, error_len: Some(1) }', /home/miversen/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-0.10.38/src/error.rs:223:40 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace 22:05:01.228 ERROR mux::connui > while running ConnectionUI loop: recv_timeout: channel is empty and disconnected

Server Logs

03:04:48.790 ERROR wezterm_mux_server_impl::pki > runtime dir is /root/.local/share/wezterm/pki 03:04:48.791 ERROR wezterm_mux_server::ossl > listening with TLS on "0.0.0.0:1234" 03:05:00.757 ERROR wezterm_mux_server::ossl > failed TlsAcceptor: the handshake failed: error:0A000418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1584:SSL alert number 48

Anything else?

Completely unrelated but I noticed that TlsDomainClient refers to config.tls_domains which according to wezterm is incorrect.

I don't know what else to provide here, if I am missing anything, please let me know!

As it stands today, you can either:

Use bootstrap_via_ssh to bootstrap the trust between client and server, and have wezterm internally generate a certificate for the client to use. The way it works is to ssh to the server, then run wezterm cli tlscreds to ask the mux server to vend out a certificate for the client, and the client will then use that to connect via TLS.
or: use your existing PKI infrastructure to manage your CA and client certificates. The verification doesn't allow for self-signed certs, but you should be able to make DIY CA to issue and sign your certificate with that (you can google for examples of doing that), and configure the client and server pem_ca accordingly.

I recommend using bootstrap_via_ssh for simplicity, because it requires no real state to be managed by you, and the chances are reasonable that if you're allowing network access to the container you probably trust ssh for that too.

I recommend using bootstrap_via_ssh for simplicity, because it requires no real state to be managed by you, and the chances are reasonable that if you're allowing network access to the container you probably trust ssh for that too.

You are correct, however it also means I need to have an ssh daemon running in a docker container (as the target here is to run a mux server in docker). Alternatively, this same thing will occur on systems that don't have an ssh server (such as connecting to a mux server on a windows machine). Of course, I could install an ssh server, but then I might as well just ssh in as opposed to simply connecting to an existing mux server.

Use bootstrap_via_ssh to bootstrap the trust between client and server, and have wezterm internally generate a certificate for the client to use. The way it works is to ssh to the server, then run wezterm cli tlscreds to ask the mux server to vend out a certificate for the client, and the client will then use that to connect via TLS.

I noticed the wezterm cli tlscreds command but I am not sure how to use it. I have complete control over the container, so there is no reason I can't have it spit out the creds as requested here, but I don't understand how I am supposed to provide those creds to wezterm to be consumed.

wezterm cli tlscreds is an implementation detail used by the bootstrap logic, so it doesn't have a way to expose the data. I suppose that it could offer a --json parameter that exposes it. This is the data it could output:

https://github.com/wez/wezterm/blob/c1f495099ef544eda78546191c0026e88e1fdd62/codec/src/lib.rs#L623-L630

so if there was such a json flag, it would provide the bytes to the CA and certificate PEMs, and you could then stash that somewhere to use explicitly with your client.

Note that those credentials are only valid for the lifetime of the mux server; if it is restarted, you will need to generate a current copy.

Just to make sure I understand correctly. With the above, this would add a --json flag to the wezterm cli tlscreds command?

Is this to input your creds to the mux server (via a proper json formatted string)? I honestly think half of my issue here is that I don't fully understand how the client is supposed to connect to the server if you choose to not use the ssh bootstrap.

I would hate to see you put in extra work on something that might be able to be explained via documentation instead.

If that flag was added, in your docker container, when you start the mux server, you would run wezterm cli tlscreds --json and then parse the output and save the CA and cert data somewhere. Alternatively, maybe that command could output them both together as a single file of PEM data to make that a bit easier and avoid the json parsing.

Then you would convey the creds somehow to your client, and configure your client to reference them:

config.tls_clients = {
  {
    name = "my-docker-name",
    remote_address = ...,
    pem_cert = "path to the certificate you saved from above",
    pem_ca = "path to the CA you saved from above",
  }
}

Ahh that would be perfect I think. That is basically what I am doing now (with the shared self signed certs and such)

Is that more or less what is happening inside wezterm with the ssh bootstrapper?

Yeah, wezterm is essentially doing that for you. I've pushed the --pem option. You'll want to do something like:

$ wezterm cli tlscreds --pem > /dev/shm/creds.pem

then securely transfer that file to the client, and set both pem_cert and pem_ca to point to it. That file includes the client certificate, client private key and server CA certificate in a single file.

if you're connecting directly from the host machine into the container running inside the host you can perhaps be less paranoid and maybe just simplify that to a docker exec NAME wezterm cli tlscreds --pem > /dev/shm/wezterm-NAME-creds.pem on the host and then literally configure the TLS domain to use /dev/shm/wezterm-NAME-creds.pem

(I have not tested trying to use the result of this in this way, but I think there's an 80% chance that it will work without me having to dig deeper!)

After a bit of tweaking the above configurations, I cannot seem to get this to connect. The added --pem flag does exactly what it should do (thanks for that!) but I am still getting a self-signed certificate in certificate chain error on the client.

On the host I am seeing an unknown ca error when the handshake is initiated.

-- server (container) config
local wezterm = require("wezterm")
local config = wezterm.config_builder and wezterm.config_builder() or {}

config.tls_servers = {{
    bind_address = '0.0.0.0:1234',
}}

return config

-- client (host) config
local wezterm = require("wezterm")
local config = wezterm.config_builder and wezterm.config_builder() or {}

config.tls_clients = {{
    name = "dummy",
    remote_address = "0.0.0.0:1234",
    accept_invalid_hostnames = true,
    pem_private_key = '/tmp/creds.pem',
    pem_ca = '/tmp/creds.pem',
    pem_cert = '/tmp/creds.pem',
    remote_wezterm_path = '/usr/bin/wezterm'
}}

return config

Server logs

02:15:11.945  ERROR  wezterm_mux_server_impl::pki > runtime dir is /root/.local/share/wezterm/pki
02:15:11.946  ERROR  wezterm_mux_server::ossl     > listening with TLS on "0.0.0.0:1234"
02:15:14.408  ERROR  wezterm_mux_server::ossl     > failed TlsAcceptor: the handshake failed: error:0A000418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1584:SSL alert number 48

Client Logs

21:15:14.408  ERROR  mux::connui > 
Failed: SslConnector for 0.0.0.0:1234 with host name 0.0.0.0

Caused by:
    0: the handshake failed: error:0A000086:SSL routines:tls_post_process_server_ce
rtificate:certificate verify failed:ssl/statem/statem_clnt.c:1889:: self-signed cer
tificate in certificate chain
    1: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate 
verify failed:ssl/statem/statem_clnt.c:1889:
    2: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate 
verify failed:ssl/statem/statem_clnt.c:1889:

21:15:14.889  ERROR  env_bootstrap > panic at /home/miversen/.cargo/registry/src/gi
thub.com-1ecc6299db9ec823/openssl-0.10.38/src/error.rs:223:40 - !?
   0: env_bootstrap::register_panic_hook::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/b
oxed.rs:2032:9
      std::panicking::rust_panic_with_hook
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:692:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:579:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/sys
_common/backtrace.rs:137:18
   4: rust_begin_unwind
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:575:5
   5: core::panicking::panic_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/pa
nicking.rs:64:14
   6: core::result::unwrap_failed
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/re
sult.rs:1791:5
   7: <openssl::error::Error as core::fmt::Display>::fmt
   8: core::fmt::write
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1208:17
   9: core::fmt::Formatter::write_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1662:9
  10: <&T as core::fmt::Display>::fmt
  11: core::fmt::write
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1208:17
  12: core::fmt::Formatter::write_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1662:9
  13: <openssl::ssl::error::Error as core::fmt::Display>::fmt
  14: core::fmt::write
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1208:17
  15: core::fmt::Formatter::write_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1662:9
  16: <openssl::ssl::error::HandshakeError<S> as core::fmt::Display>::fmt
  17: core::fmt::write
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1208:17
  18: anyhow::fmt::<impl anyhow::error::ErrorImpl>::debug
  19: core::fmt::write
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:1208:17
  20: core::fmt::Write::write_fmt
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/fm
t/mod.rs:192:9
      alloc::fmt::format::format_inner
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/alloc/src/f
mt.rs:612:9
  21: <wezterm_client::domain::ClientDomain as mux::domain::Domain>::attach::{{clos
ure}}
  22: wezterm_gui::async_run_terminal_gui::{{closure}}
  23: <async_task::runnable::Builder<M>::spawn_local::Checked<F> as core::future::f
uture::Future>::poll
  24: async_task::raw::RawTask<F,T,S,M>::run
  25: window::spawn::SpawnQueue::run
  26: <window::os::x11::connection::XConnection as window::connection::ConnectionOp
s>::run_message_loop
  27: wezterm_gui::run_terminal_gui
  28: wezterm_gui::main
  29: std::sys_common::backtrace::__rust_begin_short_backtrace
  30: std::rt::lang_start::{{closure}}
  31: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::cal
l_once
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/core/src/op
s/function.rs:606:13
      std::panicking::try::do_call
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:483:40
      std::panicking::try
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:447:19
      std::panic::catch_unwind
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
ic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/rt.
rs:148:48
      std::panicking::try::do_call
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:483:40
      std::panicking::try
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
icking.rs:447:19
      std::panic::catch_unwind
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/pan
ic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/d5a82bbd26e1ad8b7401f6a718a9c57c96905483/library/std/src/rt.
rs:148:20
  32: main
  33: <unknown>
  34: __libc_start_main
  35: _start

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error {
 valid_up_to: 0, error_len: Some(1) }', /home/miversen/.cargo/registry/src/github.c
om-1ecc6299db9ec823/openssl-0.10.38/src/error.rs:223:40
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
21:15:14.890  ERROR  mux::connui   > while running ConnectionUI loop: recv_timeout:
 channel is empty and disconnected

Attempted usage is as follows

Start a container with port 1234 bound to port 1234
Copy the above server configuration into /root/.config/wezterm/wezterm.lua
Pull down the wezterm git repository in said container and build it
Copy the 3 relevant wezterm binaries from the target build directory into some location on the shell path (I put them in /usr/bin/
From outside the container, run docker exec $CONTAINER_NAME wezterm cli tlscreds --pem > /tmp/creds.pem
Attempt to connect wezterm in the client with wezterm in the container.
Observe above error

If you would like, I can strip out the wezterm specific part of my container and get you a dockerfile for it for test :)

@miversen33 I got this working, here's what I did

Server

Requires step cli

# Create CA and two private/public key pairs (one for server, one for client)
mkdir -p ~/.local/share/wezterm/pki
cd ~/.local/share/wezterm/pki
step certificate create --insecure --no-password --profile root-ca --not-after 43800h "Example Root CA" root_ca.crt root_ca.key
step certificate create server.com server.com.crt server.com.key --kty=RSA --profile leaf --ca ./root_ca.crt --ca-key ./root_ca.key --insecure --no-password --not-after 43800h
step certificate create $USER client.com.crt client.com.key --kty=RSA --profile leaf --ca ./root_ca.crt --ca-key ./root_ca.key --insecure --no-password --not-after 43800h

~/.wezterm.lua

local wezterm = require("wezterm")
local config = wezterm.config_builder and wezterm.config_builder() or {}
config.tls_servers = {{
    bind_address = '0.0.0.0:47777',
    pem_private_key = '/home/<username>/.local/share/wezterm/pki/server.com.key',
    pem_cert = '/home/<username>/.local/share/wezterm/pki/server.com.crt',
    pem_ca = '/home/<username>/.local/share/wezterm/pki/root_ca.crt',
    pem_root_certs = {'/home/<username>/.local/share/wezterm/pki/root_ca.crt'},
}}
return config

Client

mkdir -p ~/.local/share/wezterm/pki
scp <SERVER>:.local/share/wezterm/pki/{client.com.key,client.com.crt,root_ca.crt} .local/share/wezterm/pki/

config.tls_clients = {
    {
        name = '<DOMAIN NAME>',
        remote_address = '<server>:47777',
        accept_invalid_hostnames = true,
        pem_private_key = '<HOME DIRECTORY>/.local/share/wezterm/pki/client.com.key',
        pem_cert = '<HOME DIRECTORY>/.local/share/wezterm/pki/client.com.crt',
        pem_ca = '<HOME DIRECTORY>/.local/share/wezterm/pki/root_ca.crt',
        pem_root_certs = { '<HOME DIRECTORY>/.local/share/wezterm/pki/root_ca.crt' },
        local_echo_threshold_ms = 1000,
    },
}

wez / wezterm