search

WebThingsIO / webthing-rust

Rust implementation of a Web Thing server
Mozilla Public License 2.0
211 stars 25 forks source link

How to use TLS? #55

Open DazWilkin opened 3 years ago

DazWilkin commented 3 years ago

The README.md is lacking in documenting how to use TLS Support:

"If you need TLS support for the server, you'll need to compile with the ssl feature set."

I'm willing to augment the README.md with the solution but I'm unable to get it working myself :disappointed:

I'm unable to curl the TLS endpoint (403):

curl \
--insecure \
--key privatekey.pem \
--cert certificate.pem \
--write-out '%{response_code}' \
https://${HOST}:8888

And I'm unable to discover the device in the gateway.

thing-url-adapter:
Failed to connect to https://${HOST}.local:8888:
FetchError: request to https://${HOST}.local:8888/ failed, reason: self signed certificate

I believe, in the example code, I should replace:

webthing = "0.13.2"

with, e.g.:

[dependencies.webthing]
path = "../webthing-rust"
version = "0.13.2"
features = ["ssl"]

Then:

openssl req \
-x509 \
-nodes \
-days 365 \
-newkey rsa:2048 \
-keyout privatekey.pem \
-out certificate.pem \
-subj /CN=${HOST}

And:

let mut server = WebThingServer::new(
    ThingsType::Multiple(things, "Rusty-Device".to_owned()),
    Some(8888),
    None,
    Some(("privatekey.pem".to_string(), "certificate.pem".to_string())),
    Box::new(Generator),
    None,
)
mrstegeman commented 3 years ago

You're doing the right thing. A couple things to note:

  1. The Web Thing add-on will not currently accept self-signed certificates. This is why we don't widely advertise the feature. TLS on a private network is tricky.
  2. You generally get a 403 status when the hostname doesn't match what's expected. Make sure you set the third parameter of your WebThingServer::new() call appropriately, e.g. Some($HOST) (obviously replacing $HOST with what you're using in your shell commands).
DazWilkin commented 3 years ago

Thank you for the prompt response(s)!

I've been using the Python SDK as a proxy for the Rust SDK.

If I configure the Python example (on 8887):

server = WebThingServer(
    things=MultipleThings([light, sensor], 'Pythonic-Device'),
    port=8887,
    hostname="my-host",
    ssl_options={
        "keyfile": "privatekey.pem",
        "certfile": "certificate.pem",
})

NOTE Good call, thanks... I've replaced my-host in these examples with $(hostname)

I'm (still) unable browse the Python devices using Gateway (because of the self-signed issue so I'll drop trying this) but I am able to curl the endpoint:

curl \
--silent \
--insecure \
--key privatekey.pem \
--cert certificate.pem \
https://${HOST}.local:8887 \
| jq -r '.[].title'
Pythonic Lamp
Pythonic Humidity Sensor

But, using (what I think is) the same configuration with the Rust example (except on 8888), I get 403:

let mut server = WebThingServer::new(
    ThingsType::Multiple(things, "Rusty-Device".to_owned()),
    Some(8888),
    Some("my-host".to_string()),
    Some(("privatekey.pem".to_string(), "certificate.pem".to_string())),
    Box::new(Generator),
    None,
);
server.start(None).await

And:

curl \
--silent \
--insecure \
--key privatekey.pem \
--cert certificate.pem \
https://${HOST}.local:8888 \
--write-out '%{response_code}'
403

I'm either configuring server incorrectly or the code backing it has an issue?

Here are the Gateway log (errors) for completeness but I'll stop pursuing TLS on my private network:

thing-url-adapter:
Failed to connect to https://${HOST}.local:8888:
FetchError: request to https://${HOST}.local:8888/ failed, reason: self signed certificate

thing-url-adapter:
Failed to connect to https://${HOST}.local:8887:
FetchError: request to https://${HOST}.local:8887/ failed, reason: self signed certificate
mrstegeman commented 3 years ago

Try using my-host.local instead of my-host in your Rust code.

DazWilkin commented 3 years ago

Yes, no combination appears to work :-(

Will have another look tomorrow.

DazWilkin commented 3 years ago

No resolution :-(

I used the Actix Web rustls example.

I am able to use the example's certs with the example with both rustls and openssl and both work.

The example works with my cert|key and openssl

However, I am unable to use my cert|key with the example and rustls.

IIUC webthing-rust uses openssl so this may be a red-herring but the rustlsuses ServerConfg and the code fails for me:

config.set_single_cert(cert_chain, keys.remove(0)).unwrap();

I suspect because my cert is self-signed?

However, flipping this around, I'm unable to curl the webthing-rust sample using the actix-web sample's cert|key

IIUC webthing-rust is using actix-web and openssl and so it's still unclear why this doesn't work.

Couple of other perhaps red-herrings but....

webthing-rust file server.rs line ~145 implements HostValidatorMiddelware with call function. When I debug the code, this fails using curl even if I add --header "Host: hades-canyon". If I manually, inject a value into the function, it fails with HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8).

webthing-rust file server.rs line ~942 WebThingsServer::start builds an extensive list of hosts on my machine:

get_addresses() = ["127.0.0.1", "172.17.0.1", "172.18.0.1", "172.19.0.1", "172.20.0.1", "172.21.0.1", "172.22.0.1", "172.23.0.1", "172.24.0.1", "172.25.0.1", "172.26.0.1", "172.27.0.1", "192.168.1.150", "192.168.1.186", "[::1]"]

I'm unsure why this would be necessary to gather these addresses if the service should only accept hostname(.domain):port

Rather flummoxed.

mrstegeman commented 3 years ago

IIUC webthing-rust is using actix-web and openssl

Yes, that's correct.

I'm unsure why this would be necessary to gather these addresses if the service should only accept hostname(.domain):port

Given that these things live on a local network, and mDNS resolution doesn't always work (especially on Windows), we add the local IP addresses to list of allowed hostnames.

This verification is all done to prevent DNS rebinding.

DazWilkin commented 3 years ago

The Host header is being dropped|discarded with TLS and the absence of this header appears to cause a prompt 403 (see below).

non-TLS:

webthing = { path = "../webthing-rust", version = "0.13.2" }

And:

curl  \
--silent \
 --write-out '%{response_code}'
--output /dev/nill
http://hades-canyon.local:8888/

Response:

200

With server.rs after the fn call signature:

println!("{:?}", req.headers());

Yields:

HeaderMap { inner: {"accept": One("*/*"), "host": One("hades-canyon.local:8888"), "user-agent": One("curl/7.68.0")} }

NOTE it includes the host header (and it is correct).

TLS

webthing = { path = "../webthing-rust", version = "0.13.2", features = ["ssl"] }

And:

curl \
--silent \
--insecure \
--key ./secrets/privatekey.pem \
--cert ./secrets/certificate.pem \
--header "dog: Freddie" \
--write-out '%{response_code}' \
 --output /dev/null \
https://hades-canyon.local:8888/

Response:

403

And:

HeaderMap { inner: {"accept": One("*/*"), "user-agent": One("curl/7.68.0"), "dog": One("Freddie")} }

NOTE No header and so:


if host.is_none() {
  return Either::Right(ok(
    req.into_response(HttpResponse::Forbidden().finish().into_body())
 ));
}

Even if I try to manually inject a Host header, it is dropped:

curl \
--silent \
--insecure \
--key ./secrets/privatekey.pem \
--cert ./secrets/certificate.pem \
--header "dog: Freddie" \
--write-out '%{response_code}' \
 --output /dev/null \
https://hades-canyon.local:8888/

Response:

403
mrstegeman commented 3 years ago

Can you try adding --http-1.1 to your TLS curl command?

DazWilkin commented 3 years ago

Hmmm..... As you updated the thread, I was about to add this

Yes, I think it's http/2

DazWilkin commented 3 years ago

Yes, that's it... I now get 200s

mrstegeman commented 3 years ago

Ok, great. Glad we got that figured out!

It looks like we may be able to disable http/2 with this.

DazWilkin commented 3 years ago 
curl \
--http1.1 \
--silent \
--insecure \
--key ./secrets/privatekey.pem \
--cert ./secrets/certificate.pem  \
https://hades-canyon.local:8888/

Yields:

200

And:

HeaderMap { inner: {"accept": One("*/*"), "host": One("hades-canyon.local:8888"), "user-agent": One("curl/7.68.0")} }
DazWilkin commented 3 years ago

I tried:

let mut builder = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap();
builder
    .set_private_key_file(o.0.clone(), SslFiletype::PEM)
    .unwrap();
builder.set_certificate_chain_file(o.1.clone()).unwrap();
builder.set_alpn_protos(b"\x06spdy/1\x08http/1.1").unwrap();

NOTE Tried both b"\x06spdy/1\x08http/1.1" and b"\x08http/1.1"

But I get 403s still if I drop the --http1.1

And the Host header isn't received by the call fn:

HeaderMap { inner: {"accept": One("*/*"), "user-agent": One("curl/7.68.0")} }

Is this perhaps because the server is unable to then upgrade the HTTP connection for WebSockets?

mrstegeman commented 3 years ago

Hmm, very strange. I don't have any suggestions right now, other than forcing HTTP/1.1.

DazWilkin commented 3 years ago

Thanks very much for your patience with this.

We have something of a diagnosis.

mrstegeman commented 3 years ago

I just released a new version today that has an option to disable host validation.

DazWilkin commented 3 years ago

Thank you updating it and me :smile:

Using the example.

With:

WebThingServer::new(
  ThingsType::Multiple(things, "Rusty-Device".to_owned()),
  Some(8888),
  None,
  ssl,
  Box::new(Generator),
  None,
  Some(true),
)

And:

curl \
--insecure \
--key ./secrets/privatekey.pem \
--cert ./secrets/certificate.pem \
--write-out '%{response_code}' \
https://${HOST}:8888

With /:

Client:

curl: (92) HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)

Server:

thread 'actix-rt:worker:0' panicked at 'called `Option::unwrap()` on a `None` value'
.../.cargo/registry/src/github.com-1ecc6299db9ec823/webthing-0.14.0/src/server.rs:408:42

Because handle_get_things:

let host = req.headers().get("Host").unwrap().to_str().unwrap();

With /0:

thread 'actix-rt:worker:0' panicked at 'called `Option::unwrap()` on a `None` value'
.../.cargo/registry/src/github.com-1ecc6299db9ec823/webthing-0.14.0/src/server.rs:464:50

Because handle_get_thing but same reason; tries to get the host header

With /0/properties and /0/properties/brightness, I get 200s :smile:

Neither handle_get_properties nor handle_get_property attempt to read the host header

mrstegeman commented 3 years ago

Ahh, that's tricky. We use the host header to generate the links arrays. I think we're back at an impasse here, where we need to just disable http/2 somehow.

DazWilkin commented 3 years ago

I appreciate your time spent on this, thank you!

There's some complexity which makes it difficult for me to try to be more helpful:

These points are by way of explanation and not seeking a response.

There's a discrepancy between the e.g. Python (which works) and Rust (which doesn't) implementation. My expectation is that all WebThings SDKs behave equivalently but the issue here may be in the underlying SSL implementation over which you've no control.

Perhaps this issue can serve as a lighthouse so others don't founder on this rock?

Otherwise, this isn't a breaking issue for me.

mrstegeman commented 3 years ago
  • IIUC http/2 doesn't permit Host headers

Honestly not sure about that.

  • I don't know why (beyond being on the latest-greatest) WebThings (Rust SDK) uses http/2

We don't explicitly use it. It's just part of the actix-web framework we're using.

  • I don't know why the different handlers behave differently with the Host header

The Host header is actually used to generate the thing description in certain handlers. In all other cases, it's only used to prevent DNS rebinding attacks.

  • I don't know whether there's a dependency between use of http/2 and websockets in this SDK

No, there's not.