[guide section]: How to customize TLS

[jdisanti]

A note for the community

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

Tell us about your request

• Add more documentation to aws_smithy_client::Client docs on how to customize TLS.
• Point builder docs at that client documentation.
• Write a developer guide section on customizing TLS, and link to (or cover) ca-certs and using-native-tls-instead-of-rustls examples from it

Tell us about the problem you're trying to solve.

N/A

Are you currently working around this issue?

N/A

Additional context

No response

[oxlade39]

I previously raised #334 and later ca-certs was added which I thought I could follow.

I need to do this behind a proxy and I have tried to add hyper_proxy::ProxyConnector. Rather unfortunately my corporate proxy is just an IP address and rustls does not support this.

This means I'm back to attempting to configure native-tls with a custom root cert. I just can't work out the right combination of types. The closest I've gotten is below but this doesn't compile. Some better documentation would really be appreciated here.

use std::{fs::File, env};
use std::io::BufReader;

use aws_config::provider_config::ProviderConfig;
use aws_config::profile::ProfileFileCredentialsProvider;
use aws_sdk_s3::Region;
use aws_smithy_client::hyper_ext;
use hyper::Client;
use hyper_proxy::{Proxy, Intercept, ProxyConnector};
use hyper_tls::HttpsConnector;
use native_tls::Certificate;
use std::path::Path;

fn load_ca_cert(pem_file: &Path) -> Result<Certificate, CertLoadError> {
    use std::fs;

    let bytes =
        fs::read(pem_file).map_err(|e| CertLoadError::Io(format!("Loading {:?}", pem_file), e))?;

    Certificate::from_pem(&bytes).map_err(CertLoadError::TlsError)
}

#[derive(Debug)]
enum CertLoadError {
    TlsError(native_tls::Error),
    Io(String, std::io::Error),
}

#[tokio::main]
async fn main() {
    env_logger::init();

    // insert your root CAs
    let certificate: native_tls::Certificate =
        load_ca_cert(&Path::new("/etc/ssl/certs/ca-certificates.crt")).expect("Failed to load your CA cert");

    let native_tls_connector = native_tls::TlsConnector::builder()
        .add_root_certificate(certificate)
        .build()
        .expect("Building native_tls::TlsConnector");

    let tokio_tls_tls_connector = tokio_native_tls::TlsConnector::from(native_tls_connector);

    let mut hyper_http_connector = hyper::client::HttpConnector::new();
    hyper_http_connector.enforce_http(false);

    let hyper_tls_https_connector = hyper_tls::HttpsConnector::from((
        hyper_http_connector,
        tokio_tls_tls_connector,
    ));

    let client_main = Client::builder().build::<_, hyper::Body>(hyper_tls_https_connector);

    let https_proxy = env::var("https_proxy").unwrap();
    let proxy_uri = https_proxy.replace("http", "https").parse().unwrap();
    let proxy = Proxy::new(Intercept::All, proxy_uri);
    let pc = ProxyConnector::from_proxy(tokio_tls_tls_connector, proxy).unwrap();

    let profile_creds = ProfileFileCredentialsProvider::builder()
        .profile_name("profile-name")
        .build();

    // Currently, aws_config connectors are buildable directly from something that implements `hyper::Connect`.
    // This enables different providers to construct clients with different timeouts.
    let provider_config = ProviderConfig::default()
        .with_tcp_connector(pc.clone());

    let shared_conf = aws_config::from_env()
        .region(Region::new("us-east-1"))
        .credentials_provider(profile_creds)
        .configure(provider_config)
        .load()
        .await;
    let s3_config = aws_sdk_s3::Config::from(&shared_conf);
    // however, for generated clients, they are constructred from a Hyper adapter directly:
    let s3_client = aws_sdk_s3::Client::from_conf_conn(
        s3_config,
        hyper_ext::Adapter::builder().build(pc),
    );
    let buckets = s3_client.list_buckets().send().await.unwrap();
    let items = buckets.buckets().unwrap();
    print!("buckets: {}", items[0].name.clone().unwrap());
}

[rcoh]

is it possible to add a DNS name to /etc/hosts so that you can refer to the proxy by a DNS name? You could also do the same thing by using a custom DNS resolver in Hyper 💭

[oxlade39]

is it possible to add a DNS name to /etc/hosts so that you can refer to the proxy by a DNS name? You could also do the same thing by using a custom DNS resolver in Hyper 💭

Unfortunately I don't have write access to /etc/hosts. I'll try and work out how to add a custom DNS revolver in Hyper, that sounds promising.

[oxlade39]

@rcoh sorry I'm still having trouble working this all out. I'm trying to find an example of specifying a custom DNS resolver for Hyper but I've not managed to find much documentation on this. I don't suppose you have a link to an example or pointer in the right direction?

[rcoh]

does this help? https://docs.rs/hyper/latest/hyper/client/connect/index.html

[oxlade39]

Sorry, I'm afraid not. I was aware of the above documentation and I really appreciate you patience but it's not clear to me how one would go about combining the below code with the custom DNS resolver example.

The hyper client itself appears to be hidden behind layers of other abstraction for aws_smithy, Proxy and rust_tls.

use std::{fs, io};
use std::{fs::File, env};
use std::io::BufReader;

use aws_config::provider_config::ProviderConfig;
use aws_config::profile::ProfileFileCredentialsProvider;
use aws_sdk_s3::Region;
use aws_smithy_client::hyper_ext;
use hyper_proxy::{Proxy, Intercept, ProxyConnector};
use rustls::{RootCertStore, Certificate, internal::msgs::codec::Codec};
use rustls_pemfile::certs;

#[tokio::main]
async fn main() {
    env_logger::init();

    // insert your root CAs
    let f = File::open("/etc/ssl/certs/ca-certificates.crt").unwrap();
    let mut reader = BufReader::new(f);    
    let mut root_store = RootCertStore::empty();

    for cert in certs(&mut reader).unwrap() {
        root_store.add(&Certificate(cert));
    }

    let config = rustls::ClientConfig::builder()
        .with_safe_defaults()
        .with_root_certificates(root_store)
        .with_no_client_auth();
    let rustls_connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_tls_config(config.clone())
        .https_only()
        .enable_http1()
        .enable_http2()
        .build();

    let https_proxy = env::var("https_proxy").unwrap();
    let proxy_uri = https_proxy.replace("http", "https").parse().unwrap();
    let proxy = Proxy::new(Intercept::All, proxy_uri);
    let pc = ProxyConnector::from_proxy(rustls_connector, proxy).unwrap();

    let profile_creds = ProfileFileCredentialsProvider::builder()
        .profile_name("fiaim-deployer-dev")
        .build();

    // Currently, aws_config connectors are buildable directly from something that implements `hyper::Connect`.
    // This enables different providers to construct clients with different timeouts.
    // let provider_config = ProviderConfig::default()
    //     .with_tcp_connector(pc.clone());
    let provider_config = ProviderConfig::default();
    let shared_conf = aws_config::from_env()
        .region(Region::new("us-east-1"))
        .credentials_provider(profile_creds)
        .configure(provider_config)
        .load()
        .await;
    let s3_config = aws_sdk_s3::Config::from(&shared_conf);
    // however, for generated clients, they are constructred from a Hyper adapter directly:
    let s3_client = aws_sdk_s3::Client::from_conf_conn(
        s3_config,
        hyper_ext::Adapter::builder().build(pc),
    );
    let buckets = s3_client.list_buckets().send().await.unwrap();
    let items = buckets.buckets().unwrap();
    print!("buckets: {}", items[0].name.clone().unwrap());
}

I fear I may be fundamentally misunderstanding.

[rcoh]

I'll see if I can whip up an example, but I think you want this: https://docs.rs/hyper-rustls/latest/hyper_rustls/struct.HttpsConnectorBuilder.html#method.wrap_connector-1

Which will enable you to drop in your own custom HttpConnector with DNS stubbed out

[rcoh]

here's a snippet that compiles for me in the context of the larger example you posted above.

use std::net::SocketAddr;
use std::iter;
use hyper::client::HttpConnector
// ... snip ...

    let config = rustls::ClientConfig::builder()
        .with_safe_defaults()
        .with_root_certificates(root_store)
        .with_no_client_auth();

    let resolver = tower::service_fn(|_name| async {
        // update to _always_ return the corp IP address. This will enable you to pass in a DNS
        // name but will always return your corp IP address
        Ok::<_, Infallible>(iter::once(SocketAddr::from(([127, 0, 0, 1], 8080))))
    });
    let http_conector = HttpConnector::new_with_resolver(resolver);

    let rustls_connector = hyper_rustls::HttpsConnectorBuilder::new()
        .with_tls_config(config.clone())
        .https_only()
        .enable_http1()
        .wrap_connector(http_conector);

[oxlade39]

that's great, thanks. It makes sense now.

It's compiling fine but panicking now "invalid URL, scheme is not http" That's probably something else in my setup that's wrong although I do wonder if it's because I'm passing an HttpConnector to wrap_connector.

I'll keep digging. Thanks again.

[oxlade39]

I just had to turn off enforce_http: https://github.com/hyperium/hyper/issues/1009#issuecomment-592522889