How do I connect to an HTTPS node?

[scull7]

I'm receiving the following error:

ERROR es-install :: Client(ClientError { inner: Error(Request, State { next_error: Some(Error(Hyper(Error(Connect, Custom { kind: InvalidInput, error: NotHttp })), "https://localhost:9201/mt-ledger?pretty=true")), backtrace: None }) })

Here is the code which produces this error:

pub mod client {
    use elastic::{
        client::{prelude::SyncClient, SyncClientBuilder},
        error::Error,
    };
    use std::result::Result;
    pub fn make() -> Result<SyncClient, Error> {
        SyncClientBuilder::new()
            .static_node("https://localhost:9201")
            .params_fluent(move |p| p.url_param("pretty", true))
            .build()
    }
}

use mtledger::elasticsearch::client as EsClient;

fn main() {
    let client = EsClient::make().expect("Failed to create ES client");

    match client.ping().send() {
        Ok(response) => {
            println!("PING :: {:?}", response);
        }
        Err(error) => {
            println!("ERROR es-ping :: {:?}", error);
        }
    }
}

How do I connect to an HTTPS node?

[mwilliammyers]

This should work:

let client = SyncClientBuilder::new()
    .static_node("https://es.example.com:9200")
    .params_fluent(|p| {
        p.header(
            AUTHORIZATION,
            HeaderValue::from_str(&format!(
                "Basic {}",
                encode(r#"username:password"#)
            ))
           .unwrap(),
        )
    })
    .build()
    .unwrap();

see #363

[scull7]

Thank you for the reply, a couple questions:

1. Where does the encode function come from?
2. Where does the NotHttp error originate and what does it mean?

I dug around and couldn't find either item in the source code, though it might just be me being dumb again as with thinking this is because of the https protocol and not the lack of authorization.

[scull7]

So to answer my own question 1, base64::encode is the function in question, which again I should have known since the BasicAuth header needs to be encoded 🤦‍♂ Unfortunately I found it the hard way by digging through the reqwest code: https://docs.rs/reqwest/0.9.22/src/reqwest/request.rs.html#241

[scull7]

I've updated my code and I'm still receiving the same error:

pub mod client {
    use elastic::{
        client::{prelude::SyncClient, SyncClientBuilder},
        error::Error,
        http::header::{HeaderValue, AUTHORIZATION},
    };
    use std::result::Result;
    pub fn make() -> Result<SyncClient, Error> {
        SyncClientBuilder::new()
            .static_node("https://localhost:9201")
            .params_fluent(move |p| {
                p.url_param("pretty", true).header(
                    AUTHORIZATION,
                    HeaderValue::from_str(&format!("Basic {}", base64::encode(r#"admin:admin"#)))
                        .expect("ERROR :: Basic Auth header failed"),
                )
            })
            .build()
    }
}

cargo run --bin es-ping
    Finished dev [unoptimized + debuginfo] target(s) in 0.09s
     Running `target/debug/es-ping`
ERROR es-ping :: Client(ClientError { inner: Error(Request, State { next_error: Some(Error(Hyper(Error(Connect, Custom { kind: InvalidInput, error: NotHttp })), "https://localhost:9201/?pretty=true")), backtrace: None }) })

[scull7]

I'm attempting to connect to an OpenDistro installation.

The following command works:

curl -XGET https://localhost:9201 -u admin:admin --insecure

which outputs:

{
  "name" : "e9ea03f22617",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "_j6r0p9mT2ynIedg6mOn-g",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[scull7]

Seems that I may be correct in my first thought that this is something to do with me attempting to connect via https vs http.

https://github.com/hyperium/hyper/blob/8b878a805ac6a95a758b6ef08f6410449c65648a/src/client/connect/http.rs#L344

The NotHttp error is a [hyper] error that refers to an InvalidUrl error.

impl StdError for InvalidUrl {
    fn description(&self) -> &str {
        match *self {
            InvalidUrl::MissingScheme => "invalid URL, missing scheme",
            InvalidUrl::NotHttp => "invalid URL, scheme must be http",
            InvalidUrl::MissingAuthority => "invalid URL, missing domain",
        }
    }
}

How do I include HTTPS support?

[mwilliammyers]

My hunch is that reqwest is not accepting the certificates that OpenDistro is presenting. Does it not work if you run curl like (note the missing --insecure):

curl -XGET https://localhost:9201 -u admin:admin

That being said—this doesn't seem to match the error message you are getting?

If certs are indeed an issue we will need to at least update the docs (but preferably add something to make this a lot more ergonomic). In the meantime, you can use the http_client method on SyncClientBuilder to pass in a reqwest::Client that you have already configured with either add_root_certificate or danger_accept_invalid_certs.

In production you might want to use a validated cert chain and skip the above steps...?

If all of the above doesn't work/certs aren't the issue then I will get an OpenDistro container running and dig into this deeper.

[mwilliammyers]

I went ahead and verified that this works:

use elastic::{
    client::SyncClientBuilder,
    http::header::{HeaderValue, AUTHORIZATION},
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let http_client = reqwest::Client::builder()
        .danger_accept_invalid_certs(true)
        .build()?;

    let client = SyncClientBuilder::new()
        .static_node("https://localhost:9201")
        .http_client(http_client)
        .params_fluent(|p| {
            p.header(
                AUTHORIZATION,
                HeaderValue::from_str(&format!("Basic {}", base64::encode("admin:admin"))).unwrap()
            )
        })
        .build()?;

    println!("{:#?}", client.ping().send()?);

    Ok(())
}

[dependencies]
elastic = "0.21.0-pre.5"
# elastic_derive = "0.21.0-pre.5"
reqwest = "0.9.22"
base64 = "0.11.0"

[mwilliammyers]

As of: #401 (6ed243570bb64fdf398cafe0518c5d9a02321c76) you can now do:

use elastic::client::SyncClientBuilder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let http_client = reqwest::Client::builder()
        .danger_accept_invalid_certs(true)
        .build()?;

    let client = SyncClientBuilder::new()
        .static_node("https://localhost:9201")
        .http_client(http_client)
        .params_fluent(|p| p.basic_auth("admin", Some("admin")).unwrap())
        .build()?;

    println!("{:#?}", client.ping().send()?);

    Ok(())
}

[scull7]

@mwilliammyers The danger_accept_invalid_certs functions is hidden behind the tls feature flag. When I enable this feature I receive the following errors pop out (request v0.9.22):

   Compiling reqwest v0.9.22
error[E0282]: type annotations needed
   --> /Users/nsculli/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.9.22/src/async_impl/client.rs:198:9
    |
139 |         let mut connector = {
    |             ------------- consider giving `connector` a type
...
198 |         connector.set_timeout(config.connect_timeout);
    |         ^^^^^^^^^ cannot infer type
    |
    = note: type must be known at this point

error[E0308]: mismatched types
   --> /Users/nsculli/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.9.22/src/tls.rs:270:21
    |
270 |     fn default() -> TlsBackend {
    |        -------      ^^^^^^^^^^ expected enum `tls::TlsBackend`, found ()
    |        |
    |        implicitly returns `()` as its body has no tail or `return` expression
    |
    = note: expected type `tls::TlsBackend`
               found type `()`

error: aborting due to 2 previous errors

What version of reqwest are you using? What configuration?

[scull7]

I've also opened an issue for reqwest: https://github.com/seanmonstar/reqwest/issues/700

[scull7]

The following configuration worked:

[dependencies]
# https://github.com/elastic-rs/elastic/issues/400#issuecomment-549579150
elastic = { git = "https://github.com/elastic-rs/elastic" }
elastic_derive = { git = "https://github.com/elastic-rs/elastic"}
# elastic = "~0.21.0-pre.5"
# elastic_derive = "~0.21.0-pre.5"

[dependencies.reqwest]
version = "~0.9"
default-features = false
features = ["rustls-tls"]

Code to generate the client:

pub mod client {
    use elastic::{
        client::{prelude::SyncClient, SyncClientBuilder},
        error::Error,
    };
    use reqwest::Client as Reqwest;
    use std::result::Result;
    pub fn make() -> Result<SyncClient, Error> {
        let http_client = Reqwest::builder()
            .danger_accept_invalid_certs(true)
            .build()
            .expect("ERROR :: Could not build reqwest client");
        SyncClientBuilder::new()
            .static_node("https://localhost:9201")
            .http_client(http_client)
            .params_fluent(|p| p.basic_auth("admin", Some("admin")).unwrap())
            .build()
    }
}

Result:

PING :: PingResponse { name: "e9ea03f22617", cluster_name: "docker-cluster", tagline: "You Know, for Search", version: ClusterVersion { number: "7.2.0", build_hash: "508c38a", build_date: "2019-06-20T15:54:18.811730Z", build_snapshot: false, lucene_version: "8.0.0" } }

@mwilliammyers Thank you for your help!

[mwilliammyers]

No problem!

Oops, I wish docs.rs showed what features are needed (if any) for a given function...

Glad you figured it out!

[scull7]

Yeah, it certainly wasn't clear, but it works now. Perhaps it would be a good example to include? With some way to find it using OpenDistro as a keyword? Anyone trying to use their docker container for local development against Amazon's ES implementation will run into this.

[mwilliammyers]

Yeah I actually tried adding it as a runnable example but ran into a danger_accept_invalid_certs method not found probably because of that same issue. The weird thing is when I used it in a separate project I didn't need to specify any features for reqwest - it was exactly as I specified earlier...

I will try to get it working and add an example or put it in the docs somewhere sometime soon.

[TomPridham]

is it necessary to use a new reqwest client to connect to anything other than localhost?

[mwilliammyers]

@TomPridham Nope! Just if you want to do something fancy like bypass certificate validation like he did. Otherwise you can just use the default one that elastic makes automatically for you.