vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.16k stars 1.6k forks source link

datadog_metrics sink from Windows host results in certificate errors #10107

Open thethomp opened 3 years ago

thethomp commented 3 years ago

Vector Version

vector 0.17.3 (x86_64-pc-windows-msvc d72c6e7 2021-10-21)

Vector Configuration File

datadog-bb-metrics:
    type: datadog_metrics
    inputs:
    - transform-concurrency-log-to-metrics
    api_key: ***
    default_namespace: test

Debug Output

Expected Behavior

Metrics are sent to Datadog without issue.

Actual Behavior

Metrics are not sent to Datadog and instead just get these logs lines repeated returned:

Nov 18 03:04:17.430 DEBUG sink{component_kind="sink" component_id=datadog-bb-metrics component_type=datadog_metrics component_name=datadog-bb-metrics}:request{request_id=0}:http: vector::internal_events::http_client: Sending HTTP request. uri=https://0-17-3-vector.agent.datadoghq.com/api/v1/series method=POST version=HTTP/1.1 headers={"content-type": "application/json", "dd-api-key": "***", "user-agent": "Vector/0.17.3 (x86_64-pc-windows-msvc d72c6e7 2021-10-21)", "accept-encoding": "identity"} body=[222 bytes]
Nov 18 03:04:18.740  WARN sink{component_kind="sink" component_id=datadog-bb-metrics component_type=datadog_metrics component_name=datadog-bb-metrics}:request{request_id=0}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl\statem\statem_clnt.c:1916:: self signed certificate in certificate chain

Example Data

This occurs with any metrics data trying to be sent.

Additional Context

I tried running Vector from several different Windows hosts and this error occurs on all of them, regardless of their networking configuration (I tried with zero firewall or admin restrictions). What's interesting is the log sinks from the same exact configuration work just fine (though they do hit a different Datadog endpoint). Another interesting note is that there are Datadog agents running on these same hosts, sending metrics just fine, which I assume is hitting a similar (if not the same) endpoint, though I'm not positive on that. This is what makes me suspicious that maybe there's something up on the Vector side.

It would be interesting to see if it can be reproduced by your team.

References

jszwedko commented 2 years ago

For others, it looks like forcing TLS 1.2 fixes this:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12