datadog log and metric sink does not use private endpoints

[mbettadp]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We setup the Private Endpoints using the docs here: https://docs.datadoghq.com/agent/guide/private-link/?tab=connectfromsameregion and we have a DataDog Metrics and Logs sink setup. However, the sinks are not sending traffic over the private endpoint.

Configuration

No response

Version

0.39.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hey! Could you share your Vector configuration? At least for the sinks.

[mbettadp]

Sure thing! This is our config at the moment. I did not have the endpoint specified initially, I was relying on the default. That did not work so I added this endpoint specifically which also seems like it did not work.

[sinks.dd_logs_sink_1] type = "datadog_logs" inputs = [ "fanout_logs_route.output1" ] endpoint = "https://http-intake.logs.datadoghq.com/api/v3/logs" default_api_key = "${DD_API_KEY}" batch.timeout_secs = 10 batch.max_bytes = 2000000 buffer.max_events = 4000 buffer.when_full = "${BUFFER_ON_FULL}" request.concurrency = "adaptive" request.retry_attempts = 10

[pront]

The v3 stands out to me. Did you try https://http-intake.logs.datadoghq.com/api/v2/logs?

[mbettadp]

I can try that for sure. I will get back to you with the results. Just so I know, what is the default endpoint that the sink uses?

[pront]

I can try that for sure. I will get back to you with the results. Just so I know, what is the default endpoint that the sink uses?

https://github.com/vectordotdev/vector/blob/master/src/sinks/datadog/logs/config.rs#L87-L89 which resolves to https://http-intake.logs.datadoghq.com/api/v2/logs

[mbettadp]

So we tried with this endpoint (both setting it explicitly and using just the default) and it did not work. We did not see any traffic going over our private endpoint

[jszwedko]

This sounds like it could potentially be a DNS or private link configuration issue. Are you able to use dig from the same host/container as Vector is running in to verify that the IP addresses returned for http-intake.logs.datadoghq.com are the expected private DNS entries?

[mbettadp]

Yup, our tests from the host seemed to all work. We saw it being resolved correctly and we did see traffic hit the private link corresponding to the times we ran the tests from the host. It's just the traffic exiting out of the Datadog sink that seems to somehow not flow over the private link for some reason. I can't verify it's respecting the endpoint that we set in the sink config

[jszwedko]

Yup, our tests from the host seemed to all work. We saw it being resolved correctly and we did see traffic hit the private link corresponding to the times we ran the tests from the host. It's just the traffic exiting out of the Datadog sink that seems to somehow not flow over the private link for some reason. I can't verify it's respecting the endpoint that we set in the sink config

Interesting. You can run with -vv (trace level logs) to see the requests that Vector is making.