Monitor external endpoints

[ArjonBu]

It would be great if the tool can be extended to monitor external endpoints (HTTP, TCP and DNS).

This way we could monitor the outgoing network related infrastructure too from each k8s node.

[clementnuss]

good idea, will try to implement that 👍🏼

[grantkl]

Any update on this?

[clementnuss]

I'm not sure whether a dedicated DNS or TCP mode should be introduced.

However I'm in favor of permitting to query additional endpoints.

I was thinking of an env variable as follows:

KUBENURSE_EXTRA_CHECKS=<type label>:<http endpoint>,...
KUBENURSE_EXTRA_CHECKS=cloud_api_endpoint|https://your-cloud-api-endpoint.cloud:6789/readyz,some_other_endpoint|http://endpoint.tld/ok,...

what do you think ?

[grantkl]

For my use case, DNS is enough. TCP could be interesting though.

Your ENV variables seem like they are more focused on TCP though? Maybe separate ENV variables for DNS vs TCP?

[clementnuss]

the thing is that there's no "dns" only check at the moment.

we always make a direct HTTP check towards an endpoint. only sometimes we use an IP address to connect, and sometimes we use a DNS name to connect.

If you want to monitor DNS, I think hostlookuper could be the right tool for that use case.

note that at the moment, hostlookuper exports vmrange buckets instead of prometheus le buckets, which breaks compatibility with other metrics collection system than VictoriaMetrics. I'll try to fix that issue soon.

but I think it would be exactly the right tool to monitor DNS servers and endpoints.

[grantkl]

I mean a TCP check would inherently use DNS anyways so it would work for me 😄 We already use kubenurse, so am I'm hoping I can replace some custom built checks we have around external connectivity.

[clementnuss]

I agree, that'd be best.

But one last thing: kubenurse is only making HTTP queries, no TCP checks. All the instrumentation in the code is around httptrace, and I don't want to introduce a new set of metrics just for the TCP case.

if your endpoint is an http server, then my env var solution would work

KUBENURSE_EXTRA_CHECKS=cloud_api_endpoint|https://your-cloud-api-endpoint.cloud:6789/readyz,some_other_endpoint|http://endpoint.tld/ok,...

if it's not, maybe a solution such as the blackbox exporter running as a DaemonSet might me more appropriate

[grantkl]

I believe I can make HTTP work, Thanks!

[clementnuss]

ok, will try to implement that in the coming days, and will let you know once it's done!

thanks for the feedback :)

[grantkl]

@clementnuss were you able to get something implemented for this?

[clementnuss]

hi @grantkl

yes, finally ! 🙃

as described in the docs, usage is as follows:

you can try it for youself with the pre-release (sha256:62957254924c3845640edd01fc12b436caed5ecd0d348a2ec6bb04b573d13ab0). please let me know how this fits your needs: https://github.com/users/clementnuss/packages/container/kubenurse/247399033?tag=v1.13.2-prerelease

[grantkl]

Nice! We're going to test this out!

[clementnuss]

did you have the time to test it ?