Betterstack Metrics Vector Error

[samuelexferri]

Hello, I can't get the dashboard and metrics to work, the logs are fine. I'm using betterstack-logs-1.1.2 in a Kubernetes with Traefik and Cert Manager.

My betterstack-values.yaml:

metrics-server:
  enabled: true

vector:
  hostNetwork:
    enabled: true
  customConfig:
    sources:
      better_stack_kubernetes_logs:
        type: kubernetes_logs
        extra_label_selector: "app=my-engine"
    sinks:
      better_stack_http_sink:
        auth:
          token: <TOKEN>
      better_stack_http_metrics_sink:
        auth:
          token: <TOKEN>

My logs:

2024-04-26T07:35:35.667928Z  INFO vector::app: Log level is enabled. level="info"
2024-04-26T07:35:35.669251Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2024-04-26T07:35:40.435062Z  INFO source{component_kind="source" component_id=better_stack_kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="pool-kfolvp7yd-ybqmr"
2024-04-26T07:35:40.451872Z  INFO source{component_kind="source" component_id=better_stack_kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Including matching files. ret=["**/*"]
2024-04-26T07:35:40.451912Z  INFO source{component_kind="source" component_id=better_stack_kubernetes_logs component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Excluding matching files. ret=["**/*.gz", "**/*.tmp"]
2024-04-26T07:35:45.132014Z  INFO vector::topology::running: Running healthchecks.
2024-04-26T07:35:45.132294Z  INFO vector: Vector has started. debug="false" version="0.37.1" arch="x86_64" revision="cb6635a 2024-04-09 13:45:06.561412437"
2024-04-26T07:35:45.132319Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-04-26T07:35:45.132419Z  INFO vector::topology::builder: Healthcheck passed.
2024-04-26T07:35:45.132542Z  INFO vector::topology::builder: Healthcheck passed.
2024-04-26T07:35:45.180427Z  INFO source{component_kind="source" component_id=better_stack_kubernetes_logs component_type=kubernetes_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2024-04-26T07:35:50.178452Z ERROR source{component_kind="source" component_id=better_stack_kubernetes_metrics_nodes component_type=http_client}: vector::internal_events::http_client_source: HTTP request processing error. url=https://betterstack-logs-metrics-server/apis/metrics.k8s.io/v1beta1/nodes error="Timeout error: request exceeded 5s" error_type="request_failed" stage="receiving" internal_log_rate_limit=true
2024-04-26T07:35:50.180134Z ERROR source{component_kind="source" component_id=better_stack_kubernetes_metrics_pods component_type=http_client}: vector::internal_events::http_client_source: HTTP request processing error. url=https://betterstack-logs-metrics-server/apis/metrics.k8s.io/v1beta1/pods error="Timeout error: request exceeded 5s" error_type="request_failed" stage="receiving" internal_log_rate_limit=true
2024-04-26T07:36:00.185253Z  WARN source{component_kind="source" component_id=better_stack_kubernetes_metrics_nodes component_type=http_client}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:0A000086:SSL routines:(unknown function):certificate verify failed:ssl/statem/statem_clnt.c:2092:: self-signed certificate in certificate chain error_type="request_failed" stage="processing" internal_log_rate_limit=true
2024-04-26T07:36:00.185360Z ERROR source{component_kind="source" component_id=better_stack_kubernetes_metrics_nodes component_type=http_client}: vector::internal_events::http_client_source: HTTP request processing error. url=https://betterstack-logs-metrics-server/apis/metrics.k8s.io/v1beta1/nodes error=CallRequest { source: hyper::Error(Connect, Custom { kind: Other, error: ConnectError { error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167772294, library: "SSL routines", function: "(unknown function)", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 2092 }]))) }, verify_result: X509VerifyResult { code: 19, error: "self-signed certificate in certificate chain" } } }) } error_type="request_failed" stage="receiving" internal_log_rate_limit=true
2024-04-26T07:36:00.189320Z  WARN source{component_kind="source" component_id=better_stack_kubernetes_metrics_pods component_type=http_client}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:0A000086:SSL routines:(unknown function):certificate verify failed:ssl/statem/statem_clnt.c:2092:: self-signed certificate in certificate chain error_type="request_failed" stage="processing" internal_log_rate_limit=true

I already try with tls.verify_certificate and tls.verify_hostname to false. Any help?

Thanks in advance

[curusarn]

Hi @samuelexferri,

Thank you for reaching out!

Could you share how you run Kubernetes? EKS, Digital Ocean, something else?

Could you try using the values.yaml configuration for Minikube? 🙏 It has helped other people with SSL issues.

https://betterstack.com/docs/logs/kubernetes/#2-set-up-helm-chart

Let me know if this helps. 🙏 I'm happy to look further into this.

Thanks again for raising this!

[samuelexferri]

Hi, thanks for the quick response! It works now!