CrashLoopback with basic install - kube 1.28

[Vrtak-CZ]

Hi,

I'm trying to use this helm chart to monitor our kubernetes cluster (EKS - v1.28.5-eks-5e0fdde). But pods constantly crashing.

metrics-server:                                                                                                                                                                                                                                             
  enabled: false                                                                                                                                                                                                                                            
vector:                                                                                                                                                                                                                                                     
  customConfig:                                                                                                                                                                                                                                             
    sinks:                                                                                                                                                                                                                                                  
      better_stack_http_metrics_sink:                                                                                                                                                                                                                       
        auth:                                                                                                                                                                                                                                               
          token: <REDACTED>                                                                                                                                                                                                                   
      better_stack_http_sink:                                                                                                                                                                                                                               
        auth:                                                                                                                                                                                                                                               
          token: <REDACTED>                                                                                                                                                                                                                   
    sources:                                                                                                                                                                                                                                                
      better_stack_kubernetes_metrics_nodes:                                                                                                                                                                                                                
        endpoint: https://metrics-server/apis/metrics.k8s.io/v1beta1/nodes                                                                                                                                                                                  
        tls:                                                                                                                                                                                                                                                
          verify_certificate: false                                                                                                                                                                                                                         
          verify_hostname: false                                                                                                                                                                                                                            
      better_stack_kubernetes_metrics_pods:                                                                                                                                                                                                                 
        endpoint: https://metrics-server/apis/metrics.k8s.io/v1beta1/pods                                                                                                                                                                                   
        tls:                                                                                                                                                                                                                                                
          verify_certificate: false                                                                                                                                                                                                                         
          verify_hostname: false

I have tryed to upgrade vector to latest version and looks like it did not help.

vector:
  image:                                                                                                                                                                                                                                                    
    tag: 0.36.0-distroless-libc

logs

2024-02-19T00:39:56.562799Z ERROR transform{component_kind="transform" component_id=better_stack_kubernetes_metrics_pods_parser component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for \"for_each\
" at (53:1014): function call error for \"for_each\" at (268:1012): function call error for \"push\" at (397:691): function call error for \"to_float\" at (500:555): Invalid floating point number \"954u\": invalid float literal" error_type="conversion_
failed" stage="processing" internal_log_rate_limit=true                                                                                                                                                                                                     
thread 'vector-worker' panicked at src/transforms/remap.rs:592:48:                                                                                                                                                                                          
event will be set                                                                                                                                                                                                                                           
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace                                                                                                                                                                               
thread 'vector-worker' panicked at src/topology/builder.rs:920:30:                                                                                                                                                                                          
internal error: entered unreachable code: join error or bad poll                                                                                                                                                                                            
2024-02-19T00:39:56.563495Z ERROR transform{component_kind="transform" component_id=better_stack_kubernetes_metrics_pods_parser component_type=remap}: vector::topology: An error occurred that Vector couldn't handle: the task panicked and was aborted. 

[curusarn]

Hi @Vrtak-CZ,

Thank you for reporting this issue!

Could you share what version of metrics-server are you running? 🙏 Have you tried different metrics-server version by any chance?

Thanks again for reaching out!

[Vrtak-CZ]

@curusarn i'm running latest it means registry.k8s.io/metrics-server/metrics-server:v0.7.0

[curusarn]

Thank you @Vrtak-CZ!

I'm passing this on to the team to reproduce and fix.

[samuelexferri]

I'm following this issue, I can't get the dashboard and metrics to work, the logs are fine.

2024-04-09T13:44:49.925975Z ERROR source{component_kind="source" component_id=better_stack_kubernetes_metrics_nodes component_type=http_client component_name=better_stack_kubernetes_metrics_nodes}: vector::internal_events::http_client_source: HTTP request processing error. url=https://betterstack-logs-metrics-server/apis/metrics.k8s.io/v1beta1/nodes error=CallRequest { source: hyper::Error(Connect, Custom { kind: Other, error: ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" }) }) } error_type="request_failed" stage="receiving" internal_log_rate_limit=true
2024-04-09T13:44:49.948495Z  WARN source{component_kind="source" component_id=better_stack_kubernetes_metrics_pods component_type=http_client component_name=better_stack_kubernetes_metrics_pods}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name or service not known error_type="request_failed" stage="processing" internal_log_rate_limit=true

Using this values.yaml and default metric-server:

metrics-server:
  enabled: false

vector:
  hostNetwork:
    enabled: true
  customConfig:
    sources:
      better_stack_kubernetes_logs:
        type: kubernetes_logs
        extra_label_selector: "app=my-engine"
    sinks:
      better_stack_http_sink:
        auth:
          token: <TOKEN>
      better_stack_http_metrics_sink:
        auth:
          token: <TOKEN>

[PetrHeinz]

Thank you for your patience on this one 🙌

The newly released version 1.1.2 addresses the issue and prevents Vector instance from crashing.