Connection Refused on port 9301

[eXpire163]

Describe the bug

When setting up Squid Exporter as a sidecar to squid on AWS Managed Kubernetes (EKS) the exported started up fine and was reachable via a service. Never the less the liveness probe on tcp or http always failed with:

Liveness probe failed: dial tcp x.x.x.2:9301: connect: connection refused

To Reproduce

Terraform deployment:

resource "kubernetes_deployment" "squid-proxy" {
  metadata {
    name = var.app-name
    labels = {
      app        = var.app-name
    }
    namespace = kubernetes_namespace.staging-proxy.metadata[0].name
  }

  spec {
    replicas = 3
    strategy {
      rolling_update {
        max_unavailable = "1"
      }
    }

    selector {
      match_labels = {
        app = var.app-name
      }
    }

    template {
      metadata {
        labels = {
          app        = var.app-name
          pipelineid = var.pipeline_label
        }
        annotations = {
          pipelineid = var.pipeline_label
          allowlist  = local.allowlist_sha1
          squidconfig = local.squid_config_sha1
        }
      }
      spec {
        container {
          image = "xxx.amazonaws.com/internet-proxy:${var.image-tag}"
          name  = "squid"
          resources {
            limits = {
              cpu    = "1"
              memory = "1Gi"
            }
            requests = {
              cpu    = "250m"
              memory = "512Mi"
            }
          }
          port {
            container_port = 3128
          }
          volume_mount {
            mount_path = "/etc/squid/squid-allowlist"
            name       = "allowlist"
            read_only  = true
          }
          volume_mount {
            mount_path = "/etc/squid/squid.conf"
            sub_path    = "squid.conf"
            name       = "squid-config"
            read_only  = true
          }
          liveness_probe {
            tcp_socket {
              port = "3128"
            }
          }
          readiness_probe {
            exec {
              command = ["squidclient", "-h", "localhost", "cache_object://localhost/counters"]
            }
          }
        }
        container {
          image = "xxx.amazonaws.com/squid-exporter:latest"
          name  = "squid-exporter"
          resources {
            limits = {
              cpu    = "200m"
              memory = "1Gi"
            }
            requests = {
              cpu    = "100m"
              memory = "212Mi"
            }
          }
          port {
            container_port = 3129
            name = "metrics"
          }
          env {
            name = "SQUID_HOSTNAME"
            value = "127.0.0.1"
          }
          env{
            name = "SQUID_PORT"
            value = "3128"
          }
      #    env{
      #      name = "SQUID_EXPORTER_LISTEN"
       #     value = ":3129"
       #   }

          liveness_probe {
            tcp_socket {
              port = 3129
            }
            failure_threshold = 2
            period_seconds = 15
            initial_delay_seconds = 15
          }

        }
        volume {
          name = "allowlist"
          config_map {
            name = kubernetes_config_map.allow-list.metadata[0].name
          }
        }
        volume {
          name = "squid-config"
          config_map {
            name = kubernetes_config_map.squid-config.metadata[0].name
          }
        }
      }
    }
  }
}

Expected behavior Liveness probe should work

OS (please complete the following information):

• OS: Amazon Linux for EKS 1.21

Solution

          env{
           name = "SQUID_EXPORTER_LISTEN"
           value = ":3129"
         }

Notes

Without specifying the exporter port the log said : listening on "10.11.12.13:9301" (node ip changed for this post) With specifying the exporter port the log said : listening on ":3129"

[boynux]

Do you know if liveness probes work on localhost bindings too?

Can you change to --h 0.0.0.0 to see if that solves the issue?

[boynux]

update: I quickly double checked that, it looks like liveness probes are done via routable IPs inside the cluster as such I think binding the exporter to localhost won't work.