Unable to Override Consul Sidecar Service Proxy Port

[hichon78]

Nomad version

Output from nomad version

Nomad v1.1.2 (60638a086ef9630e2a9ba1e237e8426192a44244)

Operating system and Environment details

CentOS 7 Consul 1.8.5

Issue

I'm not sure whether this should be logged against nomad or consul. But the definition is in the Nomad job spec, so I'll start here. According to this doc, we should be able to override the consul sidecar proxy port via the service.connect.sidecar_service.port parameter.

https://www.consul.io/docs/connect/registration/sidecar-service#SidecarServiceDefaults

However, setting this doesn't seem to have any effect on the sidecar listening port. The port assignment is still dynamic and adheres to the default port range. I tried setting to a port outside of the default range as well as one within it. Both cases, the port was still dynamically assigned.

Reproduction steps

Create a service and sidecar proxy. I was using the dashboard demo for my testing:

    service {
      name = "dashboard"
      port = "http"
      connect {
        sidecar_service {
          port = "8443"
          proxy {
            upstreams {
              destination_name  = "counting"
              local_bind_port   = 5000
            }
          }
        }
      }
    }

Expected Result

The sidecar proxy should listen on the port provided in the configuration.

Actual Result

The sidecar proxy port was dynamically assigned and the configuration was ignored.

Job file (if appropriate)

job "count-dash" {
  type = "service"
  datacenters = ["primary"]

  group "count-api" {
    count = 1

    constraint {
      operator = "distinct_hosts"
    }

    network {
      mode = "bridge"
      port "http" {
        to = -1
      }
    }

    service {
      name = "counting"
      port = "9001"

      connect {
        sidecar_service {
          proxy {}
        }
      }

      check {
        name     = "Counting Service Health Check"
        type     = "http"
        method   = "GET"
        protocol = "http"
        port     = "http"
        interval = "10s"
        timeout  = "2s"
        path     = "/health"
        expose   = true
      }
    }

    task "counting" {
      driver = "docker"
      config {
        image = "hashicorp/counting-service:0.0.2"
      }
    }
  }

  group "dashboard" {
    count = 1
    constraint {
      operator = "distinct_hosts"
    }

    network {
      mode = "bridge"

      port "http" {
        host_network = "private"
        static  = 9002
        to      = 9002
      }

    }

    service {
      name = "dashboard"
      port = "http"
      connect {
        sidecar_service {
          port = "21111"
          proxy {
            upstreams {
              destination_name  = "counting"
              local_bind_port   = 5000
            }
          }
        }

        sidecar_task {
          config {
            args = [
              "-c",
              "${NOMAD_SECRETS_DIR}/envoy_bootstrap.json",
              "-l",
              "debug",
              "--concurrency",
              "${meta.connect.proxy_concurrency}",
              "--disable-hot-restart"
            ]
          }
        }
      }

      check {
        name      = "Dashboard Health Check"
        type      = "http"
        method    = "GET"
        protocol  = "http"
        port      = "http"
        interval  = "10s"
        timeout   = "2s"
        path      = "/health"
      }
    }

    task "dashboard" {
      driver = "docker"

      env {
        COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_counting}"
      }

      config {
        image = "hashicorp/dashboard-service:0.0.4"
        ports = ["http"]
      }
    }
  }
}

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

N/A

[lgfa29]

Thank you @Logik78.

I was able to reproduce it, and it does seem like we're dropping the sidecar service port information when going from the jobspec to the Consul API call.

[hichon78]

Note, it also doesn't appear that the default configurations are obeyed either:

https://www.consul.io/docs/agent/options#sidecar_min_port

setting sidecar_min_port and sidecar_max_port doesn't do anything.

And also note that the defaults documented (21000 - 21255) are not obeyed, since I see port values of 27xxx and 28xxx

[hichon78]

What is the actual default range of ports used for the sidecar proxies. I have an ingress proxy situated in a DMZ network zone that is connected to the mesh network and talks through the sidecars. I need to limit the firewall rules to a specific port range but this issue is currently blocking this.