in multi networks port forwarding doesn't work

[zakabluk]

I try use multi network. I have two network interfaces. I created host_network at client configuration.

  host_network "public" {
    interface = "eno1"
    cidr = "1.1.1.1/32"
  }

And added to job file config

 group "nginx" {
    count = 1

        network {

            port "admin" {
              to = 8080
              static = 80
              host_network = "public"
            }
        }
    service {
        name = "nginx"
        port = "admin"
      }

and afrer launched my job I got:

Allocation Addresses
Label   Dynamic  Address
*admin  yes      1.1.1.1:80 -> 8080

but when I get information about docker container

root@lim1:~/test$ sudo docker ps
CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS               NAMES
a8b0743e8173        bunkerity/bunkerized-nginx   "/opt/entrypoint.sh"   55 seconds ago      Up 54 seconds       80/tcp, 443/tcp     nginx-fcb299df-c234-637c-98d2-82adb98022d2

I inspected my monad job

                "Name": "nginx",
                "Networks": [
                    {
                        "CIDR": "",
                        "DNS": null,
                        "Device": "",
                        "DynamicPorts": null,
                        "IP": "",
                        "MBits": 10,
                        "Mode": "",
                        "ReservedPorts": [
                            {
                                "HostNetwork": "public",
                                "Label": "admin",
                                "To": 8080,
                                "Value": 80
                            }

I expected that my nginx will be work on public ip and use port 8080 instead 80 which set in docker container.

Nomad version

Nomad v0.12.5

Operating system and Environment details

Debian 5.7.10-1~bpo10+1

Issue

Reproduction steps

create in client config host_network and launch job.

Job file (if appropriate)

job "nginx-revers-proxy" {
  datacenters = ["dc1"]

  group "nginx" {
    count = 1

        network {

            port "admin" {
              to = 8080
              static = 80
              host_network = "public"
            }
        }

      service {
        name = "nginx"
        port = "admin"
      }

    task "nginx" {
      driver = "docker"

      config {
        image = "bunkerity/bunkerized-nginx"

        volumes = [
          "local:/etc/nginx/conf.d",
        ]
      }

      template {
        data = <<EOF
events {}

http {
  server {
    location / {
      proxy_pass http://nomad-ws;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      # Nomad blocking queries will remain open for a default of 5 minutes.
      # Increase the proxy timeout to accommodate this timeout with an
      # additional grace period.
      proxy_read_timeout 310s;

      # Nomad log streaming uses streaming HTTP requests. In order to
      # synchronously stream logs from Nomad to NGINX to the browser
      # proxy buffering needs to be turned off.
      proxy_buffering off;

      # The Upgrade and Connection headers are used to establish
      # a WebSockets connection.
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      # The default Origin header will be the proxy address, which
      # will be rejected by Nomad. It must be rewritten to be the
      # host address instead.
      proxy_set_header Origin "${scheme}://${proxy_host}";
    }
  }

  upstream nomad-ws {
    ip_hash;
    server 10.222.3.204:4646;
    server 10.222.0.31:4646;
    server 10.222.3.128:4646;
  }
}
EOF

        destination   = "local/nomad-revers.conf"
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }

      resources {
        network {
          mbits = 100

        }
      }

    }
  }
}

[tgross]

Hi @zakabluk! I suspect this was fixed by https://github.com/hashicorp/nomad/pull/8822, which hasn't made it into the changelog for the upcoming 0.13.0 yet. Can you try this same jobspec against the current master?

[tgross]

@zakabluk just wanted to follow up on this. With the current Nomad 1.0.0-rc I was able to verify this works. Something I noticed while trying it was that you don't have a ports configuration for the container, so Docker isn't getting the port mapping you need. See this simplified example:

job "example" {
  datacenters = ["dc1"]

  group "www" {

    network {
      port "www" {
        to           = 8001
        static       = 80
        host_network = "alternate"
      }
    }

    task "httpd" {
      driver = "docker"

      config {
        image   = "busybox:1"
        command = "httpd"
        args    = ["-v", "-f", "-p", "8001", "-h", "/www"]
        volumes = ["/tmp/www:/www:ro"]
        ports   = ["www"]
      }

      resources {
        cpu    = 256
        memory = 128
      }
    }
  }
}

$ nomad job run ./example.nomad
==> Monitoring evaluation "bb287821"
    Evaluation triggered by job "example"
==> Monitoring evaluation "bb287821"
    Evaluation within deployment: "c4643fd3"
    Allocation "d075fbf8" created: node "300e0a23", group "www"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "bb287821" finished with status "complete"

$ nomad alloc status d07
...

Allocation Addresses
Label  Dynamic  Address
*www   yes      10.199.0.11:80 -> 8001
...

vagrant@nomad-server01$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                NAMES
c26dc55c0d6b        busybox:1           "httpd -v -f -p 8001…"   3 seconds ago       Up 2 seconds        10.199.0.11:80->8001/tcp, 10.199.0.11:80->8001/udp   httpd-d075fbf8-b40b-0432-8c8f-fa4c552e930b

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.