http: proxy error: dial tcp i/o timeout

[BLuchterhand]

I've been testing a small Consul/Nomad environment in Vagrant and recently moved to a cluster of EC2s. I believe that at one point Fabio (completely unconfigured) was working just fine but is now giving me the following error when trying to access Fabio's 9999 port to be routed to my services:

For some context: My cluster is currently a collection of machines on their own networks so they don't communicate using their private IPs which Fabio seems to try connecting with (the 172.xx.xx.xx address that it tries in the logs). The services themselves are properly port forwarded and are accessible by the public IP of the consul/nomad client that is hosting it and the proper port, so my guess is that Fabio just has no way of translating the private IP that it displays in the routing table to the proper public IP that points to the correct machine hosting the services to be routed to.

Here's the job file I've been testing with:

job "webserver" {
  datacenters = ["us-central"]

  group "webserver" {
    task "server" {
      driver = "docker"
      config {
        image = "hashicorp/demo-prometheus-instrumentation:latest"
      }

      resources {
        cpu = 500
        memory = 256
        network {
          mbits = 10
          port  "http"{}
        }
      }

      service {
        name = "webserver"
        port = "http"

        tags = [
          "urlprefix-/",
        ]

        check {
          type     = "http"
          path     = "/"
          interval = "2s"
          timeout  = "2s"
        }
      }
    }
  }
}

And here's Fabio's routing table pointing to the service's internal IP:

Is there anything obvious that I'm missing? I'd greatly appreciate some help... I can post my Nomad/Consul configurations as well if it helps, I'm just afraid of oversharing information if it's not necessary. If you need to see that please let me know!

[nathanejohnson]

fabio uses the service address as it is presented to consul. You've blurred out the full IP, but assuming the IP addresses you've obscured are in the 172.16/12 subnet then it's trying to connect to the private IP's and can't. This isn't really a fabio problem it sounds like, but fabio needs to be able to reach the services in order to be a reverse proxy for them.