search

ContainerSolutions / minimesos

The experimentation and testing tool for Apache Mesos - NO LONGER MAINTANED!
https://www.minimesos.org
Apache License 2.0
428 stars 61 forks source link

cluster fails to start: open of weave-scope.json #443

Closed gregzuro closed 8 years ago

gregzuro commented 8 years ago

Brand new installation of minimesos on Ubuntu trusty.

$ sudo minimesos init

...

$ sudo minimesos --debug up
08:23:54.191 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-zookeeper-3431856387-3961993792]
08:23:55.732 [main] DEBUG c.c.m.container.AbstractContainer - Container 22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26 is up and running
08:23:55.795 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-master-3431856387-2959406543]
08:23:57.287 [main] DEBUG c.c.m.container.AbstractContainer - Container df4ef7286c69b054f2ece2c242587a4c87c472cc41e51b0f483e256fccccba6d is up and running
08:23:57.341 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-agent-3431856387-217212859]
08:23:58.679 [main] DEBUG c.c.m.container.AbstractContainer - Container e458036376723751c8632a61cb69761a3cd16a0f5f9cbd1214b304509b7fc774 is up and running
08:23:58.701 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-marathon-3431856387-3786941047]
08:23:59.966 [main] DEBUG c.c.m.container.AbstractContainer - Container f5462925b01510408aba874066fd886d3fc869035974795c407124378be21221 is up and running
08:23:59.990 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-consul-3431856387-1691621991]
08:24:01.320 [main] DEBUG c.c.m.container.AbstractContainer - Container 23fec8bb5e85932fa7ac449da195684c067ec61553dafc0b14b0ca2a658d3432 is up and running
08:24:01.354 [main] DEBUG c.c.m.container.AbstractContainer - Creating container [minimesos-registrator-3431856387-1275330981]
08:24:02.619 [main] DEBUG c.c.m.container.AbstractContainer - Container 5d4c87991ed1f22d6e1f8538f5b3715bf08e98237798d21a27461befc3434428 is up and running
An error, which was handled, occurred
com.containersol.minimesos.MinimesosException: Failed to open https://raw.githubusercontent.com/ContainerSolutions/minimesos/e2a43362f4581122762c80d8780d09b567783f1a/apps/weave-scope.json as URL
    at com.containersol.minimesos.cluster.MesosCluster.getInputStream(MesosCluster.java:376) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.marathon.MarathonContainer.installMarathonApps(MarathonContainer.java:233) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.cluster.MesosCluster.start(MesosCluster.java:132) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.cluster.MesosCluster.start(MesosCluster.java:112) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.main.CommandUp.execute(CommandUp.java:104) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.main.Main.run(Main.java:125) ~[minimesos.jar:0.9.0]
    at com.containersol.minimesos.main.Main.main(Main.java:56) ~[minimesos.jar:0.9.0]
Caused by: java.net.UnknownHostException: raw.githubusercontent.com
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_72-internal]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_72-internal]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_72-internal]
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_72-internal]
    at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) ~[na:1.8.0_72-internal]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[na:1.8.0_72-internal]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) ~[na:1.8.0_72-internal]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1105) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:999) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1513) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) ~[na:1.8.0_72-internal]
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) ~[na:1.8.0_72-internal]
    at java.net.URL.openStream(URL.java:1045) ~[na:1.8.0_72-internal]
    at com.containersol.minimesos.cluster.MesosCluster.getInputStream(MesosCluster.java:374) ~[minimesos.jar:0.9.0]
    ... 6 common frames omitted
$

in parallel with:

$ sudo docker events
2016-05-22T01:23:52.666477738-07:00 container create 22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335 (image=containersol/minimesos:0.9.0, name=elated_dubinsky)
2016-05-22T01:23:52.667909783-07:00 container attach 22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335 (image=containersol/minimesos:0.9.0, name=elated_dubinsky)
2016-05-22T01:23:52.673215377-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335, name=bridge, type=bridge)
2016-05-22T01:23:52.783474561-07:00 container start 22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335 (image=containersol/minimesos:0.9.0, name=elated_dubinsky)
2016-05-22T01:23:54.560110690-07:00 container create 22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26 (image=jplock/zookeeper:3.4.6, name=minimesos-zookeeper-3431856387-3961993792, version=3.4.6)
2016-05-22T01:23:54.585045362-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26, name=bridge, type=bridge)
2016-05-22T01:23:54.585552288-07:00 volume mount 1148605abdd0670a8fe17360b3943749f0bbd9af101e7832d32b610c28d28d3d (container=22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26, destination=/tmp/zookeeper, driver=local, propagation=, read/write=true)
2016-05-22T01:23:54.585588131-07:00 volume mount d5ca2e3fd2093c78f1a34347006f4e2830b41a4b25dd88678045d72fc6065403 (container=22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26, destination=/opt/zookeeper/conf, driver=local, propagation=, read/write=true)
2016-05-22T01:23:54.711472810-07:00 container start 22bd57716d3c8933eb9126aa02f1361ff95901615ba23677185c510bfdad0a26 (image=jplock/zookeeper:3.4.6, name=minimesos-zookeeper-3431856387-3961993792, version=3.4.6)
2016-05-22T01:23:55.888666310-07:00 container create df4ef7286c69b054f2ece2c242587a4c87c472cc41e51b0f483e256fccccba6d (image=containersol/mesos-master:0.25.0-0.2.70.ubuntu1404, name=minimesos-master-3431856387-2959406543)
2016-05-22T01:23:56.141553229-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=df4ef7286c69b054f2ece2c242587a4c87c472cc41e51b0f483e256fccccba6d, name=bridge, type=bridge)
2016-05-22T01:23:56.271741477-07:00 container start df4ef7286c69b054f2ece2c242587a4c87c472cc41e51b0f483e256fccccba6d (image=containersol/mesos-master:0.25.0-0.2.70.ubuntu1404, name=minimesos-master-3431856387-2959406543)
2016-05-22T01:23:57.435409190-07:00 container create e458036376723751c8632a61cb69761a3cd16a0f5f9cbd1214b304509b7fc774 (image=containersol/mesos-agent:0.25.0-0.2.70.ubuntu1404, name=minimesos-agent-3431856387-217212859)
2016-05-22T01:23:57.481244560-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=e458036376723751c8632a61cb69761a3cd16a0f5f9cbd1214b304509b7fc774, name=bridge, type=bridge)
2016-05-22T01:23:57.650841637-07:00 container start e458036376723751c8632a61cb69761a3cd16a0f5f9cbd1214b304509b7fc774 (image=containersol/mesos-agent:0.25.0-0.2.70.ubuntu1404, name=minimesos-agent-3431856387-217212859)
2016-05-22T01:23:58.784227200-07:00 container create f5462925b01510408aba874066fd886d3fc869035974795c407124378be21221 (image=mesosphere/marathon:v0.15.3, name=minimesos-marathon-3431856387-3786941047)
2016-05-22T01:23:58.809992601-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=f5462925b01510408aba874066fd886d3fc869035974795c407124378be21221, name=bridge, type=bridge)
2016-05-22T01:23:58.953480463-07:00 container start f5462925b01510408aba874066fd886d3fc869035974795c407124378be21221 (image=mesosphere/marathon:v0.15.3, name=minimesos-marathon-3431856387-3786941047)
2016-05-22T01:24:00.051618619-07:00 container create 23fec8bb5e85932fa7ac449da195684c067ec61553dafc0b14b0ca2a658d3432 (image=containersol/consul-server:0.6-1, name=minimesos-consul-3431856387-1691621991)
2016-05-22T01:24:00.064317143-07:00 network connect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=23fec8bb5e85932fa7ac449da195684c067ec61553dafc0b14b0ca2a658d3432, name=bridge, type=bridge)
2016-05-22T01:24:00.304745743-07:00 container start 23fec8bb5e85932fa7ac449da195684c067ec61553dafc0b14b0ca2a658d3432 (image=containersol/consul-server:0.6-1, name=minimesos-consul-3431856387-1691621991)
2016-05-22T01:24:01.519111203-07:00 container create 5d4c87991ed1f22d6e1f8538f5b3715bf08e98237798d21a27461befc3434428 (image=gliderlabs/registrator:v6, name=minimesos-registrator-3431856387-1275330981)
2016-05-22T01:24:01.532656292-07:00 network connect a6200d48b113b6bc4bf5599eeaf7a9a0203e8d79566b840a01a360fa594e0213 (container=5d4c87991ed1f22d6e1f8538f5b3715bf08e98237798d21a27461befc3434428, name=host, type=host)
2016-05-22T01:24:01.601008058-07:00 container start 5d4c87991ed1f22d6e1f8538f5b3715bf08e98237798d21a27461befc3434428 (image=gliderlabs/registrator:v6, name=minimesos-registrator-3431856387-1275330981)
2016-05-22T01:24:10.000644617-07:00 container die 22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335 (exitCode=1, image=containersol/minimesos:0.9.0, name=elated_dubinsky)
2016-05-22T01:24:10.104753725-07:00 network disconnect f5e2f8e69d11a91f731d2b877b859809159607490dd8aa0975de170003f0d2b9 (container=22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335, name=bridge, type=bridge)
2016-05-22T01:24:10.154152721-07:00 container destroy 22721c5ce8d5943e16da78cd7baedf5c3ab66025b400046ef14ea2bdfeb14335 (image=containersol/minimesos:0.9.0, name=elated_dubinsky)
$ curl https://raw.githubusercontent.com/ContainerSolutions/minimesos/e2a43362f4581122762c80d8780d09b567783f1a/apps/weave-scope.json
{
  "id": "weave-scope",
  "cpus": 1,
  "mem": 128,
  "instances": 1,
  "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ],
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "weaveworks/scope:0.13.1",
      "network": "HOST",
      "privileged": true,
      "parameters": [
        { "key": "pid", "value": "host" },
        { "key": "name", "value": "weavescope" }
      ]
    },
    "volumes": [
      {
        "containerPath": "/var/run/docker.sock",
        "hostPath": "/var/run/docker.sock",
        "mode": "RW"
      }
    ]
  },
  "args": ["--probe.docker", "true"],
  "env": {
    "CHECKPOINT_DISABLE": ""
  },
  "portDefinitions": [
    { "port": 4040, "protocol": "tcp", "name": "http" }
  ]
}

NB:

$ echo $DOCKER_HOST
tcp://localhost:4243
$ sudo docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64
$ cat minimesosFile 
minimesos {
    clusterName = "greg"
    loggingLevel = "INFO"
    mapAgentSandboxVolume = false
    mesosVersion = "0.25"
    timeout = 60

    agent {
        imageName = "containersol/mesos-agent"
        imageTag = "# derive from mesos version"
        loggingLevel = "# INHERIT FROM CLUSTER"
        portNumber = 5051

        resources {

            cpu {
                role = "*"
                value = 1
            }

            disk {
                role = "*"
                value = 200
            }

            mem {
                role = "*"
                value = 256
            }

            ports {
                role = "*"
                value = "[31000-32000]"
            }
        }
    }

    consul {
        imageName = "containersol/consul-server"
        imageTag = "0.6-1"
    }

    marathon {
        imageName = "mesosphere/marathon"
        imageTag = "v0.15.3"

        // Add 'app { marathonJson = "<path or URL to JSON file>" }' for every task you want to execute
        app {
            marathonJson = "https://raw.githubusercontent.com/ContainerSolutions/minimesos/e2a43362f4581122762c80d8780d09b567783f1a/apps/weave-scope.json"
        }
    }

    master {
        aclJson = null
        authenticate = false
        imageName = "containersol/mesos-master"
        imageTag = "# derive from mesos version"
        loggingLevel = "# INHERIT FROM CLUSTER"
    }

    registrator {
        imageName = "gliderlabs/registrator"
        imageTag = "v6"
    }

    zookeeper {
        imageName = "jplock/zookeeper"
        imageTag = "3.4.6"
    }
}
adam-sandor commented 8 years ago

Hi @gregzuro,

Quite a strange issue, I'll try if a clean install works for me.

adam-sandor commented 8 years ago

Reproduced it, now to the fixing part...

adam-sandor commented 8 years ago

@gregzuro I fixed the problem on my machine by starting docker-machine with the google dns server docker-machine create -d virtualbox --virtualbox-memory 8192 --virtualbox-cpu-count 1 --engine-opt dns=8.8.8.8 minimesos. But if I understand correctly you are running on Ubuntu with curl actually downloading the file, so I'm baffled there. Can you explain what your setup is exactly?

frankscholten commented 8 years ago

Reopening. Issue was closed because of https://github.com/ContainerSolutions/minimesos/commit/4f0356cca509d8cd8afdc2fe2f36366701f06e40

gregzuro commented 8 years ago

@adam-sandor I'm not sure what else is relevant.

I recall that I didn't have docker-machine installed at the time I tried this, though...

adam-sandor commented 8 years ago

Do you work on Linux or another OS? On Linux you don't need docker-machine to run Docker. The problem you described was caused on my machine (Mac) by the docker-machine VM having an incorrect DNS configuration. That's why I updated the documentation page on how to create the Minimesos VM with the Google DNS server. If you are however running on Linux then Docker is using the the DNS settings of your machine and I don't have a clue what could be wrong with the download of the weave-scope.json file.

frankscholten commented 8 years ago

Let's fix this by catching the exception and printing a message like 'Could not load remote Marathon file at $URL. Please check your DNS settings and network connectivity'

gregzuro commented 8 years ago

I was using ubuntu 14.04.

I just installed a brand new ubuntu 15.10 and I get the same error.

frankscholten commented 8 years ago

Can you paste your /etc/resolv.conf? Are you behind a proxy?

gregzuro commented 8 years ago

This did turn out to be a DNS problem. Adding a --dns to my DOCKER_OPTS fixed it. Thanks.