Closed Eclion closed 7 years ago
rade
commented
8 years ago Can you ping es2 from es1? And what is the output of weave ps (on both hosts) and weave status dns (output from one host suffices)? And the output of docker inspect on your two es containers.
Eclion
commented
8 years ago Hi @rade No, ping doesn't work. However, we don't have the "no route to host" we had with Docker's overlay network.
Below are the information you asked:
weave ps on host1: weave:expose 96:c6:a2:a3:8f:59 51b338976dfb ae:4d:02:62:96:5e 10.32.0.1/12
weave ps on host2: weave:expose fa:0d:cb:0a:9d:8e 4de08b485dfb c2:63:17:04:69:b1 10.40.0.0/12
weave status dns (on host2, it is the same on host1): es1 10.32.0.1 51b338976dfb 96:c6:a2:a3:8f:59 es2 10.40.0.0 4de08b485dfb fa:0d:cb:0a:9d:8e
docker inspect es1
[
{
"Id": "51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279",
"Created": "2016-09-22T12:19:06.34814866Z",
"Path": "/w/w",
"Args": [
"/sbin/entrypoint",
"bin/elasticsearch"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31558,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-09-22T12:19:06.753063783Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:448c7c068b4867d3b252d857a8fb79a0011596f798186e2a863ad851638bc48b",
"ResolvConfPath": "/var/lib/docker/containers/51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279/hostname",
"HostsPath": "/var/lib/docker/containers/51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279/hosts",
"LogPath": "/var/lib/docker/containers/51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279/51b338976dfbd335f87f9deb9be466680b1cc17fdfd7f165d168b97de076e279-json.log",
"Name": "/es1",
"RestartCount": 0,
"Driver": "devicemapper",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/docker/volumes/ab5e2ef0ae5c7070bc7b27d8ebca68103ff610de1885663f4836668ace9e433b/_data:/w:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [
"172.17.0.1"
],
"DnsOptions": [],
"DnsSearch": [
"."
],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"StorageOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"BlkioIOps": 0,
"BlkioBps": 0,
"SandboxSize": 0
},
"GraphDriver": {
"Name": "devicemapper",
"Data": {
"DeviceId": "307",
"DeviceName": "docker-253:11-2149580928-99aac7b9cc949107dd52400c57a44423a75c83263c2e79da9635f63f5ed418e3",
"DeviceSize": "10737418240"
}
},
"Mounts": [
{
"Source": "/var/lib/docker/volumes/ab5e2ef0ae5c7070bc7b27d8ebca68103ff610de1885663f4836668ace9e433b/_data",
"Destination": "/w",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Name": "31de053e4eefa14d7b40b6d55ae6172873b1426f3b5cc42c7d81843df7c68463",
"Source": "/var/lib/docker/volumes/31de053e4eefa14d7b40b6d55ae6172873b1426f3b5cc42c7d81843df7c68463/_data",
"Destination": "/persistent",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "7ff488a6c079765dd32ffeea32fa7660fc4064238d9e0b611be2bc0e96d00a5b",
"Source": "/var/lib/docker/volumes/7ff488a6c079765dd32ffeea32fa7660fc4064238d9e0b611be2bc0e96d00a5b/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "es1",
"Domainname": "weave.local",
"User": "elasticsearch",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"9200/tcp": {},
"9300/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/share/elasticsearch/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin",
"ES_VERSION=2.3.2",
"ES_HOME=/usr/share/elasticsearch"
],
"Cmd": [
"bin/elasticsearch"
],
"Image": "elasticsearch",
"Volumes": {
"/persistent": {},
"/usr/share/elasticsearch/data": {}
},
"WorkingDir": "/usr/share/elasticsearch",
"Entrypoint": [
"/w/w",
"/sbin/entrypoint"
],
"OnBuild": null,
"Labels": {
"Architecture": "x86_64",
"Authoritative_Registry": "registry.access.redhat.com",
"BZComponent": "rhel-server-docker",
"Build_Host": "rcm-img02.build.eng.bos.redhat.com",
"Name": "redhat/rhel7",
"Release": "16",
"Vendor": "Red Hat, Inc.",
"Version": "7.1"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "4d3c55b3fbf6b0edfcac9379d52a1031b8d1299cccc0cb1ed15fe719c7abd94c",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"9200/tcp": null,
"9300/tcp": null
},
"SandboxKey": "/var/run/docker/netns/4d3c55b3fbf6",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "fc3ea1944f9ccb1c1015c4fa5abc70745a0a22643b72476fd752a40f8d62d2ee",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:03",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "1e9a6981d0914257d71465638f7ad72c68f32124ac9f00dd9eab45fa99e5b4f2",
"EndpointID": "fc3ea1944f9ccb1c1015c4fa5abc70745a0a22643b72476fd752a40f8d62d2ee",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:03"
}
}
}
}
]docker inspect es2:
[
{
"Id": "4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14",
"Created": "2016-09-22T12:19:16.356055058Z",
"Path": "/w/w",
"Args": [
"/sbin/entrypoint",
"bin/elasticsearch"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5696,
"ExitCode": 0,
"Error": "",
"StartedAt": "2016-09-22T12:19:16.788123941Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:4cdfdc9ca2a98fd67de4b8606ff10b0d89c18ec9c0133231eedca0711fc86219",
"ResolvConfPath": "/var/lib/docker/containers/4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14/hostname",
"HostsPath": "/var/lib/docker/containers/4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14/hosts",
"LogPath": "/var/lib/docker/containers/4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14/4de08b485dfb9a0e1b82df28aa1c6c55e30a0286161fed35382a76f7da56ca14-json.log",
"Name": "/es2",
"RestartCount": 0,
"Driver": "devicemapper",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/lib/docker/volumes/7e0ea8278efaf752446aa93bede1d33e57575214e16a7a416fac3ff9c0242138/_data:/w:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [
"172.17.0.1"
],
"DnsOptions": [],
"DnsSearch": [
"."
],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"StorageOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"BlkioIOps": 0,
"BlkioBps": 0,
"SandboxSize": 0
},
"GraphDriver": {
"Name": "devicemapper",
"Data": {
"DeviceId": "380",
"DeviceName": "docker-253:11-2149580928-6964a4619607c077c0241adb45d9841c56baf8a4ec666069bd4725b62612436c",
"DeviceSize": "10737418240"
}
},
"Mounts": [
{
"Source": "/var/lib/docker/volumes/7e0ea8278efaf752446aa93bede1d33e57575214e16a7a416fac3ff9c0242138/_data",
"Destination": "/w",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Name": "e971975230153f63b9ba608d0a143c4a3e1878e896680e387aacc13bd39c3448",
"Source": "/var/lib/docker/volumes/e971975230153f63b9ba608d0a143c4a3e1878e896680e387aacc13bd39c3448/_data",
"Destination": "/persistent",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "892a9a8c5542aa1eedbe1c3ac315ae3c3fbe1d357707bf6356434c420e47d125",
"Source": "/var/lib/docker/volumes/892a9a8c5542aa1eedbe1c3ac315ae3c3fbe1d357707bf6356434c420e47d125/_data",
"Destination": "/usr/share/elasticsearch/data",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "es2",
"Domainname": "weave.local",
"User": "elasticsearch",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"9200/tcp": {},
"9300/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/share/elasticsearch/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin",
"ES_VERSION=2.3.2",
"ES_HOME=/usr/share/elasticsearch"
],
"Cmd": [
"bin/elasticsearch"
],
"Image": "elasticsearch",
"Volumes": {
"/persistent": {},
"/usr/share/elasticsearch/data": {}
},
"WorkingDir": "/usr/share/elasticsearch",
"Entrypoint": [
"/w/w",
"/sbin/entrypoint"
],
"OnBuild": null,
"Labels": {
"Architecture": "x86_64",
"Authoritative_Registry": "registry.access.redhat.com",
"BZComponent": "rhel-server-docker",
"Build_Host": "rcm-img02.build.eng.bos.redhat.com",
"Name": "redhat/rhel7",
"Release": "16",
"Vendor": "Red Hat, Inc.",
"Version": "7.1"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "43d33ee6b001576156a2e413448deb1671cd0a4e346456f6b3c5b09bf5e39fad",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"9200/tcp": null,
"9300/tcp": null
},
"SandboxKey": "/var/run/docker/netns/43d33ee6b001",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "1710f8483b2548560d19580506826e582c3779310ba1eb47cbe8f4e1be36adb2",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:03",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "f48b0a3e6a1dba518c6baec9f3f20c13e4f0a6f7cd492221e374cce36570e1c1",
"EndpointID": "1710f8483b2548560d19580506826e582c3779310ba1eb47cbe8f4e1be36adb2",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:03"
}
}
}
}
]
bboreham
commented
8 years ago @eclion are you still looking at this?
All the data you supplied looks fine, yet you say you could not ping from one container to another. Very mysterious.
bboreham
commented
8 years ago I'm going to close this as unable to reproduce.
Eclion
commented
8 years ago Hello @bboreham. Sorry for the very late answer.
After investigating our docker configuration, we found out that the docker.advertise parameter was wrong: it was set at 0.0.0.0 instead of our machine IP. Unfortunately, we didn't try to use Weave after because it solved our problem with Docker Swarm. However, I don't think the problem was on the Weave network side.
Thank you a lot for your time.
Hello !
We are trying to use Weave Net on our RHEL cluster (as docker's overlay network doesn't support the last kernel version 3.10 used by RHEL). We have to use a isolated private network so we downloaded and loaded the docker images used by weave. On our VMs, we have docker v1.11.2 and we don't have docker machine. Then we used the following commands: Host 1: weave launch host2 Host 2: weave launch host1 Host 1: eval $(weave env) Host 2: eval $(weave env)
then we ran one elasticsearch based docker container on both hosts (named es1 & es2), but when we tried to run "curl es1:9200" from inside es2, we got a timeout error.
We don't have any error on weave containers but this one: Error checking version: Get https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=1.11.2&flag_kernel-version=3.10.0-327.el7.x86_64&os=linux&signature=PkK5Mwa%2BRKlVF2t%2F%2Fnudi2MP3zXtPlr8IUPIpBCpcb4%3D&version=1.6.1: dial tcp: lookup checkpoint-api.weave.works on 10.64.73.15:53: server misbehaving This error is due to the inaccessibility of internet from our RHEL VMs.
Here is the output of weave status:
And here is the iptables on one of our hosts:
Any idea about how we can solve this issue ? Please tell me if more information is required. :)
Thanks !