ChrisAubuchon
commented
8 years ago Can you post the Mesos master state? From http://<mesos-master>:5050/master/state.json. I'm specifically looking for those tasks.
Open Lax77 opened 8 years ago
ChrisAubuchon
commented
8 years ago Can you post the Mesos master state? From http://<mesos-master>:5050/master/state.json. I'm specifically looking for those tasks.
Lax77
commented
8 years ago Here is sample state seen for one such container:
{
"id": "hulk_sensu-server.2064fb31-3bef-11e6-bc00-424cfb1b27f9",
"name": "sensu-server.hulk",
"framework_id": "4b6fa780-f06f-4917-a10f-caaa57e7fbc6-0000",
"executor_id": "",
"slave_id": "aa0449f8-0c7c-4967-bcac-066d8c62d05b-S2",
"state": "TASK_RUNNING",
"resources": {
"cpus": 0.1,
"disk": 0,
"mem": 256
},
"statuses": [
{
"state": "TASK_RUNNING",
"timestamp": 1466980893.35396,
"labels": [
{
"key": "Docker.NetworkSettings.IPAddress",
"value": "10.100.0.179"
}
],
"container_status": {
"network_infos": [
{
"ip_address": "10.100.0.179",
"ip_addresses": [
{
"ip_address": "10.100.0.179"
}
]
}
]
}
}
],
"labels": [
{
"key": "task_type",
"value": "infra"
}
],
"discovery": {
"visibility": "FRAMEWORK",
"name": "sensu-server.hulk",
"ports": {
"ports": [
{
"number": 4567,
"name": "sensuserverport",
"protocol": "tcp"
}
]
}
},
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.internal.com:5915/lkota/sensu-server",
"network": "HOST",
"privileged": false,
"parameters": [
{
"key": "net",
"value": "services"
}
],
"force_pull_image": true
},
"network_infos": [
{
"ip_addresses": [
{}
],
"labels": {}
}
]
}
},I am guessing primary reason for the meson-consul registering with port 0 is because of this API: https://github.com/CiscoCloud/mesos-consul/blob/c442e3dd053399dcc28742f82e4b20b7007d4d15/mesos/register.go#L167. I could be wrong too.
ChrisAubuchon
commented
8 years ago That's what I'm thinking as well Re: port 0
For the other case you mentioned, are the port numbers the same in the discovery info? If so, that would explain why only one instance is being registered since mesos-consul uses the port number to create the consul service ID.
Lax77
commented
8 years ago Yeah, port numbers are same (4567). Here is the mess state response with both containers info.
{
"id": "hulk_sensu-server.2064fb31-3bef-11e6-bc00-424cfb1b27f9",
"name": "sensu-server.hulk",
"framework_id": "4b6fa780-f06f-4917-a10f-caaa57e7fbc6-0000",
"executor_id": "",
"slave_id": "aa0449f8-0c7c-4967-bcac-066d8c62d05b-S2",
"state": "TASK_RUNNING",
"resources": {
"cpus": 0.1,
"disk": 0,
"mem": 256
},
"statuses": [
{
"state": "TASK_RUNNING",
"timestamp": 1466980893.35396,
"labels": [
{
"key": "Docker.NetworkSettings.IPAddress",
"value": "10.100.0.179"
}
],
"container_status": {
"network_infos": [
{
"ip_address": "10.100.0.179",
"ip_addresses": [
{
"ip_address": "10.100.0.179"
}
]
}
]
}
}
],
"labels": [
{
"key": "task_type",
"value": "infra"
}
],
"discovery": {
"visibility": "FRAMEWORK",
"name": "sensu-server.hulk",
"ports": {
"ports": [
{
"number": 4567,
"name": "sensuserverport",
"protocol": "tcp"
}
]
}
},
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.internal.com:5915/lkota/iot-sensu-server",
"network": "HOST",
"privileged": false,
"parameters": [
{
"key": "net",
"value": "services"
}
],
"force_pull_image": true
},
"network_infos": [
{
"ip_addresses": [
{}
],
"labels": {}
}
]
}
},
{
"id": "hulk_sensu-server.20652242-3bef-11e6-bc00-424cfb1b27f9",
"name": "sensu-server.hulk",
"framework_id": "4b6fa780-f06f-4917-a10f-caaa57e7fbc6-0000",
"executor_id": "",
"slave_id": "aa0449f8-0c7c-4967-bcac-066d8c62d05b-S2",
"state": "TASK_RUNNING",
"resources": {
"cpus": 0.1,
"disk": 0,
"mem": 256
},
"statuses": [
{
"state": "TASK_RUNNING",
"timestamp": 1466980893.3635,
"labels": [
{
"key": "Docker.NetworkSettings.IPAddress",
"value": "10.100.0.180"
}
],
"container_status": {
"network_infos": [
{
"ip_address": "10.100.0.180",
"ip_addresses": [
{
"ip_address": "10.100.0.180"
}
]
}
]
}
}
],
"labels": [
{
"key": "task_type",
"value": "infra"
}
],
"discovery": {
"visibility": "FRAMEWORK",
"name": "sensu-server.hulk",
"ports": {
"ports": [
{
"number": 4567,
"name": "sensuserverport",
"protocol": "tcp"
}
]
}
},
"container": {
"type": "DOCKER",
"docker": {
"image": "docker.internal.com:5915/lkota/iot-sensu-server",
"network": "HOST",
"privileged": false,
"parameters": [
{
"key": "net",
"value": "services"
}
],
"force_pull_image": true
},
"network_infos": [
{
"ip_addresses": [
{}
],
"labels": {}
}
]
}
},Yep. That's it. Mesos-consul uses the consul agent address and port number in the service ID. Which was fine when ip-per-container wasn't there. PR https://github.com/CiscoCloud/mesos-consul/pull/88 should take care of that.
Lax77
commented
8 years ago Ok, Thanks Chris. If I pull the latest docker image it should have the fix? or do I have to build one with the latest code pull?
Lax77
commented
8 years ago Re-tried with the latest image. Could see multiple instances on same box issue got fixed. Now I could see all the container's IPs being of multi instances on the same box being registered into Consul
As per Port 0, still see IP per task containers listing one with actual port (in my case 4567) and one with port 0
ChrisAubuchon
commented
8 years ago PR https://github.com/CiscoCloud/mesos-consul/pull/89 should take care of the port 0 service.
Lax77
commented
8 years ago Thanks for the quick fix Chris. Just verified the latest image, Port 0 issue is also gone now.
Here is what I am doing. I am using mesos IP per task (thru Calico) where in each container is given a separate IP. I had a case where multiple instances of a task deployed on the same node, with each instance having assigned an IP of its own(10.100.0.179, 10.100.0.180) . So far good, but when it comes to registering those instances into Consul, mesos-consul happens to register only one of the instance (10.100.0.179) and skipping the other one.
Is there a known limitation with mesos-consul?
Also whenever I register IP per container service thru mesos-consul I see 2 entries with same IP within Consul. One with the Port service is listening on and one with port 0. Why eg: If my container has an IP 10.100.0.155 and listening on port 8000. Consul sees 2 entries one with 10.100.0.155:8000 10.100.0.155:0
Why does mesos-consul registers with port 0? I tried with mesos-ip-order as 'docker,mesos,host' and also as just 'docker'. Both the times I get an entry with port 0.