search

bilelmsekni / OpenStack-Folsom-Install-guide

A full installation guide for OpenStack Folsom with Quantum
161 stars 136 forks source link

Meta-data server can't be reach... #92

Closed goku417 closed 11 years ago

goku417 commented 11 years ago

Hi guys,

I need your help. When my vms try to reach the meta-data server, I always have these errors:

2013-03-21 17:08:52,802 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: url error [timed out]

2013-03-21 17:09:43,854 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: url error [timed out]

2013-03-21 17:10:01,873 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: url error [timed out]

And i already put my rule in IPtables to dnat 169.254.169.254 to my controller node. Here the rule: iptables -t nat -A PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.4.74:8775

And also,

2013-03-21 15:19:07 1252 INFO nova.metadata.wsgi.server [-] 192.168.4.74 - - [21/Mar/2013 15:19:07] "GET /latest/meta-data/ HTTP/1.1" 404 278 0.124937

2013-03-21 15:31:17 1252 INFO nova.api.ec2 [-] 0.121127s 192.168.4.74 GET /openstack None:None 404 [curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3] text/plain text/html

2013-03-21 15:31:17 1252 INFO nova.metadata.wsgi.server [-] 192.168.4.74 - - [21/Mar/2013 15:31:17] "GET /openstack HTTP/1.1" 404 278 0.122755

2013-03-21 15:31:33 1267 INFO nova.ec2.wsgi.server [-] 192.168.4.74 - - [21/Mar/2013 15:31:33] "GET /openstack HTTP/1.1" 404 455 0.001182

Did someone has the same problem and solved it ?

bilelmsekni commented 11 years ago

You can debug this using:

http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html

regards, Bilel

2013/3/21 goku417 notifications@github.com

Hi guys,

I need your help. When my vms try to reach the meta-data server, I always have these errors:

2013-03-21 17:08:52,802 - util.py[WARNING]: ' http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: url error [timed out]

2013-03-21 17:09:43,854 - util.py[WARNING]: ' http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: url error [timed out]

2013-03-21 17:10:01,873 - util.py[WARNING]: ' http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: url error [timed out]

And i already put my rule in IPtables to dnat 169.254.169.254 to my controller node. Here the rule: iptables -t nat -A PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.4.74:8775

And also,

2013-03-21 15:19:07 1252 INFO nova.metadata.wsgi.server [-] 192.168.4.74 -

  • [21/Mar/2013 15:19:07] "GET /latest/meta-data/ HTTP/1.1" 404 278 0.124937

2013-03-21 15:31:17 1252 INFO nova.api.ec2 [-] 0.121127s 192.168.4.74 GET /openstack None:None 404 [curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3] text/plain text/html

2013-03-21 15:31:17 1252 INFO nova.metadata.wsgi.server [-] 192.168.4.74 -

  • [21/Mar/2013 15:31:17] "GET /openstack HTTP/1.1" 404 278 0.122755

2013-03-21 15:31:33 1267 INFO nova.ec2.wsgi.server [-] 192.168.4.74 - - [21/Mar/2013 15:31:33] "GET /openstack HTTP/1.1" 404 455 0.001182

Did someone has the same problem and solved it ?

— Reply to this email directly or view it on GitHubhttps://github.com/mseknibilel/OpenStack-Folsom-Install-guide/issues/92 .

Best regards,

Bilel Msekni | Research & Development Engineer Institut Mines-Telecom | TELECOM & Management SudParis 9 rue Charles Fourier, 91011 EVRY Cedex, France Mobile: +33 6 49 52 42 17

goku417 commented 11 years ago

Thank you for your fast anwser! Everything pings but i still can't connect to the metadata server. If i check the log, it seems like it have nothing on the metadata server. How can i verify if i have something on the metadata server ?

Thanks Bunpa

goku417 commented 11 years ago

Update on my status! I can now ping the external network from my vm but still can't talk with the metadata server. Anyone have an idea ?

Thanks Bunpa

f905201 commented 11 years ago

"route add -net 50.50.1.0/24 gw 192.168.100.102 " seems to make VM able to talk to metadata server, the controller node. I have tested that if I remove this rule, vm can't reach my controller node,'192.168.100.51'.

goku417 commented 11 years ago

what represent 192.168.100.102 ?

f905201 commented 11 years ago

following the 3-nic-vlan guide, '192.168.100.102' seems to be a gateway that enable vm to talk to external network, so does metadata server

goku417 commented 11 years ago

I add the route to my controller, but still don't have a communication between my VMs and my metadata server....

f905201 commented 11 years ago

I have no idea how to help you debugging, but I find that if my compute node is able to access external network, my VMs will also have problem to talk to metadata server. On the other hand, if I keep compute node having 100.10.10.53 & 100.20.20.53 just like what guide say that unable to reach internet, my VM can talk to metadata server.

hope this can help you

goku417 commented 11 years ago

Thx for your help f905201 it's really appreciated ! it's been a month I work on this project for my company.

I still can't reach my metadata... =(

I get this message on my VM:

2013-04-03 10:04:45,521 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [51/120s]: url error [timed out]

2013-04-03 10:05:36,574 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [102/120s]: url error [timed out]

2013-04-03 10:05:53,592 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: url error [timed out]

2013-04-03 10:05:54,593 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds

But my VMs can reach the internet...

I decided to run tcpdump to watch the traffic on my controller node and here's the log (I only watch for port 8775):

14:05:37.577505 IP 192.168.6.4.57624 > 10.0.1.2.8775: Flags [S], seq 470106563, win 14600, options [mss 1460,sackOK,TS val 4294924369 ecr 0,nop,wscale 5], length 0

14:05:38.574767 IP 192.168.6.4.57624 > 10.0.1.2.8775: Flags [S], seq 470106563, win 14600, options [mss 1460,sackOK,TS val 4294924619 ecr 0,nop,wscale 5], length 0

14:05:40.578723 IP 192.168.6.4.57624 > 10.0.1.2.8775: Flags [S], seq 470106563, win 14600, options [mss 1460,sackOK,TS val 4294925120 ecr 0,nop,wscale 5], length 0

14:05:44.586231 IP 192.168.6.4.57624 > 10.0.1.2.8775: Flags [S], seq 470106563, win 14600, options [mss 1460,sackOK,TS val 4294926122 ecr 0,nop,wscale 5], length 0

14:05:52.610738 IP 192.168.6.4.57624 > 10.0.1.2.8775: Flags [S], seq 470106563, win 14600, options [mss 1460,sackOK,TS val 4294928128 ecr 0,nop,wscale 5], length 0

the packets reach the controller node but the controller doesn't give any reply ... and here's what I get when I do ps -ef | grep nova:

root@controller:/home/controller# ps -ef | grep nova nova 13623 1 0 Apr01 ? 00:00:00 su -s /bin/sh -c exec nova-cert --config-file=/etc/nova/nova.conf nova nova 13624 13623 0 Apr01 ? 00:04:27 /usr/bin/python /usr/bin/nova-cert --config-file=/etc/nova/nova.conf nova 13635 1 0 Apr01 ? 00:00:00 su -s /bin/sh -c exec nova-consoleauth --config-file=/etc/nova/nova.conf nova nova 13636 13635 0 Apr01 ? 00:04:32 /usr/bin/python /usr/bin/nova-consoleauth --config-file=/etc/nova/nova.conf nova 13647 1 0 Apr01 ? 00:00:00 su -s /bin/sh -c exec nova-novncproxy --config-file=/etc/nova/nova.conf nova nova 13648 13647 0 Apr01 ? 00:00:21 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf nova 13663 1 0 Apr01 ? 00:00:00 su -s /bin/sh -c exec nova-scheduler --config-file=/etc/nova/nova.conf nova nova 13664 13663 0 Apr01 ? 00:04:48 /usr/bin/python /usr/bin/nova-scheduler --config-file=/etc/nova/nova.conf nova 15151 1 0 Apr02 ? 00:00:00 su -s /bin/sh -c exec nova-api --config-file=/etc/nova/nova.conf nova nova 15152 15151 0 Apr02 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 15157 15152 0 Apr02 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 15158 15152 0 Apr02 ? 00:00:05 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 15159 15152 0 Apr02 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 15160 15152 0 Apr02 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nova 15480 13648 0 14:03 ? 00:00:00 [nova-novncproxy] nova 15481 13648 0 14:03 ? 00:00:00 [nova-novncproxy] nova 15482 13648 0 14:03 ? 00:00:00 [nova-novncproxy] nova 15483 13648 0 14:03 ? 00:00:00 [nova-novncproxy] nova 15484 13648 0 14:03 ? 00:00:00 [nova-novncproxy] nova 15485 13648 0 14:03 ? 00:00:00 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf root 15495 15438 0 14:10 pts/1 00:00:00 grep --color=auto nova

Well if nova-api is running that means my metadata should run too...

Hope this will help to solve my problem...

markharding commented 11 years ago

We are having the exact same issue as goku417

no instance data found in start-local

ci-info: lo    : 1 127.0.0.1       255.0.0.0       .

ci-info: eth0  : 1 10.5.5.3        255.255.255.0   fa:16:3e:ab:28:c7

ci-info: route-0: 0.0.0.0         10.5.5.1        0.0.0.0         eth0   UG

ci-info: route-1: 10.5.5.0        0.0.0.0         255.255.255.0   eth0   U

cloud-init start running: Tue, 02 Apr 2013 23:23:12 +0000. up 10.78 seconds

2013-04-02 23:26:03,104 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: url error [timed out]

2013-04-02 23:26:54,157 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: url error [timed out]

2013-04-02 23:27:12,177 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: url error [timed out]

2013-04-02 23:27:13,179 - DataSourceEc2.py[CRITICAL]: giving up on md after 120 seconds

We think this could be an issue with a recent update to quantum in regards to the iptables routing to the dhcp and l3 services. Any suggestions or workaround would be appreciated! We're really anxious to get something up and running. Thanks a lot!

bilelmsekni commented 11 years ago

You can make sure that you can ping the VM from the controller node. And you can ping out the controller node( cause it has the metadata) from the network node ( cause it has the L3 agent) through the metadata server IP described in the l3_agent.ini file. More details can be found here: http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html

goku417 commented 11 years ago

Thx for your help mseknibilel.

Everything can be ping

Controller to VM : OK Network to Controller : OK

Like I said in my previous comment, I did tcpdump tcp port 8775 -i eth1 on my controller node (The interface eth1 is the one that I put in my l3_agent.ini file as my metadata server). When you check the log (in my previous comment), you saw that the controller node receive the packets from a VM but doesn't respond to the request ... so im wondering what happen to the packets...

FYI, I dont have iptables on it, so no firewall no rules , accept all packet.

goku417 commented 11 years ago

It's been a month I try to fix the problem but no result ... So the best solution is to reinstall again ... Thank you to you guys ! Your help was very appreciated ! =D

robertoafilho commented 6 years ago

Just be sure to have enable_isolated_metadata = True in /etc/neutron/dhcp_agent.ini and then systemctl restart neutron-dhcp-agent : and from that point, cloud metadata will be served from dhcp too.