rossella
commented
11 years ago @admytren thanks for filing this and for collecting all the logs!
In the dhcp agent log I can see a repeated error, every time we create a network the method "plug" in our plugin is called and it fails every time because the request to the api gets ECONNREFUSED . This means that the plugin always fails in doing the binding.
2013-07-23 20:50:11 ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp. Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 131, in call_driver getattr(driver, action)() File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/dhcp.py", line 124, in enable reuse_existing=True) File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 554, in setup namespace=namespace) File "/usr/lib/python2.7/dist-packages/quantum/plugins/midonet/agent/midonet_agent.py", line 125, in plug host = self.mido_api.get_host(host_uuid) File "/usr/lib/pymodules/python2.7/midonetclient/api.py", line 92, in get_host self._ensure_application() File "/usr/lib/pymodules/python2.7/midonetclient/api.py", line 178, in _ensure_application self.app.get() File "/usr/lib/pymodules/python2.7/midonetclient/resource_base.py", line 58, in get res, self.dto = self.auth.do_request(uri, 'GET', headers=headers) File "/usr/lib/pymodules/python2.7/midonetclient/auth_lib.py", line 99, in do_request self.set_header_token(headers) File "/usr/lib/pymodules/python2.7/midonetclient/auth_lib.py", line 89, in set_header_token header['X-Auth-Token'] = self.get_token(force) File "/usr/lib/pymodules/python2.7/midonetclient/auth_lib.py", line 83, in get_token self.login() File "/usr/lib/pymodules/python2.7/midonetclient/auth_lib.py", line 64, in login headers=headers) File "/usr/lib/pymodules/python2.7/midonetclient/api_lib.py", line 79, in do_request headers=headers) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1543, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1293, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1229, in _conn_request conn.connect() File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 889, in connect raise socket.error, msg error: [Errno 111] ECONNREFUSED
Is the port open?
Apart from that I can see in quantum.log many of these:
2013-07-23 17:58:43 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data rval = self.proxy.dispatch(ctxt, version, method, _args) File "/usr/lib/python2.7/dist-packages/quantum/common/rpc.py", line 43, in dispatch quantum_ctxt, version, method, _kwargs) File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch return getattr(proxyobj, method)(ctxt, **kwargs) File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 173, in report_state plugin.create_or_update_agent(context, agent_state) AttributeError: 'MidonetPluginV2' object has no attribute 'create_or_update_agent'
Artem you already reported that, after reading about redundancy in the last days I'm pretty sure that this is due to the redundant setup. If you have 2 dhcp agents quantum needs a scheduler to coordinate them. The scheduler should be implemented in the plugin, that's why quantum is complaining about this 'MidonetPluginV2' object has no attribute 'create_or_update_agent'.
This is also suspicious:
2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension agent_scheduler not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension flavor not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension agent not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension port-security not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension provider not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension service-type not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension binding not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension lbaas not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension quotas not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension extraroute not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension routed-service-insertion not supported by any of loaded plugins 2013-07-23 17:58:35 WARNING [quantum.api.extensions] Extension router-service-type not supported by any of loaded plugins
It seems quantum is configured to require all this extensions. Why?
Apart from that the quantum server log is not very rich I think you weren't always using this server. There are time gaps for example from 18 to 19 and from 20 to 21 where nothing is logged.
admytren
Found an issue where metadata server integration code is not binding midolman to tap device connected to network namespace hosting tenant's metadata server.
Logs are available at: https://drive.google.com/a/midokura.jp/folderview?id=0B4y46DLVnlneekNOektsM2JhUlE&usp=sharing (only visible to Midokura organization).
The workaround is to bind to the tap device manually using CLI command to tenant's bridge: midonet> host hostX add binding port bridgeY:portZ interface tap[ID]