search

openstack-k8s-operators / nova-operator

Apache License 2.0
10 stars 44 forks source link

cold migrate between computes is not possible due to missing neutron service token from the super-conductor conf. #298

Closed gibizer closed 1 year ago

gibizer commented 1 year ago 
2023-03-22 13:05:33.041 20 ERROR nova.network.neutron [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] The [neutron] section of your nova configuration file must be configured for authentication with the networking service endpoint. See the networking service install guide for details: https://docs.openstack.org/neutron/latest/install/
2023-03-22 13:05:33.068 20 WARNING nova.scheduler.utils [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] Failed to compute_task_migrate_server: Unknown auth type: None: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.069 20 WARNING nova.scheduler.utils [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] [instance: 507bd531-6541-43a2-91a1-5052600cf6cd] Setting instance to ACTIVE state.: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server [None req-1046e0ce-73a1-430a-9e4b-79637a1a24ea edc732502b8448b2b6dcbce3afed8416 5507f208ce5d434dbbfa30468cf3c011 - - default default] Exception during message handling: neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 100, in wrapper
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 331, in migrate_server
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self._cold_migrate(context, instance, flavor,
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 426, in _cold_migrate
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     LOG.info('During %s the instance was deleted.',
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise self.value
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 385, in _cold_migrate
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     task.execute()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 26, in wrap
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.rollback(ex)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.force_reraise()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise self.value
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 23, in wrap
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return original(self)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/base.py", line 40, in execute
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     return self._execute()
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/conductor/tasks/migrate.py", line 253, in _execute
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     self.network_api.get_requested_resource_for_instance(
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 2586, in get_requested_resource_for_instance
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     neutron = get_client(context, admin=True)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 247, in get_client
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     auth_plugin = _get_auth_plugin(context, admin=admin)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 227, in _get_auth_plugin
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     _ADMIN_AUTH = _load_auth_plugin(CONF)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 88, in _load_auth_plugin
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server     raise neutron_client_exc.Unauthorized(message=err_msg)
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server neutronclient.common.exceptions.Unauthorized: Unknown auth type: None
2023-03-22 13:05:33.106 20 ERROR oslo_messaging.rpc.server

This is due to the limited service token configuration in https://github.com/openstack-k8s-operators/nova-operator/blob/master/templates/novaconductor/config/nova.conf

We need to move the conductors to use the new config generation pattern as that config already has a templated out neutron neutron section.

gibizer commented 1 year ago

I believe this was resolved. 

$ oc rsh -n openstack nova-cell0-conductor-0 grep '\[neutron\]' /etc/nova/nova.conf.d/01-nova.conf -A10
[neutron]
auth_url = http://keystone-internal.openstack.svc:5000
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = 12345678
cafile = 
region_name = regionOne
valid_interfaces = internal