search

netbox-community / netbox

The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/free-netbox-cloud/
http://netboxlabs.com/oss/netbox/
Apache License 2.0
16.27k stars 2.59k forks source link

"VirtualMachine matching query does not exist." when trying to delete Virtual Machines #2323

Closed pm17788 closed 6 years ago

pm17788 commented 6 years ago

Environment

Steps to Reproduce

  1. Using the filters in Virtual Machines, select all Virtual Machines with status "Offline" (in our case, that ends up being 1686 systems)
  2. Use the Select all 1686 virtual machines matching query checkbox
  3. Click Delete All button
  4. Wait for confirmation page to load
  5. Click Delete these 1686 virtual machines button

Expected Behavior

1686 selected Virtual Machines razed into oblivion

Observed Behavior

  1. After about 15 seconds, get back a Server Error page with the following text: 
    
Server Error

There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'virtualization.models.DoesNotExist'>

VirtualMachine matching query does not exist.

If further assistance is required, please post to the NetBox mailing list.

jeremystretch commented 6 years ago

Just tried this with 2000 virtual machines and wasn't able to reproduce the error. All 2000 VMs were deleted. Can you try it with DEBUG=True set in configuration.py and post the stack trace?

pm17788 commented 6 years ago

@jeremystretch: Will do. I just tried to delete a single VM, and got the same error.

Stack trace:


Request Method: | POST
-- | --
https://netbox.datto.net/virtualization/virtual-machines/119096/delete/
2.0.8
DoesNotExist
VirtualMachine matching query does not exist.
/usr/local/lib/python3.5/dist-packages/django/db/models/query.py in get, line 403
/usr/bin/python3
3.5.2
['/opt/netbox/netbox',  '/opt/netbox/netbox',  '/usr/local/bin',  '/usr/local/lib/python3.5/dist-packages/pip-9.0.1-py3.5.egg',  '/usr/lib/python35.zip',  '/usr/lib/python3.5',  '/usr/lib/python3.5/plat-x86_64-linux-gnu',  '/usr/lib/python3.5/lib-dynload',  '/usr/local/lib/python3.5/dist-packages',  '/usr/lib/python3/dist-packages']
Tue, 7 Aug 2018 20:34:25 +0000
Environment:

Request Method: POST
Request URL: https://netbox.datto.net/virtualization/virtual-machines/119096/delete/

Django Version: 2.0.8
Python Version: 3.5.2
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'corsheaders',
 'debug_toolbar',
 'django_filters',
 'django_tables2',
 'mptt',
 'rest_framework',
 'taggit',
 'taggit_serializer',
 'timezone_field',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'secrets',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'drf_yasg']
Installed Middleware:
('debug_toolbar.middleware.DebugToolbarMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'utilities.middleware.ExceptionHandlingMiddleware',
 'utilities.middleware.LoginRequiredMiddleware',
 'utilities.middleware.APIVersionMiddleware',
 'extras.middleware.ObjectChangeMiddleware')

Traceback:

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/related_descriptors.py" in __get__
  158.             rel_obj = self.field.get_cached_value(instance)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/mixins.py" in get_cached_value
  13.             return instance._state.fields_cache[cache_name]

During handling of the above exception ('virtual_machine'), another exception occurred:

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.5/dist-packages/django/contrib/auth/mixins.py" in dispatch
  85.         return super().dispatch(request, *args, **kwargs)

File "/usr/local/lib/python3.5/dist-packages/django/views/generic/base.py" in dispatch
  89.         return handler(request, *args, **kwargs)

File "/opt/netbox/netbox/utilities/views.py" in post
  280.                 obj.delete()

File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py" in delete
  891.         return collector.delete()

File "/usr/local/lib/python3.5/dist-packages/django/db/models/deletion.py" in delete
  307.                             sender=model, instance=obj, using=self.using

File "/usr/local/lib/python3.5/dist-packages/django/dispatch/dispatcher.py" in send
  178.             for receiver in self._live_receivers(sender)

File "/usr/local/lib/python3.5/dist-packages/django/dispatch/dispatcher.py" in <listcomp>
  178.             for receiver in self._live_receivers(sender)

File "/usr/local/lib/python3.5/dist-packages/django/utils/functional.py" in _curried
  11.         return _curried_func(*(args + moreargs), **dict(kwargs, **morekwargs))

File "/opt/netbox/netbox/extras/middleware.py" in _record_object_deleted
  37.         instance.log_change(request.user, request.id, OBJECTCHANGE_ACTION_DELETE)

File "/opt/netbox/netbox/dcim/models.py" in log_change
  1940.             related_object=self.get_component_parent(),

File "/opt/netbox/netbox/dcim/models.py" in get_component_parent
  1860.         return self.device or self.virtual_machine

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/related_descriptors.py" in __get__
  164.                 rel_obj = self.get_object(instance)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/related_descriptors.py" in get_object
  139.         return qs.get(self.field.get_reverse_related_filter(instance))

File "/usr/local/lib/python3.5/dist-packages/django/db/models/query.py" in get
  403.                 self.model._meta.object_name

Exception Type: DoesNotExist at /virtualization/virtual-machines/119096/delete/
Exception Value: VirtualMachine matching query does not exist.
jeremystretch commented 6 years ago

Ok, the issue seems to stem from logging the deletion of the VM interfaces rather than the VMs themselves.

jeremystretch commented 6 years ago

For some reason, the interfaces are being deleted after the VM gets deleted. As far as I can tell, only interfaces are affected: Other related objects (secrets, etc.) get deleted before the VM as they should. The exception is being raised when calling log_change() on a child interface tries to resolve its parent VM. The relationship is still there (i.e. virtual_machine_id still has a value) but the VM is gone.

pm17788 commented 6 years ago

Well then. Hrmm... Is this something I could have caused? The system's provenance is 2.3.2 upgraded successively to 2.4.1.

jeremystretch commented 6 years ago

@pm17788 No, it's an anomaly in the way Django chooses to order the deletion of related objects. I have a fix coming in v2.4.2.