Unable to add Plugin to nautobot

mkrsn commented 3 years ago

Environment

Python version: 3.8.5
Nautobot version: 1.0.0b1
nautobot-device-onboarding version: 1.0.0

Steps to Reproduce

Install Plugin as documented (either manual with pip install or via local_requirement and install.sh)
restart nautobot{,-rq}
Open Nautobot WebUI (receive HTTP500)

Expected Behavior

Nautobot should not explode

Observed Behavior

Both Services are starting - logs are clean
When visiting the WebUI gunicorn throws a HTTP500 and logs the following information

nautobox.log

When i remove the Plugin from PLUGINS then nautobot is working well.

glennmatthews commented 3 years ago

Thanks for the report. This looks like the same root cause as https://github.com/nautobot/nautobot/issues/95.

dgarros commented 3 years ago

Please can you try again with the latest version of nautobot 1.0.0b2 ?

mkrsn commented 3 years ago

Looks way better. Nautobot is able to start but there is still something broken. Sometimes I'm able to see the add-page for the Onboarding Plugin and sometimes i get the following Error.

fyi: /opt/nautobot/current is a symlink to /opt/nautobot/1.0.0b2/

Server Error
There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'django.db.utils.OperationalError'>

SSL error: decryption failed or bad record mac

Python version: 3.8.5
Nautobot version: 1.0.0b2

Screenshot_20210310_130201

OperationalError at /plugins/device-onboarding/
SSL error: decryption failed or bad record mac
Request Method: GET
Request URL:    https://<schnipp>/plugins/device-onboarding/
Django Version: 3.1.7
Exception Type: OperationalError
Exception Value:    
SSL error: decryption failed or bad record mac
Exception Location: /opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:  /usr/bin/python3.8
Python Version: 3.8.5
Python Path:    
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:    Wed, 10 Mar 2021 12:34:59 +0000

Traceback

Environment:

Request Method: GET
Request URL: https://<schnipp>/plugins/device-onboarding/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL error: decryption failed or bad record mac
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 91, in dispatch
    if not self.has_permission():
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 73, in has_permission
    if user.has_perms((permission_required, *self.additional_permissions)):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /plugins/device-onboarding/
Exception Value: SSL error: decryption failed or bad record mac

mkrsn commented 3 years ago

It looks like the error varnished after nautobot is running a while. I'm only able to trigger it after restarting nautobot

mkrsn commented 3 years ago

another (but related) Error, happens to my colleague. But not while using the Onboarding Plugin.

OperationalError at /
SSL SYSCALL error: EOF detected
Request Method: GET
Request URL:    https://<schnipp>/
Django Version: 3.1.7
Exception Type: OperationalError
Exception Value:    
SSL SYSCALL error: EOF detected
Exception Location: /opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:  /usr/bin/python3.8
Python Version: 3.8.5
Python Path:    
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:    Wed, 10 Mar 2021 13:06:33 +0000

Environment:

Request Method: GET
Request URL: https://<schnipp>/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']

Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL SYSCALL error: EOF detected
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/core/views/__init__.py", line 45, in get
    ConsolePort.objects.restrict(request.user, "view")
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/querysets.py", line 21, in restrict
    if user.is_superuser or permission_is_exempt(permission_required):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /
Exception Value: SSL SYSCALL error: EOF detected

mkrsn commented 3 years ago

I've disabled the onboarding plugin. The error still exists. So this Error is not related to the onboarding Plugin.

mkrsn commented 3 years ago

Thanks :)

nautobot / nautobot-app-device-onboarding